From pete-lists at bcmpweb.com Thu Apr 6 22:28:03 2006 From: pete-lists at bcmpweb.com (Peter Simmons) Date: Fri, 07 Apr 2006 08:28:03 +1200 Subject: [Z3-sqlos] zope 2.x, Five and sqlos In-Reply-To: <20060406093706.GA7328@minipas.home> References: <442BC3AE.8090705@bcmpweb.com> <20060330152125.GA7317@minipas.home> <4434B3F5.2060608@bcmpweb.com> <20060406093706.GA7328@minipas.home> Message-ID: <443579D3.3080703@bcmpweb.com> Hi Brian, Brian Sutherland wrote: > On Thu, Apr 06, 2006 at 06:23:49PM +1200, Peter Simmons wrote: > >> Hi, >> >> Thanks that helps a lot. Is anyone out there using Sqlos with Zope2.x >> and Five? >> > > Yes, I have one site in production and I know of at least one other. In > the sqlos source there is a Zope2 directory containing a FiveSQLOS Z2 > product. That provides the necessary magic glue between sqlos and Zope2. > > Excellent sounds like we have a starting place ;-) > But beware, thare be dragons here, FiveSQLOS contains some _really_ > black magic and I strongly urge you to stick to Zope3 unless you have a > very good reason not to do so. > Well we have an existing site about 3.5 years old based on plone 0.4 (although we have upgraded/back ported a number of new nicer things over the years), the client and therefore us can't afford a complete rebuild and also can't afford to wait for a framework upgrade any longer. So we pretty much have to stick with using a version of plone. The latest is plone2.5 currently beta. So we are going to migrate to that version (not a simple thing in this case, current site is python 2.1, zope 2.6.4 and we hit a wall trying to upgrade seamlessly from that about 18 months ago due to syntax changes) which looks like using zope 2.8 and the version of Five that ships with plone 2.5 (1.2 I think). We really want to move all out data out of ZODB (around 20,000 users, about 5 different types of users, over 130,000 transactions already in SQL db) and into a SQL DB which will make site maintenance much simpler and faster as one of the major things now is if we add a field to 20,000 users doing the data initialisation causes major headaches as we have to run it in batches of about 5,000 to not kill the server/site (it is running about 18 hours of the day pretty solid). We looked around and liked the approach of SQLObject, I had heard you guys had done some z3 stuff with it and we have been wanting to move dev to z3 technologies for a while (we mostly do ArchGenXML based AT stuff for more current projects at the mo' but have seen the issues with it scaling first hand) so it seemed like a nice fit in principal. As you can guess I am now just working out the details of the best fit for things. You guys haven't done a PAS plug-in that also uses SQLObject in the back have you? I installed the SQLPASPlugin which looks like it will do the trick for us but works differently to the rest so another way of doing things to document/test/remember. Would be nice to get past that if possible. Maybe if we have time at a later stage if there isn't something already. > But could you test something for me? With a fresh checkout of sqlos, > read the section in the readme "A Sample Application" and try it out? > > We will certainly test this with the versions mentioned above and sing out if we need some help/find issues. It will be a good primer for us anyway. >> Cheers, >> Pete >> >> Brian Sutherland wrote: >> >>> On Thu, Mar 30, 2006 at 11:40:30PM +1200, Peter Simmons wrote: >>> >>> >>>> Hi, >>>> >>>> Just wondering if there are any tutorial's or examples of using sqlos to >>>> use SQLObject with Zope3? >>>> >>>> >>> Yes there is a tutorial included with the source, but I think it's a bit >>> out of date. >>> >>> Probably the best way is to look at the functional testing harness, that >>> is a mini application in it's own right. It is also guaranteed to be >>> correct. >>> >>> Probably the best place to ask questions is this list, or #z3-base on >>> freenode. >>> >>> >>> >>>> Cheers, >>>> Pete >>>> _______________________________________________ >>>> z3-sqlos mailing list >>>> z3-sqlos at codespeak.net >>>> http://codespeak.net/mailman/listinfo/z3-sqlos >>>> >>>> >>>> >>> >>> >> -- >> BCMPweb Support >> Phone: +64 9 267 4676 >> Fax: +64 21 219 0650 >> Email: support at bcmpweb.com >> Web: http://www.bcmpweb.com/ >> >> >> > > From pete-lists at bcmpweb.com Thu Apr 6 23:54:02 2006 From: pete-lists at bcmpweb.com (Peter Simmons) Date: Fri, 07 Apr 2006 09:54:02 +1200 Subject: [Z3-sqlos] zope 2.x, Five and sqlos In-Reply-To: <20060406093706.GA7328@minipas.home> References: <442BC3AE.8090705@bcmpweb.com> <20060330152125.GA7317@minipas.home> <4434B3F5.2060608@bcmpweb.com> <20060406093706.GA7328@minipas.home> Message-ID: <44358DFA.3020304@bcmpweb.com> Hi Brian, Thinking about your comments "But beware, thare be dragons here, FiveSQLOS contains some _really_ black magic and I strongly urge you to stick to Zope3 unless you have a very good reason not to do so. " a bit more. Probably this hasn't been tested in a real world situation but in your opinion would this black magic be a big risk in terms of scalability and performance? One of the things that will happen with this site is it is likely to grow to a few hundred thousand users (and milliions of transactions) in the next 12 - 18 months. We are trying to future proof against that as much as possible. Our main thrust is in getting the data into a SQL db (likely mysql) which handles these numbers of objects/rows much better and fits because largely the data is relational in nature. Our general concept is let's go with some kind of object relational mapping to make the transition easier and more reuse of the exisitng code and then if/when/where required we can always do direct SQL to make things more efficient. Am more than happy for you to poke holes in our theories/plans (especially at this early stage ;-) so any thoughts/feedback based on your experience would be very much appreciated (same goes for anyone else on the list also ;-) Cheers, Pete Brian Sutherland wrote: > On Thu, Apr 06, 2006 at 06:23:49PM +1200, Peter Simmons wrote: > >> Hi, >> >> Thanks that helps a lot. Is anyone out there using Sqlos with Zope2.x >> and Five? >> > > Yes, I have one site in production and I know of at least one other. In > the sqlos source there is a Zope2 directory containing a FiveSQLOS Z2 > product. That provides the necessary magic glue between sqlos and Zope2. > > But beware, thare be dragons here, FiveSQLOS contains some _really_ > black magic and I strongly urge you to stick to Zope3 unless you have a > very good reason not to do so. > > But could you test something for me? With a fresh checkout of sqlos, > read the section in the readme "A Sample Application" and try it out? > > >> Cheers, >> Pete >> >> Brian Sutherland wrote: >> >>> On Thu, Mar 30, 2006 at 11:40:30PM +1200, Peter Simmons wrote: >>> >>> >>>> Hi, >>>> >>>> Just wondering if there are any tutorial's or examples of using sqlos to >>>> use SQLObject with Zope3? >>>> >>>> >>> Yes there is a tutorial included with the source, but I think it's a bit >>> out of date. >>> >>> Probably the best way is to look at the functional testing harness, that >>> is a mini application in it's own right. It is also guaranteed to be >>> correct. >>> >>> Probably the best place to ask questions is this list, or #z3-base on >>> freenode. >>> >>> >>> >>>> Cheers, >>>> Pete >>>> _______________________________________________ >>>> z3-sqlos mailing list >>>> z3-sqlos at codespeak.net >>>> http://codespeak.net/mailman/listinfo/z3-sqlos >>>> >>>> >>>> >>> >>> >> -- >> BCMPweb Support >> Phone: +64 9 267 4676 >> Fax: +64 21 219 0650 >> Email: support at bcmpweb.com >> Web: http://www.bcmpweb.com/ >> >> >> > > From jinty at web.de Fri Apr 7 01:08:29 2006 From: jinty at web.de (Brian Sutherland) Date: Fri, 7 Apr 2006 01:08:29 +0200 Subject: [Z3-sqlos] zope 2.x, Five and sqlos In-Reply-To: <44358CC4.1050101@bcmpweb.com> References: <442BC3AE.8090705@bcmpweb.com> <20060330152125.GA7317@minipas.home> <4434B3F5.2060608@bcmpweb.com> <20060406093706.GA7328@minipas.home> <443579D3.3080703@bcmpweb.com> <44358CC4.1050101@bcmpweb.com> Message-ID: <20060406230829.GA9793@minipas.home> On Fri, Apr 07, 2006 at 09:48:52AM +1200, Peter Simmons wrote: > Hi Brian, > > Thinking about your comments "But beware, thare be dragons here, > FiveSQLOS contains some _really_ > black magic and I strongly urge you to stick to Zope3 unless you have a > very good reason not to do so. " a bit more. The black magic of FiveSQLOS is related to the Acquisition and SQLObject meta-classes. You cannot make an object that sub-classes from both of them. But if you want to use SQLObject's in Zope2, you need an object with properties of both these classes. This is complicated by the fact that even a Zope2 SimpleItem is so complicated. For that we use a this voodoo type wrapper: http://codespeak.net/svn/z3/sqlos/trunk/Zope2/FiveSQLOS/wrapper.py But there is good news on the horizon, I just saw a mail on zope-dev talking about having a containment based alternative to Acquisition. If Acquisition is not required in Zope 2 anymore most of the dragons are slain. Also, in my opinion, the way the containers in both sqlos and FiveSQLOS work could use a big re-think. I know others are using their own custom containers, but none have been contributed back yet. > Probably this hasn't been tested in a real world situation but in your > opinion would this black magic be a big risk in terms of scalability and > performance? FiveSQLOS is untested (I only know of 3 smallish sites using this) but it is a very thin layer sqlos seems to have gotten most of the kinks out of the lower layers and I have heard of it being used in pretty large sites. The SQLObject and Zope3 combination is being used in a very large installation (http://launchpad.net). > One of the things that will happen with this site is it is likely to > grow to a few hundred thousand users (and milliions of transactions) in > the next 12 - 18 months. We are trying to future proof against that as > much as possible. Our main thrust is in getting the data into a SQL db > (likely mysql) which handles these numbers of objects/rows much better > and fits because largely the data is relational in nature. Our general > concept is let's go with some kind of object relational mapping to make > the transition easier and more reuse of the exisitng code and then > if/when/where required we can always do direct SQL to make things more > efficient. Relational-object mapping also makes it easier and more natural to use a lot (but not all) of the Zope3 technologies as a lot of them are object centric. -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From jinty at web.de Fri Apr 7 02:31:16 2006 From: jinty at web.de (Brian Sutherland) Date: Fri, 7 Apr 2006 02:31:16 +0200 Subject: [Z3-sqlos] Removing those last un-tested modules Message-ID: <20060407003116.GB9793@minipas.home> Hi all, There are still 3 completely un-tested modules in sqlos, they seem _very_ out of date: sqlos.auth sqlos.annotations sqlos.attributeannotations Is there anyone willing to step forward and write some tests/maintain these? If not, I will remove them before cutting a 0.2 release in a few days. -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From jinty at web.de Thu Apr 13 21:00:27 2006 From: jinty at web.de (Brian Sutherland) Date: Thu, 13 Apr 2006 21:00:27 +0200 Subject: [Z3-sqlos] sqlos 0.2.0 released Message-ID: <20060413190027.GB11439@minipas.home> Hi, I found the time to make a 0.2.0 release of sqlos and have uploaded it to the python cheeseshop. This release is compatible with Zope 3.2 and SQLObject 0.7. This is the fist release in egg form, so probably there are some issues with that. It can be downloaded from: http://cheeseshop.python.org/pypi/sqlos/0.2.0 -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From jinty at web.de Tue Apr 18 18:51:02 2006 From: jinty at web.de (Brian Sutherland) Date: Tue, 18 Apr 2006 18:51:02 +0200 Subject: [Z3-sqlos] next steps for SQLOS Message-ID: <20060418165102.GA30431@minipas.home> Some planned features we discussed on IRC: Firstly, there seems to be a use case for a public method to sync all dirty sqlobjects. This would be useful in two cases I see: * You are using some traditional SQL and want your changes to SQLObjects to be reflected in the database. * You want .count() to work reflect recent changes but still want to use lazyUpdate. (I see this as a failing in SQLObject, but anyway) This method would be implemented as .syncAllDirty() in a module. Secondly, these days there is sqlobject2 on the horizon and a zalchemy project which uses code very similar to that in sqlos. Eventually having all these in one place would bring advantages in code re-use and bringing the users of zope+relational object mappers into one place. As the first baby step in this direction I propose to slowly start moving sqlobject specific code into a sqlos.zsqlobject by: * moving sqlos._sqlos to sqlos.zsqlobject * re-document "from sqlos import SQLOS" to "from sqlos.zsqlobject import SQLOS" Comments? -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!"