From jinty at web.de Thu Dec 8 14:09:36 2005 From: jinty at web.de (Brian Sutherland) Date: Thu, 8 Dec 2005 14:09:36 +0100 Subject: [Z3-sqlos] Re-organizing SQLOS code. In-Reply-To: <20051126154016.GB4288@cotia> References: <20051126120247.GA8624@minipas.home> <20051126154016.GB4288@cotia> Message-ID: <20051208130936.GA13778@minipas.home> On Sat, Nov 26, 2005 at 01:40:16PM -0200, Sidnei da Silva wrote: > On Sat, Nov 26, 2005 at 01:02:47PM +0100, Brian Sutherland wrote: > | 3. In quite a few package we have the form container/__init__.py where > | __init__.py is the only file and is quite small. I want to move these > | to the form container.py. It is not likely that these modules will > | grow too much in the near future. this doesn't work right with transactions/__init__.py, because it shadows the top level transactions module. Andres and I thought to move it to sqlos._transactions.py. Private as it is very sqlos specific and probably not usable anywhere else. Any objections if we do this without backwards compatibility? -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From jinty at web.de Fri Dec 9 00:07:00 2005 From: jinty at web.de (Brian Sutherland) Date: Fri, 9 Dec 2005 00:07:00 +0100 Subject: [Z3-sqlos] RFC: Transaction simplification branch Message-ID: <20051208230700.GA19697@minipas.home> Hi, I just cut a new branch with some major simplifications to the transaction machinery of sqlos. But these are _big_, so I would like to ask for testing/comments before I go further. Perhaps I went too far? Or cut something necessary? The branch is at: http://codespeak.net/svn/z3/sqlos/branch/jinty-simplify-transactions The main changes are: * Call sync as a pre-commit hook. (kudos to Jim, http://www.zope.org/Collectors/Zope3-dev/511) * don't syncUpdate anymore on commit or sync on prepare (It was already done in the pre-commit) * Don't force the zope.app.rdb data manager to join the transaction. (Thanks to the pre-commit hook this is not necessary) * Rip out the testing support code in the transaction module and re-write the unit test in a weaker but less brittle way. We test results now, not process. * Make self.objects self._objects. * Rip out sanity checking code checking that the transaction manager always dealt with the same transaction. * The Transaction Manager doesn't take the sqlconnection as an argument anymore as that has become un-necessary with the other changes. * Get rid of the events module. - The modified subscriber disappeared as we don't call it anymore. - The expired event was hardwired into _transaction.py Rationale: When you are committing/aborting a transaction you don't really want to be firing events. This makes it very easy for people to hook in exception raising code which can make a serious mess of things... Also it was just too many layers of confusion for what was needed. -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From sidnei at awkly.org Fri Dec 9 00:42:48 2005 From: sidnei at awkly.org (Sidnei da Silva) Date: Thu, 8 Dec 2005 21:42:48 -0200 Subject: [Z3-sqlos] RFC: Transaction simplification branch In-Reply-To: <20051208230700.GA19697@minipas.home> References: <20051208230700.GA19697@minipas.home> Message-ID: <20051208234248.GB3993@cotia> On Fri, Dec 09, 2005 at 12:07:00AM +0100, Brian Sutherland wrote: | Hi, | | I just cut a new branch with some major simplifications to the | transaction machinery of sqlos. But these are _big_, so I would like to | ask for testing/comments before I go further. | | Perhaps I went too far? Or cut something necessary? | | The branch is at: | http://codespeak.net/svn/z3/sqlos/branch/jinty-simplify-transactions | | The main changes are: | | * Call sync as a pre-commit hook. | (kudos to Jim, http://www.zope.org/Collectors/Zope3-dev/511) | | * don't syncUpdate anymore on commit or sync on prepare | (It was already done in the pre-commit) | | * Don't force the zope.app.rdb data manager to join the transaction. | (Thanks to the pre-commit hook this is not necessary) | | * Rip out the testing support code in the transaction module and re-write | the unit test in a weaker but less brittle way. We test results now, | not process. | | * Make self.objects self._objects. | | * Rip out sanity checking code checking that the transaction manager | always dealt with the same transaction. | | * The Transaction Manager doesn't take the sqlconnection as an argument | anymore as that has become un-necessary with the other changes. | | * Get rid of the events module. | - The modified subscriber disappeared as we don't call it anymore. | - The expired event was hardwired into _transaction.py | | Rationale: When you are committing/aborting a transaction you don't | really want to be firing events. This makes it very easy for people | to hook in exception raising code which can make a serious mess of | things... | | Also it was just too many layers of confusion for what was needed. It looks like a lot of changes indeed, and from the description they seem very sane. The only two issues that I can recall that need testing are: 1. changes that are not commited (eg, abort() was called) shouldn't live past transaction boundaries. need to be specially carefull about changes that stick on the cache. if I recall that's the reason syncUpdate was called. 2. all changes done must be successfully commited at commit(). If that works, I'm happy. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com From jinty at web.de Fri Dec 9 02:03:43 2005 From: jinty at web.de (Brian Sutherland) Date: Fri, 9 Dec 2005 02:03:43 +0100 Subject: [Z3-sqlos] RFC: Transaction simplification branch In-Reply-To: <20051208234248.GB3993@cotia> References: <20051208230700.GA19697@minipas.home> <20051208234248.GB3993@cotia> Message-ID: <20051209010343.GB19697@minipas.home> On Thu, Dec 08, 2005 at 09:42:48PM -0200, Sidnei da Silva wrote: > It looks like a lot of changes indeed, and from the description they > seem very sane. Yeah, after the first one everything just started to unravel. There are 3 un-resolved issues: * Why do we expire objects in a successful transaction? I can't think of a reason why we need to. * Why is expireSQLObject (the old expired event) so complex??? * expireSQLObeject must never generate any SQL. I'm pretty sure it does... > The only two issues that I can recall that need > testing are: Is this test rigorous enough for what you mean? http://codespeak.net/svn/z3/sqlos/trunk/src/sqlos/ftests/test_transaction.py > 1. changes that are not commited (eg, abort() was called) shouldn't > live past transaction boundaries. need to be specially carefull > about changes that stick on the cache. if I recall that's the > reason syncUpdate was called. For an aborting transaction this is how it goes now: 1 sync is not called on any object (this absence is tested) 2 expireSQLObject is called on all dirty SQLObjects 2 if any SQL was generated, rollback is called on the SQL connection via zope.app.rdb > 2. all changes done must be successfully commited at commit(). For a committing transaction this is what happens: 1 sync is called on all dirty objects 2 expireSQLObject is called on all dirty SQLObjects 2 if any SQL was generated, commit is called on the SQL connection via zope.app.rdb -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From jinty at web.de Fri Dec 9 04:38:34 2005 From: jinty at web.de (Brian Sutherland) Date: Fri, 9 Dec 2005 04:38:34 +0100 Subject: [Z3-sqlos] RFC: Transaction simplification branch In-Reply-To: <20051209015116.GE3993@cotia> References: <20051208230700.GA19697@minipas.home> <20051208234248.GB3993@cotia> <20051209010343.GB19697@minipas.home> <20051209015116.GE3993@cotia> Message-ID: <20051209033834.GC19697@minipas.home> On Thu, Dec 08, 2005 at 11:51:16PM -0200, Sidnei da Silva wrote: > On Fri, Dec 09, 2005 at 02:03:43AM +0100, Brian Sutherland wrote: > | On Thu, Dec 08, 2005 at 09:42:48PM -0200, Sidnei da Silva wrote: > | There are 3 un-resolved issues: > | > | * Why do we expire objects in a successful transaction? I can't think > | of a reason why we need to. > | * Why is expireSQLObject (the old expired event) so complex??? > | * expireSQLObeject must never generate any SQL. I'm pretty sure it does... > > I would need to look at those. Off the top of my head, I think we only > need to expire 'dirty' objects. me too, and any object that has had syncUpdate called is not dirty. So I removed the expiring after a successful transaction on my branch. see below for the resolution of the other issues. Anyway, I'm ready to unleash this branch on the world, i.e. merge it to trunk. Anyone with objections? > | > 1. changes that are not commited (eg, abort() was called) shouldn't > | > live past transaction boundaries. need to be specially carefull > | > about changes that stick on the cache. if I recall that's the > | > reason syncUpdate was called. > | > | For an aborting transaction this is how it goes now: > | > | 1 sync is not called on any object (this absence is tested) > | 2 expireSQLObject is called on all dirty SQLObjects > | 2 if any SQL was generated, rollback is called on the SQL connection > | via zope.app.rdb > > Doesn't sync need to be called to fetch the previous values from the > database? Or just expires is good enough? Sadly no, Andres went with a magnifying glass through SQLObject and found that we need to set _SO_createValues as well. We think this is a SQLObject bug. There is a now a regression test for it on the branch. expireSQLObject now looks like this: def expireSQLObject(obj): """Expire the SQLObject. Try to make sure none of the data makes it to the next transaction. This function should not cause SQL to be sent as it is not defined whether the SQL connection will commit before or after this is executed. """ for connection in connCache.values(): connection.cache.expire(obj.id, obj.__class__) # Expire object values. The transaction has either been aborted or # committed. obj.expire() # Calling sync() would commit the changes because expire() # doesn't clear _SO_createValues. # XXX this is thought to be a workaround for a bug in sqlobject # and should hopefully go away soon unproxied = removeSecurityProxy(obj) unproxied._SO_createValues = {} -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From stuart.bishop at canonical.com Fri Dec 9 06:50:45 2005 From: stuart.bishop at canonical.com (Stuart Bishop) Date: Fri, 09 Dec 2005 12:50:45 +0700 Subject: [Z3-sqlos] RFC: Transaction simplification branch In-Reply-To: <20051209033834.GC19697@minipas.home> References: <20051208230700.GA19697@minipas.home> <20051208234248.GB3993@cotia> <20051209010343.GB19697@minipas.home> <20051209015116.GE3993@cotia> <20051209033834.GC19697@minipas.home> Message-ID: <43991B35.7060504@canonical.com> >>| * Why do we expire objects in a successful transaction? I can't think >>| of a reason why we need to. You need to expire objects after a successful transaction so you can see database changes made by other processes or threads. This is particularly important for Zope where you have a number of handler threads and possibly multiple application servers in a ZEO environment. -- Stuart Bishop http://www.canonical.com/ Canonical Ltd. http://www.ubuntu.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/z3-sqlos/attachments/20051209/6616df7e/attachment.pgp From jinty at web.de Fri Dec 9 12:32:29 2005 From: jinty at web.de (Brian Sutherland) Date: Fri, 9 Dec 2005 12:32:29 +0100 Subject: [Z3-sqlos] RFC: Transaction simplification branch In-Reply-To: <43991B35.7060504@canonical.com> References: <20051208230700.GA19697@minipas.home> <20051208234248.GB3993@cotia> <20051209010343.GB19697@minipas.home> <20051209015116.GE3993@cotia> <20051209033834.GC19697@minipas.home> <43991B35.7060504@canonical.com> Message-ID: <20051209113229.GA7533@minipas.home> On Fri, Dec 09, 2005 at 12:50:45PM +0700, Stuart Bishop wrote: > > >>| * Why do we expire objects in a successful transaction? I can't think > >>| of a reason why we need to. > > You need to expire objects after a successful transaction so you can see > database changes made by other processes or threads. This is particularly > important for Zope where you have a number of handler threads and possibly > multiple application servers in a ZEO environment. Great, I reverted my patch and added a big warning because I don't want to imagine writing a test for that. But there is another issue with what you bring up, so far we only expire objects that have been changed during the transaction. By what you say, we should be expiring _every_ object on our thread. -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From stuart.bishop at canonical.com Fri Dec 9 12:54:51 2005 From: stuart.bishop at canonical.com (Stuart Bishop) Date: Fri, 09 Dec 2005 18:54:51 +0700 Subject: [Z3-sqlos] RFC: Transaction simplification branch In-Reply-To: <20051209113229.GA7533@minipas.home> References: <20051208230700.GA19697@minipas.home> <20051208234248.GB3993@cotia> <20051209010343.GB19697@minipas.home> <20051209015116.GE3993@cotia> <20051209033834.GC19697@minipas.home> <43991B35.7060504@canonical.com> <20051209113229.GA7533@minipas.home> Message-ID: <4399708B.7080203@canonical.com> Brian Sutherland wrote: > On Fri, Dec 09, 2005 at 12:50:45PM +0700, Stuart Bishop wrote: > >>>>| * Why do we expire objects in a successful transaction? I can't think >>>>| of a reason why we need to. >> >>You need to expire objects after a successful transaction so you can see >>database changes made by other processes or threads. This is particularly >>important for Zope where you have a number of handler threads and possibly >>multiple application servers in a ZEO environment. > > Great, I reverted my patch and added a big warning because I don't want > to imagine writing a test for that. > > But there is another issue with what you bring up, so far we only expire > objects that have been changed during the transaction. By what you say, > we should be expiring _every_ object on our thread. Yes - in our app we blow away the thread's entire cache at the transaction start (but our publisher and sqlos layers have been customised/lobotomised enough to not be generally useful to others). name = getUtility(IConnectionName).name key = (thread.get_ident(), name) cache = sqlos.connection.connCache if cache.has_key(key): del cache[key] -- Stuart Bishop http://www.canonical.com/ Canonical Ltd. http://www.ubuntu.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/z3-sqlos/attachments/20051209/7f7e4ab1/attachment.pgp From jinty at web.de Fri Dec 9 14:18:55 2005 From: jinty at web.de (Brian Sutherland) Date: Fri, 9 Dec 2005 14:18:55 +0100 Subject: [Z3-sqlos] RFC: Transaction simplification branch In-Reply-To: <4399708B.7080203@canonical.com> References: <20051208230700.GA19697@minipas.home> <20051208234248.GB3993@cotia> <20051209010343.GB19697@minipas.home> <20051209015116.GE3993@cotia> <20051209033834.GC19697@minipas.home> <43991B35.7060504@canonical.com> <20051209113229.GA7533@minipas.home> <4399708B.7080203@canonical.com> Message-ID: <20051209131855.GA10240@minipas.home> On Fri, Dec 09, 2005 at 06:54:51PM +0700, Stuart Bishop wrote: > Brian Sutherland wrote: > > On Fri, Dec 09, 2005 at 12:50:45PM +0700, Stuart Bishop wrote: > > > >>>>| * Why do we expire objects in a successful transaction? I can't think > >>>>| of a reason why we need to. > >> > >>You need to expire objects after a successful transaction so you can see > >>database changes made by other processes or threads. This is particularly > >>important for Zope where you have a number of handler threads and possibly > >>multiple application servers in a ZEO environment. > > > > Great, I reverted my patch and added a big warning because I don't want > > to imagine writing a test for that. > > > > But there is another issue with what you bring up, so far we only expire > > objects that have been changed during the transaction. By what you say, > > we should be expiring _every_ object on our thread. > > Yes - in our app we blow away the thread's entire cache at the transaction > start (but our publisher and sqlos layers have been customised/lobotomised > enough to not be generally useful to others). > > name = getUtility(IConnectionName).name > key = (thread.get_ident(), name) > cache = sqlos.connection.connCache > if cache.has_key(key): > del cache[key] Doesn't this leave the problem of objects you have a reference to in your app, but have not changed? Not very common I imagine... Anyway, thanks for the great suggestions/discussion, I marked it down in the TODO list until I have some more free time. -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!" From jinty at web.de Fri Dec 9 15:41:05 2005 From: jinty at web.de (Brian Sutherland) Date: Fri, 9 Dec 2005 15:41:05 +0100 Subject: [Z3-sqlos] sqlos and per site databases In-Reply-To: References: Message-ID: <20051209144105.GA11821@minipas.home> On Tue, Nov 01, 2005 at 01:51:29PM +0100, Andreas Elvers wrote: > Hi, > > I am having problems with sqlos and local utilities. I have a folder > which is a site. This site has a local utility which is a psycopg > adapter named as 'psycopg'. I have registered this name with sqlos. When > adding a SQLObjectContainer to this site folder my guess is, that this > container should use the db adapter 'psycopg'. > > While adding I get a "'NoneType' object is not callable" in > connection.py on line 96. This is because queryUtility really does not > find my local Utility although context point to the newly created > SQLObjectContainer and the utility name is currect either. Hi Andreas, I found you test for the issue in the issue tracker and committed it to the trunk. Though it is disabled until the issue is fixed. I can see at least two problems: * Sometimes the connection descriptor does not have the right context when querying the utilities. We could probably fix that on a case by case basis by getting the connection before the sqlobject call with the right context. Here is a sci-fi replacement of the container items() method: def items(self): """Return a sequence-like object containing tuples of the form (name, object) for the objects that appear in the folder. """ # we need to get the connection ourselves to avoid breaking # local # utilities that set the connection ut = zapi.getUtility(IConnectionName, context=self) connection = getConnection(self, ut.name) # iterate through the factories returning all the results for factoryName in self.allowedFactories(): factory = zapi.getUtility(IISQLObject, factoryName, context=self) for obj in factory.select(connection=connection): name = '%s.%s' % (factoryName, obj.id) yield (name, contained(obj, parent=self, name=name)) * The connection cache into sqlos.connection only can cope with one connection name per thread. This will seriously break if you do something like having two local connections with the same name. Unfortunately, I don't have the time to fix these now... -- Brian Sutherland Metropolis - "it's the first movie with a robot. And she's a woman. And she's EVIL!!"