[Z3-zemantic] Re: big zemantic storage / some changes

[Michel Pelletier]

On Sun, 2005-05-15 at 19:37 -0300, David Pratt wrote:
> Hi Michel.
> 
> Thank you for your replies to the thread.  This is very helpful.  Is 
> there interest in a sql backend for Zemantic?

Absolutely.

> The comments about large storage is real concern with the zodb only 
> option in my view when you think how much metadata can be collected per 
> record.

Keep in mind that all our ZODB scale experiments have been just that,
experiments.  Tarek is using a very experimental version of Zemantic
that includes a text index that index every literal value, so that
obviously adds a lot of weight that skews the impression of what a
"true" triple store should get.  In the next version of Zemantic (which
I'm working on a first sketch today) the Zope 3 catalog will resposible
for text indexing and other interpretive indexing.  Zemantic will go
back to being a straight RDF store.

>   My feeling is it could lead to big RAM being needed for the 
> application (this is unacceptable to me since not everyone can afford 
> beefy servers to make this work)  Second the repository for me is 
> something that I believe ought be able to be used more generally 
> instead of tied specifically to the zope application so that is 
> possible potentially for other applications to interact with it.  

I wouldn't gamble that ZODB uses much more memory than other databases,
but your second point is very good, a sql backend would be perfect for
applications from different frameworks that you want to share a store.

> My 
> hope is to use it as a container for other rdf data you wish to store 
> and query as well from other sources than just the zope application.

Zemantic works that way now, the RDF does not need to come from "inside"
Zope.

> The archetypes ids for me are not important since I don't work with 
> Plone but CMF due to GPL but I can see its potential role in Plone.   
> Let me know when you are starting this rewrite since I have a strong 
> interest in this and may have also tried some sql storage ideas by then.

Great, Dan and I have talked a bit about a SQL backend.  A plain SQL
backend for rdflib would be pretty easy, it would look something like
the existing sleepycat backend just with SQL commands instead of
berkeley.  The trickyh part is when you want to get a sql backend (and
the sleepycat one too) to work with Zope's (well, ZODB's) transaction
management.

Dan and I figured that backends that want to "play well" with Zope's
transactions need to implement some kind of interface that actually does
the database level commit and rollback and then a "wrapper" class that
interacts with Zope's TM.  It would look someting like

from rdflib import Graph
from mySQLBackend import SQLBackend
from zemantic import TransactionalBackend

g = Graph(TransactionalBackend(SQLBackend(param1=..., param2=...)))

TransactionalBackend would expect the SQL bckend to implement methods
like rollback() and commit() (see code in Zope for things like database
adapters and sessions to see how to interact with ZODB transactional
boundaries).

-Michel