[z3-checkins] r29713 - z3/deliverance/branches/namespaced/doc

[paul at codespeak.net]

Author: paul
Date: Fri Jul  7 09:22:28 2006
New Revision: 29713

Added:
   z3/deliverance/branches/namespaced/doc/INSTALL.txt
   z3/deliverance/branches/namespaced/doc/INTRO.txt
   z3/deliverance/branches/namespaced/doc/NOTES.txt
   z3/deliverance/branches/namespaced/doc/NewWorld.txt
   z3/deliverance/branches/namespaced/doc/meldXREADME.txt
Log:
Need to rewrite all this stuff...for now, just putting it in one place

Added: z3/deliverance/branches/namespaced/doc/INSTALL.txt
==============================================================================
--- (empty file)
+++ z3/deliverance/branches/namespaced/doc/INSTALL.txt	Fri Jul  7 09:22:28 2006
@@ -0,0 +1,80 @@
+======================
+Setting up Deliverance
+======================
+
+The whole idea of Deliverance is that it *shouldn't* be a system.
+It's just a thin approach that leverages some serious work being done
+by others.  In this case, Deliverance gets most of its value from:
+
+  o Apache.  Don't fight it, love it!  Yeh, baby!
+
+  o mod_python for both handlers and filters.
+
+  o lxml and thus libxml2/libxslt.  This is 90% of the value
+  proposition.
+
+mod_python
+----------
+
+1) Grab mod_python.
+
+2) Make sure you can do this:
+
+  http://www.modpython.org/live/current/doc-html/inst-testing.html
+
+3) Make sure Apache can read/write to the Deliverance examples directory.
+
+
+lxml
+-----
+
+1) Get 0.6 or later of lxml:
+
+  http://codespeak.net/lxml
+
+2) Make sure you can run some of the tests.
+
+3) Make sure you install it using the same Python used in mod_python.
+   To confirm, put:
+
+  import lxml.etree
+
+...in the mptest.py module used in the mod_python Testing example.
+
+
+Deliverance
+-----------
+
+1) Put a line like this:
+
+   Include /Users/paul/projects/deliverance/examples/plonenet/etc/plonenet.conf
+
+...in your Apache http.conf file.
+
+2) Edit that plonenet.conf file:
+
+  a. Change the Alias and Location lines to something meaningful.
+
+  b. Makes sure the LoadModule points at the right mod_python.
+
+  c. Fix the PythonPath line.
+
+3) Restart Apache.
+
+4) Go to the URL you put for the Location in the conf file and add
+   "/echo123" on the URL.
+
+You should see the sample file (content/echo123.xml) with the plonenet
+theme.  If not, check the Apache error log for details.  Possibly the
+web page will return a useful error message.
+
+5) If you make changes to the theme file
+   (e.g. content/plonenettheme.xml), you currently have to "compile"
+   it manually:
+
+  a. cd to the plonenet/lib directory.
+
+  b. run sh ./compiletheme.sh
+
+  c. Restart Apache.
+

Added: z3/deliverance/branches/namespaced/doc/INTRO.txt
==============================================================================
--- (empty file)
+++ z3/deliverance/branches/namespaced/doc/INTRO.txt	Fri Jul  7 09:22:28 2006
@@ -0,0 +1,330 @@
+==============================
+Deliverance
+==============================
+Content deliver for CMS systems
+------------------------------
+
+*Note: Work in progress.  Some of the ideas, particulary the content
+ map, are in flux.*
+
+CMS systems, particularly in Zope, excel at the structured environment
+of content *production*.  This area places a strong emphasis on
+security, workflow, metadata, and other content services.
+
+For content *delivery* on public sites, though, some of this machinery
+is overkill.  The framework for content production has a nasty side
+effect of killing performance for content delivery, making reliability
+and debugging a challenge, and forcing other audiences (like web
+designers in charge of look-and-feel) to learn another way to do
+things.
+
+For this reason, many ECM packages make a formal distinction between
+content production and content delivery.
+
+Deliverance is a lightweight, semi-static system for content delivery
+of CMS resources.  It runs in mod_python, generating branded pages and
+navigation elements, giving high-performance throughput to anonymous
+visitors.  Its primary benefits:
+
+  o High performance
+
+  o Simple re-branding
+
+  o Trusted stack
+
+  o Extreme productivity
+
+It is focused on audiencces that want:
+
+  o Predictable delivery to anonymous visitors
+
+  o Some portion of an airgap (logical/physical) between the CMS and
+  the live site
+
+  o Integration with mainstream systems and technologies
+
+This document discusses how the system works, then revisits the
+benefits in detail.
+
+Overview
+---------
+
+Deliverance has two major parts:
+
+  o *Themes*.  These apply a consistent look-and-feel to content that
+  streams through Apache.  This content can be on the filesystem, in
+  Zope with mod_proxy, or using the other part of Deliverance.  In a
+  nutshell, a theme is an HTML file (plus the CSS, images, etc.)
+  containing boxes that get filled by content.
+
+  o *Content maps*.  A description of the content on a site, including
+  metadata and different organization schemes.  The content map also
+  has views that, inside Deliverance, can generate HTML for navigation
+  and other purposes.
+
+Each of these can be used without the other.
+
+
+How It Works
+-----------
+
+In a nutshell, Deliverance gets an XML map describing all the
+published content at a point in time.  It uses this map to draw
+navigation elements and issue HTTP requests for content of single
+resources.  Finally, a "theme" provides the HTML to return with named
+boxes to be filled by rules.
+
+Let's first introduce some major concepts, then walk a request through
+from start to finish, using these concepts.
+
+1) *Theme*.  Web designers don't want to learn anything new.  ZPT
+tried to embrace this, but by the time the ZPT developer has injected
+all the tal and refactored everything into macros, the web designer
+can't possible continue.
+
+A theme is the corporate identity for a site.  It is *not* a template,
+as it has zero stuff in it beyond HTML.
+
+A theme is created by saving the customer's home page and identifying
+the boxes to be replaced.  For example, <div id="sitemenu"> identifies
+a place where a generated menu should go.  Web designers are familiar
+with this, as CSS uses such selectors to apply style.
+
+Deliverance has a ruleset that does the merge between the theme file
+and the generated content.  In essense, this rule file says: "Find
+'site-menu' in the theme and replace its children with the generated
+contents with an id of 'generated-menu'."
+
+The ruleset is under the control of the integrator, who bridges the
+gap between what the CMS provides and what the UI designer makes
+available.  This is done, though, without touching anything on either
+side of the equation.
+
+This allows the theme to be applied to all pages, without touching the
+pages.  The theme engine uses XSLT (via lxml) to perform the merge.
+
+2) *Map*.  By design, content delivery is separated from content
+management.  In fact, this separation acts as insulation.  At various
+intervals, the CMS makes an "edition" or a snapshot of its contents,
+providing a map with metadata for all the contents that should be
+visible.  (In fact, the map could point at a certain *version* of a
+resource that should be visible.)
+
+This map is read by mod_python using lxml.  It serves two functions:
+
+a. *Navigation elements*.  We can draw site menus and trees without
+visiting the server.  Since these can be done in XSLT, not only is the
+performance very good, but we can draw many other kinds of pages.  For
+example, we can show all the contents modified in the last week, or
+all the contents in France.
+
+b. *Resource lookup*.  The map controls how an incoming, virtual URL
+gets mapped to a real resource.  This means the URL space can be
+placeless.  Content can appear in multiple places.  Equally, content
+from multiple CMSs, even multiple remote hosts, can be integrated into
+the same map.
+
+The identifier used for retrieving the content for the resource can be
+a normal GET or a more complicated QUERY_STRING or even xml-rpc kind
+of lookup.
+
+Note that the contents for many CMS resources are, in fact, very small
+amounts of data.  They could be cached inline in the content map and
+not looked up.  For frequent pages, this would provide a big win.
+
+Finally, some resources in the map might be virtual, meaning the page
+can be fully rendered in Deliverance.  For example, a URL to show all
+the content with the keyword of "CPS" can be serviced without a trip
+to the server.  All that is needed is an XSLT rule for generation.
+(Later, the XSLT could be eliminated with a Python extension function
+in lxml.)
+
+3) *Compilation*.  The goal is high performance.  There are certain
+aspects that never change between requests:
+
+a. The contents of the map.
+
+b. The theme and the rule file for merging.
+
+c. Site configuration, such as site menus.
+
+It makes no sense to re-parse DOMs and stylesheets on each request.
+Equally, it makes no sense to have a multi-stage pipeline when several
+parts never change.
+
+Deliverance gets a tremendous speedup by compiling the theme into a
+stylesheet.  It reads the XHTML file for the theme, identifies the
+nodes to be replaced, and generates an XSTL with xsl:value-of and
+xsl:apply-template statements in the right location.  Compilation also
+inlines the map data into the XSLT so it doesn't have to included
+later.
+
+Compilation thus gives two benefits:
+
+a.  You can re-brand stuff without learning XSLT and without touching
+the HTML of the theme file.
+
+b. Most of the work needed for per-request transformations is done on
+startup.  Specifically, we avoid the 50ms hit that the "identity
+transformation" pattern seemed to give.
+
+4) *Retrieval*.  mod_python has a Bobo-inspired publisher that walks
+the URL, traversing Python objects using a set of rules.
+
+Deliverance has a similar idea.  The URL provides an identifier into
+the map file to retrieve a map item.  The map item then gives
+instructions on how to find the content for the page and how to render
+it.
+
+In most cases, some Python code will be issued to retrieve a page from
+the CMS.  For this, a very stripped-down skin will be used in the CMS,
+or perhaps no skin at all.  For example, the URL in the map file might
+request the DAV view of the resource, thus giving just the data.  For
+CMF-based systems, this is a 10x speedup.
+
+In other cases, the map might point to a virtual page, as discussed
+above.
+
+The mapping provides some interesting possibilities for integration.
+First, Deliverance could leverage Apache's infrastructure for
+retrieval and caching.  Second, libxml2 has several Python extension
+facilities (XPath functions, custom resolvers) that allow the map to
+act as an integration facility.  Simply put some metadata on a map
+entry to make it look like a resource, with the actual retrieval being
+done with custom code.
+
+5) *Generation*.  Deliverance does not have a parser of any kind.  It
+uses XSLT to generate HTML.  As noted above, for important parts of
+usage, no XSLT knowledge is required.
+
+Using XSLT gives some benefits:
+
+  o Extremely optimized.
+
+  o Extremely documented.
+
+  o Rich tool chain.
+
+  o Maintenance burden belongs to others.
+
+XSLT has a negative reputation.  Thus, Deliverance works hard to allow
+people to avoid using it, except when they need something custom.  For
+example, navigation boxes don't have to be generated by XSLT, they
+could be in the HTML lookup up by the CMS and inserted into the theme.
+
+A Typical Request
+---------------
+
+With that background, how does Deliverance work, end-to-end?  The
+following section starts with an Apache restart, finishing with the
+last byte returned to the browser.
+
+*Note: This describes how things will be, not how they currently are.
+ lxml needs some more work for a couple of things mentioned herein.*
+
+First, Apache is started.  In the conf file, there is a section that
+maps part of the URL space to a mod_python handler.  This handler is
+part of Deliverance.
+
+When the handler module is imported, it performs some one-time
+optimizations on startup:
+
+a. Read the map file, the theme's XHTML, and the site configuration
+into XML DOMs.
+
+b. Read the "blank-compilerdoc" and the compiler stylesheet into a DOM
+and a processor, respectively.
+
+c. Merge everything from (a) into the blank-compilerdoc (later
+replaced by XInclude).
+
+d. Create a compiled theme processor by applying the compiler
+stylesheet against the blank-compilerdoc.  The output is, in fact,
+another XSLT stylesheet.  Namely, it is a "compiled" stylesheet, ready
+to be applied to each incoming request while doing the least amount of
+work needed.
+
+When a request comes in, Apache passes it off to the Python handler
+function in Deliverance.  The handler takes the relevant part of the
+URI and does an XPath lookup in the map, grabbing the node referenced
+by this URI fragment.  This map node contains instructions for the
+next two steps:
+
+  o Retrieve the contents.
+
+  o Format the contents.
+
+The handler then retrieves the X(H)TML for the contents and applies
+the compiled stylesheet.  The compiled stylesheet has a rule for
+handling anything unique about that resource type.
+
+The results are serialized and returned.
+
+Performance
+-----------
+
+Since much of the information needed for rendering a requests is
+statically contained in a specially-tuned, in-memory DOM, performance
+automatically gets a boost.  (This would be the same in Zope.)
+
+The use of XSLT, especially compiled into a well-tuned state, gives
+another big performance win.  Many operations, such as drawing a tree
+or site map, fit the XSLT pattern better than ZPT.  Also, libxslt is a
+much more actively developed project, used by 1,000x the number of
+people, than ZPT.
+
+Memory usage is likely to be an issue.  A content map with 400,000
+entries could occupy 150 Mb of real memory.  However:
+
+  o Few sites have 400,000 public resources.
+
+  o Those that do can afford a gigabyte of RAM.
+
+For requests that don't require a trip to the CMS, 130 requests/sec
+should be expected.
+
+Productivity
+-----------
+
+You can speed up a computer by buying a bigger box.  How do you speed
+up a programmer?  Unfortunately, Zope has accumulated layers and layer
+of idiosyncratic frameworks.  Some of this is hidden from the
+integrator and web designer, but some of it peeks through.
+
+Deliverance is a massive increase in UI productivity.  First and
+foremost, the entire UI can be developed outside of the CMS, using
+static models on disk.  As long as the CMS returns XML that looks the
+same as the sample documents and sample content map, everything should
+just work.
+
+Second, this approach gives multiple tools in the toolchain.  I like
+using Oxygen, a cheap but amazing XML/XSLT authoring environment.  I
+can edit the dynamic UI with files on disk, press a button, and see
+what it will look like when rendered.  If there is an error, I get a
+useful (non-ZPT!) error message, with the cursor sitting on the
+offending line.  I even get a stepwise debugger, where I can watch the
+output get rendered and set a breakpoint to see the evaluation
+context.
+
+Alternatively, someone can run an xsltproc command like this:
+
+xsltproc compiler.xsl blank-compiled.xml | xsltproc - ../tests/sampledoc1.xml
+
+...and see what the page will look like.  Finally, the simple Python
+scripts in Deliverance can be run from the command line to process
+real output in the map.  Each part of the process can be inspected to
+find the offending problem.
+
+More generally, the XML+XSLT approach is fundamentally easier.  In
+ZPT, the data model is exposed via baroque, undocumented APIs
+appearing in TAL expressions.  In XML, you just look at the file and
+visually see the data model.  XPath gives a wonderful, simple, but
+powerful way to manipulate the data model.  And although XSLT is
+baroque, so is the messy pile of deconstructed macros and slots
+appearing ad-hoc in most large-scale Zope apps.
+
+This approach gives other kinds of productivity.  For example, there
+are tons of books, and Google has an answer to every question you
+might have.  Why?  Because the installed base of XML and XSLT is four
+orders of magnitude higher than Zope+CMF+ZPT+[CPS/Plone/Silva].
+

Added: z3/deliverance/branches/namespaced/doc/NOTES.txt
==============================================================================
--- (empty file)
+++ z3/deliverance/branches/namespaced/doc/NOTES.txt	Fri Jul  7 09:22:28 2006
@@ -0,0 +1,57 @@
+==============================
+Notes during development
+==============================
+
+
+May 21
+------
+
+I think I'll make a fundamental change.  Right now I have some hairy
+Python code to work around the libxml2 features that don't appear in
+lxml: XInclude and EXSLT being the two big ones.  There is another
+part about the current approach that is bugging me: seeing the
+intermediate stages (for debugging) is painful.  Finally, the current
+approach is poorer in performance (althoug 120 requests/sec ain't too
+bad) than it could be.
+
+I thought about the current 'compile-theme.sh' which was a
+capitulation to the ideas above, for one part of the pipeline.  I now
+think I'll just bite the bullet and have a multi-line shell script
+that is run whenever any of the static stuff changes:
+
+  o The theme file.
+
+  o The theme rule file.
+
+  o The theme compiler stylesheet.
+
+  o The SXC spreadsheet with all the content.
+
+  o The static pages (home page, about us, whatever).
+
+  o The stylesheet with rules to render all the content in the
+  spreadsheet.
+
+Especially on the static pages, this will be a big win.  First, I can
+put the pages in the model, instead of hacking them into the XSLT.  I
+can do so without using lxml to add nodes to transformed content from
+the SXC, which was causing problems.
+
+I can also avoid a 'document()' call in the content rendering
+stylesheet.  This is not only a performance hit, but also some brittle
+complexity (mapping URLs to filenames on disk).
+
+I'd also get a really nice improvement in the integrity of the model,
+conceptually.  Currently, static pages exist outside the model, which
+ATM comes only from spreadsheet data.  At a minimum, the pages will be
+processed into the compiled model.  At a maximum, I might have the
+static pages in the spreadsheet itself!  (Only if the majority of the
+pages are simple enough for ReST.)
+
+Still, the biggest win will debuggability.  My brittle code for doing
+includes and multi-step XSLT compiling using lxml will be replaced by
+libxml2 command line functions.  Each stage of the output will be
+written to disk, for visual inspection.  And most importantly, I can
+sit in oXygen and hack away.
+
+

Added: z3/deliverance/branches/namespaced/doc/NewWorld.txt
==============================================================================
--- (empty file)
+++ z3/deliverance/branches/namespaced/doc/NewWorld.txt	Fri Jul  7 09:22:28 2006
@@ -0,0 +1,562 @@
+============================================================
+NewWorld Template Machinery
+============================================================
+------------------------------------------------------------
+Analysis of current skinning and proposal for a new approach
+------------------------------------------------------------
+
+
+Plone has accumulated a baroque stack of templating and skinning
+approaches with plans for adding more.
+
+This document analyzes the current issues and makes recommendations
+with working code for new approaches.  The specific focus is on how
+the UI consultant and the component consultant accomplish their jobs
+both individually and together, and throughout a project.
+
+
+Caveat
+------
+
+ZPT started with a core vision: improve workflow by letting HTML/XHTML
+authors work with their tools.  If you don't care about this vision,
+and think ZPT is for developers, then you won't care about this
+proposal.
+
+
+Summary
+-------
+
+  o The UI consultant's job needs to become more productive, more
+  joyful
+
+  o Plone needs a distinct, formal concept of themes
+
+
+Glossary
+--------
+
+  o UI consulting.  At a minimum, drawing pixels on browser screens.
+  Beyond that, collecting what needs to be done (computations,
+  business rules) as stubs to be completed by the component team.
+
+  o Theme.  The corporate identity artifacts.  HTML, CSS, JS, PNG. No
+  templating, no programming.
+
+  o OldWorld. ZPT, skins, view classes, FS templates and TTW stuff.
+
+  o NewWorld. This proposal.
+
+  o o-wrap template. The current approach to corporate id.
+  ``main_template.pt``, which provides stuff that gets pulled into
+  ``folder_contents.pt`` during evaluation.
+
+  o Main template.  o-wrap.
+
+  o Context template. ``folder_contents.pt`` etc.
+
+  o Dreamweaverish.  A plaeholder for any authoring tool, WYSIWYG or not,
+  that structurally processes XHTML and thus increases
+  productivity/quality.
+
+
+The Problem
+-----------
+
+Instead of leading off with a long-winded architecture discussion,
+let's just jump right into specific instances of problems.  These are
+issues Plone inherits from its Zope/CMF/AT/Plone stack.
+
+Here we go with a top-ten list of annoyances:
+
+1. Getting a consistent DOCTYPE.
+
+Our first example is small in scope but wonderfully illustrative of
+the challenge.
+
+Correct DOCTYPEs are critical to UI, as IE will trigger quirks mode
+unless all the pages in your site have the correct DOCTYPE.  This
+applies to any potential add-ons written by others.
+
+Like other Zope 2/3 systems, Plone's ZPTs have a challenge on this::
+
+  <metal:page define-macro="master"><metal:doctype define-slot="doctype"><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"></metal:doctype>
+  <metal:block define-slot="top_slot" />
+
+This is the top of `main_template.pt`, the "o-wrap" for Plone.  When a
+UI designer wants to structural change the look-and-feel of a Plone
+site, this template is their starting point.  ZPTs original goal was
+to look like normal HTML, and this certainly doesn't.
+
+Getting some static text at the top of each template is seemingly a
+small use case.  However, the architecture of ZPT makes this into a
+challenge with negative consequences on the subtemplates (e.g. you
+can't define top-level variables that influence the global macro.)
+
+Summary: DOCTYPEs are important.  Controlling them turns templates
+into something unusable by UI consultants.
+
+
+2. Validation.
+
+The previous problem was a small detail with large consequences.  This
+problem is larger in scale and philosophy.
+
+ZPTs original goal was simple: "Well-formed markup to please
+Dreamweaverish."  There's a flaw in this, though.
+
+DOCTYPEs tell parsers what language you are speaking.  Externally,
+they tell the browser what grammar to use.  Internally, they tell the
+UI consultant's editor what tool to use.
+
+ZPT uses the same file artifact for both purposes.  When the ZPT is
+rendered, the non-valid XHTML markup is processed and removed.  The
+result fulfills the contract of the DOCTYPE for external use.
+
+However, when Dreamweaverish opens the file, it sees a DOCTYPE for XHTML,
+then a bunch of tal: and metal: stuff that violates that markup.  When
+the UI consultant presses the "Validate" button, a hundred errors are
+produced.  Worst case, the tool's helpful features (autocomplete,
+etc.) get turned off due to non-conformance.
+
+What are the choices?
+
+  a. Decide validation and tools don't matter and make UI consultants
+  use vi. [wink]
+
+  b. Make a new DTD or XSD/RNG that includes the tal and metal
+  namespaces.  Unfortunately, this new declaration gets sent merrily
+  back to the browser, thus breaking the external contract.
+
+  c. Formally split the external stage of processing from the
+  internal.
+
+This proposal will describe (c).
+
+Ultimately, Plone needs to take a stance on the original ZPT premise.
+Does Plone advertise that templates should be usable in HTML tools
+when opened directly from disk?  If so, the current situation is
+unacceptible.  If not, then the productivity of UI consultants is
+going to be trial-and-error, reload-Zope-and-view.
+
+Summary: Validation is key to improving the quality and productivity
+of the UI consultant.  It is also an important part of UI consultant's
+tools.  The current processing approach conflates external validation
+and internal validation.  Split them.
+
+
+3. link and script in head.
+
+Systems like Plone have some site-wide resources (CSS and JS) that get
+referenced by every page.  These references are done by `<link>` and
+`<script>` elements in the `<head>`.
+
+Some pages have per-page needs for CSS and JS.  This is problematic.
+It requires a new block-level element in the context template, usually
+something like `<metal:block fill-slot="top_slot">`.  The average UI
+consultant, when confronted with this, will:
+
+  a. Wonder what in the world is a metal block.
+
+  b. Wonder why it is appearing in the head.
+
+  c. Get aggravated when the HTML page is no longer valid and none of
+  the CSS and JS is included when viewed from disk.
+
+This is worse for second-order macros like Kupu, that have to fill the
+slot of a slot.
+
+Summary: It is wrong to use in-page, non-HTML constructs to push
+markup from the context template to the o-wrap template.
+
+
+4. Corporate identity.
+
+I've written about this before for the Zope Site Themes.  I won't make
+the case again, but I'll describe the solution below.
+
+  http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/SiteThemes
+
+5. Macros and slots.
+
+Quick, what is the set of macros used in CMF?  Plone?  Zope 3?  What
+slots are available?  What custom macros and slots do you have in your
+consulting project?
+
+Zope 3 brought "explicit is better than implicit" to software.  The UI
+side needs the same thing.  But we need ideas on how to avoid
+activities that feel like programming on the UI consulting side, but
+yield the benefits of knowing what is needed and what is available.
+
+Additionally, the connection between the o-wrap and context template
+is too hard-wired, making things brittle.  Just like adapters prevent
+changes in one side from affecting the other, we need similar ideas.
+
+6. Customize button.
+
+The CMF skin system was originally designed to let UI consultants
+customize the UI without touching the artifacts they got from the
+"system".  Great idea, but with consequences:
+
+  o If you make a one-line markup change ``folder_contents.pt``, you
+  own it.  Any updates in the system's template won't be seen.
+
+  o Zope 3 might make this go away.
+
+
+7. Tedious instructions.
+
+In CMF/Plone templates, every `link`, `script`, `a`, `img`, etc.  As
+such, each of these gets a `tal:attributes="src:
+{portal_url}/foo.png"` added to it, even though the image src was
+stated on the actual src attribute.
+
+What happens?  First, nobody even bothers putting an actual src
+attribute, meaning the original premise of ZPT is thrown out the
+window.  Second, the poor UI designer is confronted with this weird
+construct and has to remember, from system to system, whether it is
+`portal_url` or some other magic chant.
+
+This is painful.  More importantly, it tosses out the original premise
+of ZPT, which then tosses out Dreamweaverish and the
+productivity/quality gains.
+
+Most importantly, this is just one example of a recurring problem:
+Zope's template processing model doesn't really parse markup.  It
+does, but internally as a black box that can't be influenced.
+
+What's needed is a processing model where the job of fixing portal
+URLs, *and similar jobs*, is moved out of the template, but is still
+controlled by the integrators.
+
+Summary: Rewriting attributes and similar constructs breaks
+productivity and highlights a recurring problem.
+
+8. Template metadata "resource forks".
+
+Since ZPTs aren't treated as documents, you can't put metadata in them
+for use in Zope/CMF.  To add metadata, developers have to manage a
+separate ".metadata" file.
+
+9. Cleanup.
+
+Let's say you wanted to cleanup the markup produced on your site.
+Perhaps the UI consultant wants to:
+
+  o Ensure common look, as described under "Corporate identity" above.
+
+  o Compress page size by removing space.
+
+  o Improve readability by indentation.
+
+  o Produce valid XHTML.
+
+The current templating system doesn't provide a place in the
+processing to intervene and take such action.
+
+10. Reload.
+
+In debug mode, template changes don't require a reload.  That's great!
+Except the official new way is to move all logic to a view class, and
+that doesn't get reloaded.  Add a new image via ZCML?  Reload.
+
+Such interruptions in "flow" are anathema in the new realm of agile
+frameworks.
+
+
+Values
+------
+
+Now that we have a list of headaches, let's step back and talk about
+the kinds of values that we want to accomplish for UI consulting in
+the future.
+
+1) Plone consulting should be UI driven.
+
+I think most would nod their head patriotically for such a statement.
+But the reality really isn't like this.  Most Plone consultant
+projects are component-driven.
+
+If we say this, let's talk about the environment used by UI consulting
+vs. component consulting.  The latter is moving towards testing,
+validation, doctests, etc.  The former is still a hit-or-miss,
+low-quality environment.  If the former is doing the driving...well,
+we want the driver to have the best quality if we hope to avoid
+accidents.
+
+We should analyze this claim.  And we should make sure that the
+highest value, beyond all others, is to make Plone projects driven by
+UI consulting.  Then, when we evaluate the headaches listed above, we
+simply choose not to accept reasons why the headaches must continue.
+
+Why do we want this?  The customer wants the UI.  They don't want the
+components.  Also, the UI consultants are better with customers.  When
+a business rule is captured in the "software", it should be expressed
+in artifacts used by UI consultants and *not* forked into new
+artifacts used by the component team.
+
+Finally, back to hijacking.  When the consultants "collaborate" with
+the UI team currently, the result is usually a mess:
+
+  - The ZPTs become unusable for Dreamweaverish nor browser preview.
+
+  - The templates are only meaningful when evaluated by the server,
+    meaning restarts, software dependencies, etc.
+
+  - When stuff breaks on the "trunk", the UI consultants are stuck.
+
+2) Quality.
+
+Let's make quality a part of UI consulting.  How can we work in a
+deliberate way and know about the quality of our UI work?
+
+3) Joy.
+
+The UI consultants have a job to do.  They are good at it and have a
+way to do it.  Let them do the work with fun and joy.  A baroque pile
+of "you just gotta figure it out" technologies is not joy.
+
+4) Flow.
+
+Other web frameworks talk about "flow": no interruptions, no alarms,
+no surprises.  Stuff makes sense.  Refactoring is fun.  You know where
+to find things.  Can Plone do better on this?
+
+
+Proposal
+--------
+
+We should continue talking about the kinds of things that drive UI
+consultants crazy.  Ways that we can move ahead of the competition.
+
+But we can also brainstorm ideas -- simple ones, wild ones -- that
+might improve things.  Maybe the wild ones will inspire the final
+ones.
+
+1) Themes.
+
+The Site Themes proposal (and related work) goes into detail on this.
+Plone 3 *must* split the corporate id part (themes) from the dynamic
+generation part (skins or templates).
+
+  o When you make a Plone site look like your own, you shouldn't touch
+  the same artifacts as the "software".
+
+  o You shouldn't see anything that looks like software
+  (metal:define-slot etc.)
+
+  o Your artifacts should still be well-formed and *valid* in both
+  your editor and browser
+
+...and a host of other reasons defined in the Site Thems URL:
+
+    http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/SiteThemes
+
+How might this work in practice?
+
+a. Save a pile of XHTML, CSS, JS, PNG to disk.
+
+b. Edit a rule file that tells Plone how to fill in boxes with dynamic
+stuff.
+
+c. Nothing in your HTML changes.
+
+d. The output of the ZPT etc. is the input to the theme.
+
+e. There are no macros or slots used to setup DOCTYPEs or specify
+which boxes to fill.
+
+f. You can run a 100 line command-line script to do the merging.  (Or
+a CGI script, or lots of other things.)
+
+g. You can analyze your pile of context templates to see if they have
+the identifiers needed for filling in boxes.
+
+h. If something breaks, the template still renders, but without the
+broken box getting filled.
+
+i. Dreamweaver, nxml-mode, etc. can still validate your corporate id.
+
+j. We can use off-the-shelf parsers etc. to implement the connection
+between context output and theme input.
+
+k. We don't have matching macro/slot declarations on each side that
+are easy to break.  The rule file controls what matches with what
+(probably with XPath).
+
+l. Based on the previous point, refactoring becomes fun again.
+
+m. The component people rejoice because their templates no longer have
+any connection to styling.  No slot filling, etc.
+
+
+2) New approaches to templating.
+
+Although I have code for the former, I'm still in progress on the
+following sections.  Since that makes it science fiction, I will
+provided bullets with "imagine if"-style explanations.
+
+a. Meaningful error messages.  Imagine that, as you worked on your UI
+consulting, mistakes (yours or others) produced useful error messages.
+Even better, imagine that you felt you could jump in and isolate the
+error.  Egads, even fix it.
+
+Also, imagine if the "system" was isolated from your part.  You know
+what "the system" is supposed to hand you.  You can prove if "the
+system" did or didn't, and thus, whether the problem is in the UI
+consulting layer.
+
+b. Meaningful templates. Imagine if the information in the template
+could be used by the system.  For example, a ``<link rel="next"
+href="foo.html"/>`` could be used by the view class or other parts of
+the processing chain.
+
+This point has a LOT of potential.  Once the template becomes a useful
+artifact, rather than tag soup parsed into a "proprietary" parse
+format, we can let the system's templates (written by competent
+component consultants) encode more meaning.
+
+c. What drew the pixel?  Imagine you could look at a rendered, dynamic
+page, and could find out what "instructions" were responsible for each
+pixel.  That alone would make NewWorld discussions worthwhile.
+
+d. Output hacking.  Imagine if you could affect every ``<a href="">``
+on the site and, in certain conditions, modify it.  Wouldn't that be
+great?  Imagine if you could add tag information to the output without
+even *touching* a single "system" artifact, such as a ZPT?
+
+e. Tool support.  Imagine if your template language had a schema.  You
+could use nxml-mode, oXygen, Dreamweaver, or a host of other tools
+that know how to tell if you screwed up your template.  Wouldn't that
+be better than visiting the browser, pressing reload, and deciphering
+a ZPT message?
+
+Really, this point is colossal.  People are long used to the benefits
+that smart code editors provide.  They provide vast improvements in
+both productivity and quality.  They can even help assist you in
+knowing what is possible in the context of something else -- even
+providing tooltips to explain choices.
+
+Beyond that, we can write tests and other tools that process the site
+and see what's going on.
+
+f. Push templates. (Courtesy of Tres.)  Imagine a system where a
+template doesn't pull information in by knowing the intricasies of the
+catalog API.  Instead, the view class prepares everything that is
+available and pushes it into the template.
+
+With this, the UI consultant could actually master some the Python
+code to assemble static Python dictionaries to be replaced later by
+the component consultants.
+
+g. Testing.  Imagine the ongoing process of UI consulting was
+testable.  Imagine that, while working on the templates and
+presentation logic, the UI consulting could leave some droppings that
+let made the UI testable.  Imagine this was done in a very natural
+way, from the perspective of the UI consultant.  It fits their brain,
+so they do it.
+
+h. Documentation.  Imagine the same thing were true for documentation.
+UI consultants deal with customers.  Customers like to know what is
+finished, in progress, and just started.  They like to know how the UI
+plugs together, what is the intent, etc.
+
+Components have lots of stuff now for this.  UI consultants have
+squat.  They generally draw boxes in Illustrator upfront, then
+immediately forget about it, as they become out-of-date.
+
+i. Business rules.  Imagine the UI consultant gets to "own" the
+conversation with the customer about business rules.  Archetypes
+provides schemas that express constraints in a reasonably-friendly
+fashion.  What more can be done like this, and how can this be done
+more in the UI consultant's domain?
+
+Why is this worth discussing?  Because the UI consultant is a better
+fit for managing this:
+
+  o Customers like pixels, not code
+
+  o UI consultants are usually more cuddly than component developers
+
+j. Optimization hints.  Imagine a templating model that could say,
+"This block doesn't change much."  For example, only on restart.  Or
+only when another thing changed.  Or etc.  Currently this is done by
+cache managers, that feel more like code.  Let the UI consultant talk
+it over with the customer and leave hints in the templates.
+
+*Note: This is a benefit of splitting out themes from templates.  We
+ can then make templates into more of a domain-specific language for
+ UI consulting without freaking out the Dreamweaver-ish people.*
+
+k. Forms and validation.  Long discussion, ask Doug Winter.
+
+
+3) Serverless UI consulting.
+
+I'm working on a desktop tool that lets you do server-ish things
+without a server.  If it pans out, it has a number of interesting
+possibilities for automating the business of UI consulting, while
+removing many of the headaches.
+
+a. Reload sucks.  Imagine you didn't have to do server restarts for
+anything related to UI consulting.  But the artifacts you created were
+still used by the component consultants (though not hijacked).
+
+b. Freedom.  Imagine you could develop a UI without caring about
+breakage by the component folks.  Without even installing a Zope.  You
+could sit in front of the customer, do work, and see 0.05 second
+response time without restarts.
+
+c. Validation.  Imagine a GUI that tells you when things need fixing.
+That's the beauty of validation and standards.  We can have a schema
+for all inputs, intermediary steps, and outputs of UI consulting.  We
+can even look at CSS rules, theme rule files, configure.zcml files,
+and more.
+
+d. Refactoring.  Want to change a "slot" from foo to bar?  Tell the
+tool and it will fix all the artifacts in your site.  Want to change a
+CSS class name?  Ditto.  Rename a template?  Find orphaned templates
+or images?
+
+e. Issue tracking.  Lots of code editors support leaving XXX comments,
+then seeing a list of what needs to be done.  ZPT?  Forget about it.
+Imagine you could do this in UI consulting.
+
+o lint-style stuff
+
+  - is input page well-formed? output?
+
+  - valid?
+
+  - where do boxes come from?
+
+  - any templates/resource not in ZCML?
+
+  - vice versa?
+
+  - list all macros/slots/exports in the site, w/ frequency counts
+
+  - list all classes/ids in the site
+
+
+
+
+Email Summary
+-------------
+
+1) This is mostly about identifying the problems.  It's only to
+   stimulate serious analysis about the productivity and joyfulness of
+   UI consulting.
+
+2) I'm willing to keep working on the proposal if people think this
+   class of problems should be talked about.
+
+3) This is meant to support, in several different, ways existing
+   systems like ZPT.
+
+4) It is also a desire to play around with completely new ideas that
+   can co-exist with old ones.
+
+5) I'm willing to have an implementation pronto.
+
+

Added: z3/deliverance/branches/namespaced/doc/meldXREADME.txt
==============================================================================
--- (empty file)
+++ z3/deliverance/branches/namespaced/doc/meldXREADME.txt	Fri Jul  7 09:22:28 2006
@@ -0,0 +1,81 @@
+========================================
+meldX, an XML-inspired version of meld3
+========================================
+
+About
+-----------
+
+meldX is the four thousandth templating system, except it's an
+un-system.  The goal is to impose a look-and-feel on content without
+touching either look-and-feel or the content.
+
+In a nutshell:
+
+.. image:: zopesitethemesdiagram.png
+
+Requirements
+------------
+
+meldX requires lxml.  It optionally requires uTidyLib if the theme
+input isn't well-formed XML.
+
+Quick Start
+-----------
+
+1. cd meldX
+
+2. python ./runpipeline.py
+
+3. Open var/finalresult.html in a browser.
+
+What happened?  The following:
+
+1. meldX retrieved the HTML content from http://www.plone.org/.
+
+2. An XSLT was created with plug points as defined in the rule file.
+
+3. This XSLT was then applied 20 times to a sample content doc.  The
+   resulting HTML looks like plone.org, but has the info contained in
+   ``contentdoc.xml``.  This document simulates what would come out of
+   the CMS.
+
+
+Less Quick Start
+----------------
+
+1. Edit rulefile.xml and change themeuri to point at some URL.
+
+2. Edit the themexpath to pick boxes to shove stuff into.
+
+3. python ./runpipeline.py
+
+4. Look at the output of each pipeline stage in the var directory.
+
+5. Open contentdoc.xml in a modern browser (IE, FF, Safari, new
+   Opera).  The XSLT will be applied in-browser.
+
+
+Background
+----------
+
+Chris McDonough has worked on meld3, a templating approach without
+templates.  meldX is similar:
+
+1) You don't put templating in the stuff the web designer sees.
+
+2) You don't write "parsers", but instead, use off-the-shelf XML
+   parsing tools.
+
+meldX is different in the following ways:
+
+1) Instead of sticking a meld:id attribute in the template, you use an
+   XPath expression to point at an id, a class, a child relationship,
+   whatever.
+
+2) You don't write Python glue code.  Instead, a rule file governs the
+   merging of content.
+
+meldX tries to fulfill the ideas explained in the `Zope Site Themes
+proposal <http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/SiteThemes>`_.
+(The content at this link is also in ``contentdoc.html``, the sample
+content in this directory.)