[kupu-dev] UnicodeDecodeError in kupu_migration.xml.pt

[Dorneles Treméa]

Hey Duncan, folks,

we hit a problem today: when trying to access the 'links' tab,
the following exception is raised:

> 2007-05-16 20:30:36 ERROR Zope.SiteErrorLog http://localhost:8080/test/kupu_library_tool/kupu_migration.xml
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 115, in publish
>   Module ZPublisher.mapply, line 88, in mapply
>   Module ZPublisher.Publish, line 41, in call_object
>   Module Shared.DC.Scripts.Bindings, line 311, in __call__
>   Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
>   Module Products.CMFCore.FSPageTemplate, line 195, in _exec
>   Module Products.CacheSetup.patch_cmf, line 18, in FSPT_pt_render
>   Module Products.CacheSetup.patch_utils, line 9, in call_pattern
>   Module Products.CMFCore.FSPageTemplate, line 134, in pt_render
>   Module Products.CacheSetup.patch_cmf, line 68, in PT_pt_render
>   Module Products.CacheSetup.patch_utils, line 9, in call_pattern
>   Module Products.PageTemplates.PageTemplate, line 104, in pt_render
>    - <FSPageTemplate at /test/kupu_migration.xml used for /test/kupu_library_tool>
>   Module TAL.TALInterpreter, line 238, in __call__
>   Module TAL.TALInterpreter, line 281, in interpret
>   Module TAL.TALInterpreter, line 457, in do_optTag_tal
>   Module TAL.TALInterpreter, line 442, in do_optTag
>   Module TAL.TALInterpreter, line 437, in no_tag
>   Module TAL.TALInterpreter, line 281, in interpret
>   Module TAL.TALInterpreter, line 715, in do_condition
>   Module TAL.TALInterpreter, line 281, in interpret
>   Module TAL.TALInterpreter, line 691, in do_loop_tal
>   Module TAL.TALInterpreter, line 281, in interpret
>   Module TAL.TALInterpreter, line 457, in do_optTag_tal
>   Module TAL.TALInterpreter, line 442, in do_optTag
>   Module TAL.TALInterpreter, line 437, in no_tag
>   Module TAL.TALInterpreter, line 281, in interpret
>   Module TAL.TALInterpreter, line 310, in do_startEndTag
>   Module TAL.TALInterpreter, line 339, in do_startTag
>   Module TAL.TALInterpreter, line 405, in attrAction_tal
>   Module Products.PageTemplates.TALES, line 227, in evaluateText
>   Module Products.PageTemplates.TALES, line 221, in evaluate
>    - URL: file:kupu/plone/kupu_plone_layer/kupu_migration.xml.pt
>    - Expression: string:u'${f/label}'
>    - Names:
>       {'container': <PloneSite at /test>,
>        'context': <PloneKupuLibraryTool at /test/kupu_library_tool>,
>        'default': <Products.PageTemplates.TALES.Default instance at 0xb7224b6c>,
>        'here': <PloneKupuLibraryTool at /test/kupu_library_tool>,
>        'loop': <Products.PageTemplates.TALES.SafeMapping object at 0xc99798c>,
>        'modules': <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at 0xb71c8c6c>,
>        'nothing': None,
>        'options': {'args': ()},
>        'repeat': <Products.PageTemplates.TALES.SafeMapping object at 0xc99798c>,
>        'request': <HTTPRequest, URL=http://localhost:8080/test/kupu_library_tool/kupu_migration.xml>,
>        'root': <Application at >,
>        'template': <FSPageTemplate at /test/kupu_migration.xml used for /test/kupu_library_tool>,
>        'traverse_subpath': [],
>        'user': <PropertiedUser 'dorneles'>}
>   Module Products.PageTemplates.Expressions, line 229, in __call__
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)

I tracked the problem down and it happens in this situation:

- you have a content-type with a field which is editable using
kupu (RichWidget, VisualWidget)

- that particular field has a given msgid for the widget label

- that msgid has a translation containing non-ascii chars in
some languages

Then, when you access the 'links' tab with a browser configured
to use one of those languages with has a non-ascii translation,
kupu tries to list all content-types and the respective available
fields. This is provided by the getKupuFields method from
kupu.plone.plonedrawers.py which returns a dictionary with these
keys: 'type', 'name', 'portal_type' and 'label'.

As the above traceback, the exception is caused by the 'f/label'
expression. The 'label' value is calculated with:

  f.widget.Label(self)

It returns an encoded string (not an unicode) and then, later,
when using it in the kupu_migration.xml, Zope internally seems
to try to interpolate the 'label' value to generate the proper
'value' attribute for the input tag:

<input type="hidden" tal:attributes="value f/label"
       name="fields.label:records" />

At that point, the 'value' label is a string containing
non-ascii chars encoded with the encoding defined in
portal_properties/site_properties/default_charset.

It will obviously fail when interpolated against any unicode
object.

The proposed fix for this issue is to transform the 'label'
value into an unicode again, as in:

  f.widget.Label(self).decode(site_encoding)

The site_encoding can be get via CMFPlone.utils.getSiteEncoding
method, this is the complete patch:

> Index: kupu/plone/plonedrawers.py
> ===================================================================
> --- kupu/plone/plonedrawers.py  (revisão 43446)
> +++ kupu/plone/plonedrawers.py  (cópia de trabalho)
> @@ -23,6 +23,7 @@
>  from Products.PythonScripts.standard import html_quote, newline_to_br
>  from Products.kupu.plone.librarytool import KupuError
>  from Products.CMFCore.utils import getToolByName
> +from Products.CMFPlone.utils import getSiteEncoding
>  import html2captioned
>  
>  try:
> @@ -762,9 +763,11 @@
>      def getKupuFields(self, filter=1):
>          """Returns a list of all kupu editable fields"""
>          inuse = getToolByName(self, 'portal_catalog').uniqueValuesFor('portal_type')
> +        site_encoding = getSiteEncoding(self)
>          for t,f,pt in self._getKupuFields():
>              if html2captioned.sanitize_portal_type(pt) in inuse or not filter:
> -                yield { 'type': t, 'name': f.getName(), 'label': f.widget.Label(self), 'portal_type':pt }
> +                yield dict(type=t, name=f.getName(), portal_type=pt,
> +                           label=f.widget.Label(self).decode(site_encoding))
>  
>      def _getKupuFields(self):
>          """Yield all fields which are editable using kupu"""

I can commit it, but I thought it would be nice to raise the
question here first.

Thoughts?

-- 
 ___________________________________________________________________

       Dorneles Treméa · Developer · Plone Solutions · Brazil

   Consulting · Training · Development · http://plonesolutions.com
 ___________________________________________________________________

  Plone Foundation · http://plone.org/foundation · Protecting Plone