[z3-five] unicodes and strings in Zope 2.9's ZPT with Zope 3's i18n

[Roman Susi]

Hi!

Philipp von Weitershausen wrote:
> Chris Withers wrote:
> 
>>Chris Withers wrote:
>>
>>>Where in the ZPT/publishing stack do the unicodes get encode to strings 
>>>then?
>>
>>Answer: probably in the publisher somewhere.
> 
> 
> Yes. In an ideal case (read: Zope 3), the application returns text as
> unicode and never has to bother with strings. The publisher will
> negotiate the best encoding with the browser and send 8 bits over the wire.

We had the same problems too. And what makes things worse is that some
legacy pieces of software (in our case it was ZopeTree) assume str, no
Unicode. Even exportimport fails in our project due to the facts strings
are stored in Unicode! (Zope 2.9)

So, it is good advice "not to mix", but it is not always clear where
things are coming from in reality.

>>Since Zope 3's i18n stuff returns unicode, you need to be ultra careful 
>>that the rest of your ZPT generates either 7-bit ascii or unicode.
> 
> 
> Right. Combining 8-bit strings and unicode can lead to terrible errors.
> I wish Python hadn't gone down that route of making strings and unicode
> objects compatible. Then it'd be crystal clear when you're doing
> something evil...
> 
> 
>>In my case, I was inserting the utf-8 encoded title of a document into a 
>>generated navigator...
> 
> 
> I see. When in doubt, just make sure that anything ending up in a ZPT is
> unicode. Of course, ideally, those document titles would be stored not
> in UTF-8 but in unicode...

We use this util function to minimize the textual bloat explicit unicode
conversions bring:

def u(s, encoding="utf-8"):
    """Convert from UTF8 to Unicode (if needed)"""
    try:
        return unicode(s, encoding)
    except TypeError:  # Already Unicode?
        return unicode(s)



> 
> Philipp

Regards,
Roman