[z3-five] Re: Extended characters in forms

[Philipp von Weitershausen]

Lennart Regebro wrote:
> On 5/11/05, Lennart Regebro <regebro at gmail.com> wrote:
> 
>>This does not work. If you have a character like ä in a form, the
>>widgets try to do unicode(input) on it and you get a
>>UnicodeDecodeError, 'ascii' codec can't decode byte 0xe5 in position
>>0: ordinal not in range(128).
>>
>>How to fix?
> 
> 
> No-one?
> 
> So far I have three ideas, none of the good:
> 1. Force people to change the default encoing in site.py.
> 2. Monkey-patch Zope3 to stop it from converting everything to unicode.
> 3. Do some magic so everything in these forms is converted to unicode
> with the charset encoding in the for request.

After finally having found some time to look at the issue myself, I have 
to add to this discussion:

* None of the three points above provide a sensible solution.

* We obviously want zope.app.form to work since we're going to be stuck 
with it for some time.

* zope.publisher automatically detects the encoding of incoming requests 
and decodes form values accordingly if they contain non-ASCII 
characters. ZPublisher apparently doesn't do that.

* The Zope 3 text widgets do a unicode() on the incoming value to ensure 
  that the value stored to the object really is a unicode object. Text 
widgets--as opposed to bytes widgets--represent unicode strings and 
since zope.publisher doesn't convert form values to unicode if the value 
doesn't contain non-ASCII characters, it has to do that.


So, one solution would be to only use Bytes in your schemas and hence 
store data in some encoding. This might be a certain but very limited 
solution and thus not practicable in general cases.

My suggestion is we re-implement the text widgets by subclassing the 
original ones and override the methods that do the unicode() conversion 
to do sane unicode decoding. I have done this in my sandbox and had to 
take a couple of hurdles doing so. I seem to be pretty close now. Edit 
forms work when I enter Chinese characters. Add forms unexplicably still 
have problems, but I think I'll come around those tomorrow or so. Once 
I'm done working out the solution, I'll explain it in detail. I'll also 
provide some functional doctests.

Needless to say I'm not impressed by Zope 2's unicode support.

Philipp