[z3-five] UTF-8, Zope 2.9.x and MSIE

[Roman Susi]

Hi,

I have a problem using Zope 2.9.x
Products.Five.form ... EditView, AddView

- scandinavian letters (e.g. ä, ö) are erroneously stored in the object
as a byte representation of utf-8 instead of correct text. And this
happens only when browsing with MSIE. Also, there is no such problem
when handling forms without any fancy EditView, AddView (just in a
method), form enctype doesn't matter.


Here two POSTs by MSIE and Firefox:


Accept-Encoding: gzip, deflate
Accept-Language: ru
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/msword, */*
Cache-Control: no-cache
Connection: Keep-Alive
Content-Length: 1955
Content-Type: multipart/form-data;
boundary=---------------------------7d6271194014e
Cookie: tree-expansion="xyxx"; tree-expansion="xyxx"; __ginger_snap="xyxx"
Host: xyxx:9080
POST /xyxx/edit.html HTTP/1.1
Referer: http://xyxx.org:9080/xyxx/edit.html
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET
CLR 1.1.4322)


-----------------------------7d6271194014e

Content-Disposition: form-data; name="field.description"



dfdfgdfgdfgdfg sdfsdf ............

-----------------------------7d6271194014e



Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Connection: keep-alive
Content-Length: 1916
Content-Type: multipart/form-data;
boundary=---------------------------41184676334
Cookie: tree-expansion="xyxx"; tree-expansion="xyxx"; __ginger_snap="xyxx"
Host: xyxx:9080
Keep-Alive: 300
POST /xyxx/edit.html HTTP/1.1
Referer: http://xyxx/xyxx/edit.html
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.8.0.4)
Gecko/20060508 Firefox/1.5.0.4


-----------------------------41184676334

Content-Disposition: form-data; name="field.description"



dfdfgdfgdfgdfg sdfsdf ............

-----------------------------41184676334


(dots are for nonASCII, encoding is utf8)
The only simingly relevant difference is
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7
field issued by Firefox...


Publisher encoding is set to utf-8 in zope.conf.

If unicode string u contains broken data, then this small function
corrects the unicode string:


def fixMSIE(us):
    return unicode(eval(repr(us)[1:]), 'utf8')


Something somewhere in Zope makes wrong assumption about encoding of the
data (or does double decoding or...?)... Any ideas how to prevent this
madness from happening? google search has not produced any meaningful
results.

Thanks!

Regards,
Roman