[GH-ISSUE #410] Unicode strings with encoding declaration are not supported. #394

New issue

Closed

opened 2026-02-27 11:11:33 +03:00 by kerem · 3 comments

kerem commented

2026-02-27 11:11:33 +03:00

Owner

Originally created by @tonioo on GitHub (Dec 4, 2013).
Original GitHub issue: https://github.com/modoboa/modoboa/issues/410

Originally assigned to: @tonioo on GitHub.

Originally created by Antoine Nguyen on 2013-06-04T11:16:36Z

Some emails cannot be parsed by lxml due to the following error:

Unicode strings with encoding declaration are not supported.

Modoboa uses unicode strings but it appears it should stay with std strings...

Originally created by @tonioo on GitHub (Dec 4, 2013). Original GitHub issue: https://github.com/modoboa/modoboa/issues/410 Originally assigned to: @tonioo on GitHub. **Originally created by Antoine Nguyen on 2013-06-04T11:16:36Z** Some emails cannot be parsed by lxml due to the following error: _Unicode strings with encoding declaration are not supported._ Modoboa uses unicode strings but it appears it should stay with std strings...

kerem

2026-02-27 11:11:33 +03:00

closed this issue
added the
bug
label

kerem commented

2026-02-27 11:11:33 +03:00

Author

Owner

@tonioo commented on GitHub (Dec 4, 2013):

Posted by Simon Kern on 2013-07-30T23:31:07Z

Is this related to the following error? In general I found force_text (from django.utils.encoding) to be pretty helpful.
If so I fixed it like that:

from django.utils.encoding import force_text
....
....
....
class EmailAddress(object):
    def __init__(self, address):
#        self.fulladdress = u2u_decode.u2u_decode(address).strip("\r\t\n")
#test fix
        self.fulladdress = force_text(address).strip("\r\t\n")
        (self.name, self.address) = parseaddr(self.fulladdress)
        if self.name == "":
            self.fulladdress = self.address
        
    def __str__(self):
        return self.fulladdress

Traceback (most recent call last):

  File "/foo/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, _callback_args, *_callback_kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
    return view_func(request, _args, *_kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/decorators.py", line 19, in wrapped_f
    return f(request, _args, *_kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/views.py", line 497, in index
    response = globals()[action](request)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/views.py", line 337, in listmailbox
    return lst.render(request, request.session["pageid"])

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/email_listing.py", line 139, in render
    listing = self.fetch(request, page.id_start, page.id_stop)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/email_listing.py", line 123, in fetch
    nbelems=self.elems_per_page))

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 165, in **init**
    self.populate(rows)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 183, in populate
    c.transform(row, newcol, self)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 40, in transform
    col["value"] = table.parse(self.name, row[self.name])

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 45, in parse
    value = getattr(IMAPheader, "parse_%s" % header)(value)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 90, in parse_from
    return IMAPheader.parse_address(value, **kwargs)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 72, in parse_address
    addr = EmailAddress(value)

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/emailutils.py", line 17, in **init**
    self.fulladdress = u2u_decode.u2u_decode(address).strip("\r\t\n")

  File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/u2u_decode.py", line 31, in u2u_decode
    u = mre.sub(decode_mime, s)

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3: ordinal not in range(128)

@tonioo commented on GitHub (Dec 4, 2013): **Posted by Simon Kern on 2013-07-30T23:31:07Z** Is this related to the following error? In general I found force_text (from django.utils.encoding) to be pretty helpful. If so I fixed it like that: <pre> from django.utils.encoding import force_text .... .... .... class EmailAddress(object): def __init__(self, address): # self.fulladdress = u2u_decode.u2u_decode(address).strip("\r\t\n") #test fix self.fulladdress = force_text(address).strip("\r\t\n") (self.name, self.address) = parseaddr(self.fulladdress) if self.name == "": self.fulladdress = self.address def __str__(self): return self.fulladdress </pre> <pre> Traceback (most recent call last): File "/foo/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response response = callback(request, _callback_args, *_callback_kwargs) File "/foo/venv/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view return view_func(request, _args, *_kwargs) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/decorators.py", line 19, in wrapped_f return f(request, _args, *_kwargs) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/views.py", line 497, in index response = globals()[action](request) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/views.py", line 337, in listmailbox return lst.render(request, request.session["pageid"]) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/email_listing.py", line 139, in render listing = self.fetch(request, page.id_start, page.id_stop) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/email_listing.py", line 123, in fetch nbelems=self.elems_per_page)) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 165, in **init** self.populate(rows) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 183, in populate c.transform(row, newcol, self) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/tables.py", line 40, in transform col["value"] = table.parse(self.name, row[self.name]) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 45, in parse value = getattr(IMAPheader, "parse_%s" % header)(value) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 90, in parse_from return IMAPheader.parse_address(value, **kwargs) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/extensions/webmail/lib.py", line 72, in parse_address addr = EmailAddress(value) File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/emailutils.py", line 17, in **init** self.fulladdress = u2u_decode.u2u_decode(address).strip("\r\t\n") File "/foo/venv/local/lib/python2.7/site-packages/modoboa/lib/u2u_decode.py", line 31, in u2u_decode u = mre.sub(decode_mime, s) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3: ordinal not in range(128) <pre>

kerem commented

2026-02-27 11:11:33 +03:00

Author

Owner

@tonioo commented on GitHub (Dec 4, 2013):

Posted by Antoine Nguyen on 2013-08-01T19:43:41Z

Unfortunately, your proposal would not solve this issue. The problem I encounter is described here:

http://lxml.de/parsing.html#python-unicode-strings

I need to find a proper way to avoid sending unicode strings to lxml...

@tonioo commented on GitHub (Dec 4, 2013): **Posted by Antoine Nguyen on 2013-08-01T19:43:41Z** Unfortunately, your proposal would not solve this issue. The problem I encounter is described here: http://lxml.de/parsing.html#python-unicode-strings I need to find a proper way to avoid sending unicode strings to lxml...

kerem commented

2026-02-27 11:11:34 +03:00

Author

Owner

@tonioo commented on GitHub (Apr 30, 2015):

This issue was moved to modoboa/modoboa-webmail#8

@tonioo commented on GitHub (Apr 30, 2015): This issue was moved to modoboa/modoboa-webmail#8