[GH-ISSUE #1788] UnicodeDecodeError during stat generation #1409

New issue

Closed

opened 2026-02-27 11:16:57 +03:00 by kerem · 2 comments

kerem commented

2026-02-27 11:16:57 +03:00

Owner

Originally created by @tsabi on GitHub (Oct 31, 2019).
Original GitHub issue: https://github.com/modoboa/modoboa/issues/1788

Impacted versions

Standard Ubuntu 18.04 Server
Modoboa: 1.14.0 (latest)
installer used: Yes
Webserver: Nginx

Steps to reproduce

Today i saw there isn't any stat generated. When i runned the logparsing from command prompt i got this error:

root@h7-mailsrv:/etc/cron.d# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
Traceback (most recent call last):
  File "/srv/modoboa/instance/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle
    p.process()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process
    for line in self.f.readlines():
  File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc1 in position 949: invalid start byte

This modification fixed it:

diff -urN modoboa_stats_/management/commands/logparser.py modoboa_stats/management/commands/logparser.py
--- modoboa_stats_/management/commands/logparser.py     2019-10-31 11:53:10.899853660 +0100
+++ modoboa_stats/management/commands/logparser.py      2019-10-31 11:47:17.264635425 +0100
@@ -53,7 +53,7 @@
         self.debug = options["debug"]
         self.verbose = options["verbose"]
         try:
-            self.f = io.open(self.logfile, encoding="utf-8")
+            self.f = io.open(self.logfile, encoding="latin1")
         except IOError as errno:
             self._dprint("%s" % errno)
             sys.exit(1)

Originally created by @tsabi on GitHub (Oct 31, 2019). Original GitHub issue: https://github.com/modoboa/modoboa/issues/1788 # Impacted versions * Standard Ubuntu 18.04 Server * Modoboa: 1.14.0 (latest) * installer used: Yes * Webserver: Nginx # Steps to reproduce Today i saw there isn't any stat generated. When i runned the logparsing from command prompt i got this error: ``` root@h7-mailsrv:/etc/cron.d# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser Traceback (most recent call last): File "/srv/modoboa/instance/manage.py", line 22, in <module> execute_from_command_line(sys.argv) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line utility.execute() File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, **cmd_options) File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(*args, **options) File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle p.process() File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process for line in self.f.readlines(): File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf8' codec can't decode byte 0xc1 in position 949: invalid start byte ``` This modification fixed it: ```diff diff -urN modoboa_stats_/management/commands/logparser.py modoboa_stats/management/commands/logparser.py --- modoboa_stats_/management/commands/logparser.py 2019-10-31 11:53:10.899853660 +0100 +++ modoboa_stats/management/commands/logparser.py 2019-10-31 11:47:17.264635425 +0100 @@ -53,7 +53,7 @@ self.debug = options["debug"] self.verbose = options["verbose"] try: - self.f = io.open(self.logfile, encoding="utf-8") + self.f = io.open(self.logfile, encoding="latin1") except IOError as errno: self._dprint("%s" % errno) sys.exit(1) ```

kerem closed this issue

2026-02-27 11:16:58 +03:00

kerem commented

2026-02-27 11:16:58 +03:00

Author

Owner

@Salamek commented on GitHub (Nov 11, 2019):

@tsabi duplicity of https://github.com/modoboa/modoboa-stats/issues/55

@Salamek commented on GitHub (Nov 11, 2019): @tsabi duplicity of https://github.com/modoboa/modoboa-stats/issues/55

kerem commented

2026-02-27 11:16:58 +03:00

Author

Owner

@tsabi commented on GitHub (Nov 23, 2019):

@tonioo Please consider my fix: at the other ticket they tried to play with character replacement, what they needed if they read the log file as utf-8 encoding. But because it is latin1 encoded it is completely wrong way, the real fix is to read the file as it encoded originally.

@tsabi commented on GitHub (Nov 23, 2019): @tonioo Please consider my fix: at the other ticket they tried to play with character replacement, what they needed if they read the log file as utf-8 encoding. But because it is latin1 encoded it is completely wrong way, the real fix is to read the file as it encoded originally.