mirror of
https://github.com/modoboa/modoboa.git
synced 2026-04-25 00:46:03 +03:00
[GH-ISSUE #3186] Mail stats not working in fresh install of 2.2.4 #1790
Labels
No labels
bug
bug
dependencies
design
documentation
duplicate
enhancement
enhancement
enhancement
feedback-needed
help-needed
help-needed
installer
invalid
looking-for-sponsors
modoboa-contacts
new-ui
new-ui
pr
pull-request
pyconfr
python
question
security
stale
webmail
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/modoboa-modoboa#1790
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @pappastech on GitHub (Feb 9, 2024).
Original GitHub issue: https://github.com/modoboa/modoboa/issues/3186
Steps to reproduce
Install modoboa with installer on Debian 12.4, configure domain(s), identities, get e-mail flowing, look at stats on new-admin and they're empty. A review of the /srv/modoboa/rrdfiles shows that only the "new_accounts.rrd" file is being created.
Current behavior
Only the new_accounts.rrd file is being created as part of the statistics cron job.
rrdfile directory:

Manually trying to run the logparser results in the following:

Expected behavior
More stats rrd files should be created including an rrd file for postfix log stats from the configured postfix log file (currently /var/log/mail.log)
@pappastech commented on GitHub (Feb 18, 2024):
Does anyone have any advice on this issue?
@tonioo commented on GitHub (Feb 21, 2024):
@pappastech Do you have a lot of traffic on this server?
@pappastech commented on GitHub (Feb 21, 2024):
Not sure on your definition of "a lot of traffic". The average is usually 20-100 e-mails per hour.
@tonioo commented on GitHub (Feb 22, 2024):
@pappastech The log parser is not very efficient since it analyses the complete file every time it runs (so every 5 mins by default).
You should check how much time is required for it to complete and then adjust the frequency.
@pappastech commented on GitHub (Feb 22, 2024):
I've run the log parser manually and it doesn't take more than 15-20 seconds. The problem is that it's not generating the rrd file at all. My guess is that it's running into an error somewhere in the processing, but doesn't log any errors itself, so I'm not sure where it's going sideways.
As a separate issue, none of the MX, DKIM, DMARC, etc. checks are working correctly at the moment either. In other words, I have all those records created in DNS, but the process to check them somehow doesn't see those records. There's a general lack of things working and it's difficult for me to troubleshoot since the cron tasks aren't logging output anywhere.
@pappastech commented on GitHub (Feb 27, 2024):
I did figure out the MX, DKIM record checks. SEPM running on the Hyper-V host was causing that problem. I'm still not seeing log parsing working though; any ideas?
@layer3-fl commented on GitHub (Mar 20, 2024):
I have something similar happening on a fresh install of 2.2.4 on top of fresh install of Debian 12. This is something I happened to notice, I don't think the graphs are something I would look at frequently.
root@mail:/home/matto# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
/srv/modoboa/env/lib/python3.11/site-packages/modoboa/maillog/management/commands/logparser.py:97: FutureWarning: Possible nested set at position 29
self._regex = {k: re.compile(v) for k, v in self._regex.items()}
drwxrwx--- 2 modoboa modoboa 4096 Mar 17 08:00 .
drwxr-xr-x 7 modoboa modoboa 4096 Mar 17 07:06 ..
-rw-r--r-- 1 root root 2448 Mar 20 18:00 new_accounts.rrd
My rrdfiles directory is owned by modoboa:modoboa but the present new_accounts.rrd is owned by root where as pappastech's new_accounts.rrd is owned by modoboa. The new accounts graph is working on my setup.
Does python version play any relevance?
root@mail:/home/matto# /srv/modoboa/env/bin/python -V
Python 3.11.2
@tonioo commented on GitHub (Mar 21, 2024):
@layer3-fl And you confirm there is mail traffic on your server?
@layer3-fl commented on GitHub (Mar 21, 2024):
@tonioo That is correct. I can also confirm my log file is /var/log/mail.log and I have not made any changes to the statistics parameters in the GUI.
@mas1701 commented on GitHub (Jul 9, 2024):
Adding to this, I have the following cronjobs on Modoboa 2.2.4 on deb12 (upgraded from 1.17 on deb10, completely gotten rid of Python2, now Python 3.11):
Running the first one manually:
And the second one:
rrd files:
After the system and modoboa upgrades, there were no more mail stats. And yes, there is some mail traffic on the system.
@tonioo commented on GitHub (Jul 10, 2024):
@mas1701 Have you checked both old and new UIs ?
@mas1701 commented on GitHub (Jul 10, 2024):
Hi tonioo,
since it does not seem production-ready yet (incomplete translations), I'm using the old UI only. Don't tell me it's deactivated there now?
I just checked the rrdfiles directory again. new_accounts.rrd gets updated, other timestamps unchanged. And it does not surprise me at all looking at the output of "update_statistics". It is also unchanged, many hours later. It's like a cat chasing it's own tail, it doesn't get anywhere.
I also still have the files of the old install in /srv/modoboa.old/rrdfiles. All the same user/group/rights settings, updated frequently when still in use.
@tonioo commented on GitHub (Jul 10, 2024):
No it should work on both versions but it could give hints. Could you please check the new UI ?
@mas1701 commented on GitHub (Jul 11, 2024):
I just re-enabled the new UI and had a look at it. Makes no difference, stats end at the same moment as with the old UI.
@tonioo commented on GitHub (Jul 11, 2024):
And they stopped since the upgrade to 2.2.4 ?
@mas1701 commented on GitHub (Jul 11, 2024):
Yes. I upgraded from Debian 10 to Debian 12 and Modoboa 1.17 to 2.2.4, following instructions on how to re-build the Python virtual environment (for Python 3.11, Python 2 remains from Debian 10 completely removed) and worked through the version-specific upgrade instructions as well. After all updates, the "migrate" job was run.
There is a time difference between the actual update (I had the system shielded from newly arriving mail during the update) and the first, newly processed e-mail, but that was caused by a Postfix/Amavis problem I encountered in Debian 12. It is fixed. It would explain a gap of some hours in the stats, but not a permanent gap.
I mostly wonder about that timestamping (?) message "rrdtool.OperationalError: /srv/modoboa/rrdfiles/new_accounts.rrd: illegal attempt to update using time 1720555200 when last update time is 1720555200 (minimum one second step)".
That is from the evening right after the upgrades, when I was doing some checks (after fixing the Postfix/Amavis problem): Tue Jul 09 2024 22:00:00 GMT+0200
I just copied rrdfiles to rrdfiles.bak and emptied the rrdfiles directory with the exception of new_accounts.rrd, because that was fresh. I ran the logparser and generate_statistics jobs again and got the same error, but with a newer timestamp. No change inside of Modoboa's UI though, just more empty space, as the old stats are moving fuirther to the left.
New timestamp from error: 1720724400 (Thu Jul 11 2024 21:00:00 GMT+0200)
That matches new_accounts.rrd with an offset of exactly 2 hrs. - maybe the GMT+0200 offset applied twice.
For now, I reverted to the old rrdfiles, but also kept the new ones.
@mas1701 commented on GitHub (Jul 17, 2024):
tonioo,
is there anything new regarding above issue? Do you need additional information?
@mas1701 commented on GitHub (Jul 31, 2024):
Antoine,
I don't mean to push you since you're probably just busy, but I'd like to hear about any progress or additional information you might have on this issue.
If you need help, for example a test system that is experiencing this issue for your debugging purposes, please let me know. I might be able to help you there by providing a running copy of that system (with sensitive data removed), so you could see for yourself what happens and what doesn't.
I suppose that might be the easiest way to get this solved, unless you already have a mostly identical system at your disposal for that purpose. But I'd imagine that to be quite hard considering the sheer number of possible OS/software variations.
Thanks in advance for all your efforts.
With kind regards
Mark
@tonioo commented on GitHub (Aug 2, 2024):
Now that I'm thinking about it, have you checked that rsyslog is installed and running? And also, have you checked if the mail.log file has any new content?
@mas1701 commented on GitHub (Aug 5, 2024):
Yes, rsyslogd is present and active,from "ps" command:
568 ? Ssl 0:09 /usr/sbin/rsyslogd -n -iNONEWhat do you expect me to find in mail.log other than normal mail flow?
(Yes, normal mail flow is working. Although this is a test run before upgrading a larger server, this smaller one does have active mail flow and is used regularly. Any problems would be noticed very quickly. The system is also being actively monitored by an Icinga2 monitoring system, including an active client-side monitoring agent.)
Edit:
As an in-between solution for the larger server, is it possible to upgrade the plugins to the newest versions w/o upgrading the base Modoboa package or will this cause problems? Thinking about dependencies and auto-reply module.
@tonioo commented on GitHub (Aug 9, 2024):
I was asking because Debian 12 does not install rsyslogd by default anymore.
If you see trafic traces in mail.log file, the parser should work. I don't know what could be the problem yet and without a way to reproduce your environment, I won't be able to understand what is going on...
@mas1701 commented on GitHub (Aug 10, 2024):
I see. Again, wouldn't it help to provide you with access to a running copy of that system?
@pappastech commented on GitHub (Oct 27, 2024):
After an upgrade to Modoboa 2.3.2, the mail stats on my system still are not working.
The log parser is still not generating any rrd files for global or the single domain hosted by this server. There is mail traffic. Anyone have any ideas?
Thanks.
@tonioo commented on GitHub (Oct 28, 2024):
@pappastech Have you checked in the mail.log file has content?
@pappastech commented on GitHub (Oct 28, 2024):
Yes. I 'tail -f' the mail.log file frequently and there's constant activity in there.
@pappastech commented on GitHub (Oct 29, 2024):
Any thoughts here @tonioo? Is there other information I can provide that would help to understand what's going on? I've dug in a couple areas but I'm not a django expert, so not always sure where to look next.
@tonioo commented on GitHub (Oct 31, 2024):
@pappastech Can you try to manually run the log parser using --verbose or --debug options?
@pappastech commented on GitHub (Oct 31, 2024):
I've tried that numerous times, but here's a fresh try and the output. Note that I've changed the actual domain hosted on this server to 'domain.com'; I'm not actually hosting 'domain.com'.
@tonioo commented on GitHub (Nov 1, 2024):
Can you show a sample of your log file? Maybe that the format is different from what the parser is expecting...
@pappastech commented on GitHub (Nov 1, 2024):
Here you go...
@pappastech commented on GitHub (Nov 1, 2024):
I guess comparing that format to a different instance of Modoboa I have running for some other domains, the date format does look different. I'm not sure why that would have happened since this was a brand new install of Modoboa earlier this year.
Here's an excerpt from another instance where the date format is different and log parsing works:
Does the log parser handle the difference in date formats?
@pappastech commented on GitHub (Nov 1, 2024):
One difference from a fresh install point of view is that the server where log parsing works is running CentOS and the server where log parsing doesn't work is running Debian. It occurs to me that the default logging behavior is possibly different just because of different Linux distros. Looking into how to change this on the Debian server...open to suggestions if people have them.
I'd prefer to not change default behavior in the OS though...I think this should be a change in the logparser function of Modoboa. So looking at that as well.
@tonioo commented on GitHub (Nov 1, 2024):
And you configured the right file path in Modoboa?
@pappastech commented on GitHub (Nov 1, 2024):
Alright, well I think the issue is that the regex in logparser.py only allowed for UTC+ time zones. I added a single line to the list of possible regex structures and now I see rrd files after running the logparser and I see stats in the web interface. There's probably a more elegant way to handle the regex option of a + versus a - in the timezone, but this seems to work.
From:
To:
@tonioo commented on GitHub (Nov 1, 2024):
Good catch!
Do you think you could create a PR with your fix?
@layer3-fl commented on GitHub (Nov 1, 2024):
Great find and fix!
Having had other projects taking priority I have had this on the back burner and haven’t looked at it further.
Your fix worked for my Debian based system as well.
Warm regards,
Mattco Incorporated dba, Layer3
Matthew Strowbridge
(o) 321.541.4159
(m) 321.722.7571
(e) @.***
Federal Cage Code: 9QPB1
NAICS Codes: 541512, 541519
On Nov 1, 2024, at 6:41 AM, Antoine Nguyen @.***> wrote:
Good catch!
Do you think you could create a PR with your fix?
—
Reply to this email directly, view it on GitHub https://github.com/modoboa/modoboa/issues/3186#issuecomment-2451672758, or unsubscribe https://github.com/notifications/unsubscribe-auth/BHCJ6OFBQK5MNRYZGXZXJNDZ6NLHLAVCNFSM6AAAAABDCDTNPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJRGY3TENZVHA.
You are receiving this because you were mentioned.
@pappastech commented on GitHub (Nov 1, 2024):
It's not obvious to me how to do this on Github.
@tonioo commented on GitHub (Nov 1, 2024):
You first need fork this repo and clone it locally, then commit your changes to your fork and go back to Github. There, you'll be asked if you want to create a PR
@pappastech commented on GitHub (Nov 1, 2024):
Alright...I think it's done.
@stale[bot] commented on GitHub (Jan 31, 2025):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.