mirror of
https://github.com/modoboa/modoboa.git
synced 2026-04-27 01:45:58 +03:00
[GH-ISSUE #2475] Amavis crashed every 20 messages: (!)Amavis::END: DB unregistering failed #1611
Labels
No labels
bug
bug
dependencies
design
documentation
duplicate
enhancement
enhancement
enhancement
feedback-needed
help-needed
help-needed
installer
invalid
looking-for-sponsors
modoboa-contacts
new-ui
new-ui
pr
pull-request
pyconfr
python
question
security
stale
webmail
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/modoboa-modoboa#1611
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @gianks on GitHub (Mar 16, 2022).
Original GitHub issue: https://github.com/modoboa/modoboa/issues/2475
Impacted versions
1.17.0
Steps to reproduce
Receive 20 or more emails.
Current behavior
Repeated tests show
Amavis crashing exactly after 20 processed emails with message:
amavis[XX]: (XX-20) (!)Amavis::END: DB unregistering failed:
Expected behavior
Amavis keeps running without any intervention.
Video/Screenshot link (optional)
@gianks commented on GitHub (Mar 16, 2022):
A possible solution seems to be setting $max_server = 2 instead of 1 in /etc/amavis/conf.d/50-user (generated by modoboa).
I did not investigate extensively why, but the magical number 20 is defined within amavis-new scritp, as maximum child requests.
I suspect that using a single server the program terminates when the first child arrives at 20 instead of spawning a new one, but it's just a guess.
@vingeni commented on GitHub (Mar 21, 2022):
Hi,
I got the same issue, as a workaround I added two lines into the service file
/etc/systemd/system/multi-user.target.wants/amavis.serviceto restart the service:@MrGeneration commented on GitHub (Mar 24, 2022):
How much system ressources did you provide your modoboa machine?
@stale[bot] commented on GitHub (May 25, 2022):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
@blackhold commented on GitHub (Jun 10, 2022):
Hello, I had the same problem in debian 11.3 with installer, I fixed incrementing $max_server = 2 in /etc/amavis/conf.d/50-user, at the moment I don't have to apply systemd configuration as mentioned by vingeni.
@IppyD commented on GitHub (Jun 27, 2022):
Same issue here. Lots of resources available in principle for one domain/~30 mailboxes, but amavis was restarting - and Monit reporting it - quite manically. I doubled memory, pursued the logs and eventually arrived here. No reports since setting
$max_server = 2.However I did note that this value is reported to need to match the value in postfix configuration (see, eg here: https://docs.iredmail.org/concurrent.processing.html ). That is
smtp-amavisvalue should equal$max_servervalue.I couldn't actually find this config value though. Is it true? Did I misunderstand?
@Spitfireap commented on GitHub (Jul 17, 2022):
same issue here, it crashes after 20 mails, I've set up the max_server to 2 to see if it works. @IppyD did you try adding
smtp-amavis_destination_concurrency_limit = 2to main.cf maybe ?@nicosqc commented on GitHub (Jul 21, 2022):
not sure if related, but on my machine with 8GB RAM and 6 cpu cores I did set $max_servers to 8 and now I have no more timeout problems on load. Before, if I sent 5-10MB .exe files to 10 different aliases on my machine half of the mails would time out and the other mailserver would resend half of them because they had the "mail queue write error" aka
postfix/smtpd[3972]: warning: timeout talking to proxy inet:[127.0.0.1]:10024I was looking into this too, I checked both master and main.cf and I could not identify the service used for amavis. There is a flag
-o smtpd_proxy_options=speed_adjustafter-o smtpd_proxy_filter=inet:[127.0.0.1]:10024with smtpd so I guess this is what automatically adjusts the amount of instances that are spun up. Either way, it works fine without either configuration change. To my understanding addingsmtp-amavis_destination_concurrency_limit = 2would serve no purpose since there is no smtp-amavis service this would refer to@stale[bot] commented on GitHub (Sep 20, 2022):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
@stefaweb commented on GitHub (Sep 28, 2022):
Hello,
I started to receive tens of "Out: 451 4.3.0 Error: queue file write error" mail from the new 2.0.2 Modoboa server I just installed. I installed the "lmtp patch" (using description in https://github.com/modoboa/modoboa-installer/pull/425).
It seems that it fails as soon as the server receives more than two or three mails at the same time. However, I have a correct server with 8Go of RAM.
Regards,
@stefaweb commented on GitHub (Sep 29, 2022):
Hello,
No more error since yesterday.
I think it would be better to have the configuration using lmtp by default because it seems that even with few messages coming to the server, the system through the proxy seems unreliable.
Regards,
@tonioo commented on GitHub (Sep 30, 2022):
FYI: https://www.postfix.org/SMTPD_PROXY_README.html#pros_cons
@stefaweb commented on GitHub (Sep 30, 2022):
Hello,
The server is an Scaleway VM cloud instance (PRO2-XXS) with 2 cores 64Bits, SSD block storage and 8Go de RAM. No swap. System is Debian bullseye uptodate.
I tried again yesterday with the default proxy setup. The system crashes and loses mails as soon as more than 2 or 3 mails are processed at the same time. Whether they come from an auth login or not. The worst is that we are not warned. We just see it in mail.log.
Regards,
@tonioo commented on GitHub (Sep 30, 2022):
Have you checked your master.cf file to check the number of allowed connections for the smtp service? (first one)
@stefaweb commented on GitHub (Sep 30, 2022):
I have the default config (modboboa-installer) + the change for amavis lmtp.
smtp inet n - - - 1 postscreenI wrote something wrong. When the system crashes, I receive a system email with the error "queue file write error" but not for all lost emails.
EDIT: and this in the log.
@tonioo commented on GitHub (Sep 30, 2022):
And what is your base RAM consumption when everything is running?
@stefaweb commented on GitHub (Sep 30, 2022):
I have this with the amavis lmtp config.
@tonioo commented on GitHub (Sep 30, 2022):
Have you also updated the maxproc setting of the smtp-amavis service? It should be equal to $max_server
@stefaweb commented on GitHub (Sep 30, 2022):
Oops, I'm lost.
I will put the default settings without lmtp and I will do new tests.
@stefaweb commented on GitHub (Sep 30, 2022):
Here is the consumption with the default settings just after a reboot.
I will now wait to see if the errors start again.
`
root@mailhub:~# free -h
total used free shared buff/cache available
Mem: 7.8Gi 1.8Gi 4.1Gi 21Mi 1.9Gi 5.7Gi
Swap: 0B 0B 0B
root@mailhub:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 4326024 36996 1933840 0 0 3404 505 171 247 8 4 86 3 0
`
Here are the default settings after the config via modoboa-installer.
Nothing added in /etc/postfix/main.cf.
@stefaweb commented on GitHub (Sep 30, 2022):
Well, that didn't drag.
I did not receive an error mail on postmaster@.
@tonioo commented on GitHub (Sep 30, 2022):
And if you increase max_servers and max proc to 2?
@stefaweb commented on GitHub (Sep 30, 2022):
I'm going to try.
It seems to only do this when there is more than one mail to process. If there is only one, I don't get the error.
@stefaweb commented on GitHub (Sep 30, 2022):
It doesn't like to set the values to 2. I put 1 back for max_servers and max proc to.
Sep 30 11:35:07 mailhub postfix/postscreen[1931]: fatal: service smtp requires a process limit of 1@tonioo commented on GitHub (Sep 30, 2022):
It's smtp-amavis service that you must update, not the smtp one
@stefaweb commented on GitHub (Sep 30, 2022):
And if we use theses commands in Amavis?
I found this old text while searching with Google.
@stefaweb commented on GitHub (Sep 30, 2022):
I no longer have smtp-amavis in master.cf because I switched to the default config without the smtp-amavis lmtp service. When I use the smtp-amavis lmtp service, I no longer have these errors.
When I used the lmtp config, I had 2 for max_servers and the max proc for the smtp-amavis service.
@stefaweb commented on GitHub (Sep 30, 2022):
It still does with these settings. I removed them.
I switched back to lmtp config as long as there are no other tests to do.
@stefaweb commented on GitHub (Sep 30, 2022):
Again me.
I tried parsing the last logs with default proxy config + above two cache commands enabled.
I can't figure out if the "timeout after END-OF-MESSAGE" is an error or normal behavior with caches. The mail arrived well in the quarantine of the GUI if needed.
I did not have any other such error during this short test.
I removed the lines of the logs which are useless.
@stefaweb commented on GitHub (Oct 1, 2022):
Hello,
I redid the tests with the default proxy configuration.
If I add these two commands in /etc/amavis/conf.d/50-user the problem disappears.
Nothing else to change.
I'm going to let it run for several days to be sure.
@stefaweb commented on GitHub (Oct 3, 2022):
Hello,
Well, that does it again twice at short intervals.
I now have amavis logging enabled for next error and waited again.
If I understand the problem correctly. A first processing of several emails is executed. At the end of this processing, an error "DB unregistering failed" occurs and the amavis process crashes.
Another email arrives a little later and the processing crashes because the amavis proxy is no longer available.
Something weird with the amavis process it seems.
If I look with a systemlctl status amavis.
If we have $max_servers = 1;
If we have $max_servers = 2;
When we set max_servers to 1, we have no child.
So, with $max_servers = 1 if the process crashes, it's the master process that crashes and we lose the amavis proxy.
The minimum value for max_servers should therefore be = 2 to be sure that amavis will be still alive if a process crash.
On the other hand, I still don't understand why the amavis child crashes with a "DB unregistering failed" under certain conditions. It hasn't done it again since Saturday.
Regards,
@Spitfireap commented on GitHub (Oct 3, 2022):
It's probably because of
max_use, with amavis it should default to 20 (https://amavis.org/README.postfix.html#basics_transport_lmtp-client) and it marks the number of precessing before terminating the instance (probably for memory management I guess). So if$max_serversis set to 1, then the master process gets terminated: amavis won't restart.@stefaweb commented on GitHub (Oct 3, 2022):
To clarify, I 'm now using fresh default install for amavis, not the lmtp version as described in https://github.com/modoboa/modoboa-installer/pull/425.
Changes to default:
Since Saturday no more crash yet with this (DB unregistering or queue file write error).
@fedi-will commented on GitHub (Oct 13, 2022):
Same problem here.
Yesterday I installed modoboa, everything working fine.
Now I wanted to test the issue at hand here:
I manually sent like 25 emails from an external account to my modoboa test account. (plain text, no attachment, "manually" sent one after the other from Thunderbird, took me several minutes, so no "burst" occurred)
Result
Active: inactive (dead) since Thu 2022-10-13 19:51:05 CEST; 26min agoand
Question
If I put
$max_servers = 2;into amavis config, then could someone please tell what exactly I have to put into postfix config, so that the values match??There have been several suggestions about that. And my /etc/postfix/master.cf has no amavis-smtp... line in it.
@stefaweb when you say in your last comment you made those changes in amavis config, did you also change/edit anything in postfix config??
About the amavis service
When this error occurs, amavis actually exits with a success status.
And as on my Debian 11 the amavis service file has
Restart=on-failure, amavis isn't even restarted. That's why @vingeni's workaround from above works at least.Exiting with success feels wrong here, right? Who is to blame for that?
@fedi-will commented on GitHub (Oct 15, 2022):
Ok, I did the following:
edited /etc/postfix/main.cf and /etc/postfix/master.cf to have amavis running as a dedicated lmtp.
This way I can set the maxproc value to 2 so that it matches the amavis value
Test
Again manually sent 25 mails, within about 1 minute, like in my previous post.
Result
All mails were received.
No errors in the logs.
I am going to keep this configuration.
@tonioo: is a failing amavis a common occurrence or only happens to the very few people from this issue?
Having set max_servers=1 in amavis config, for how big a load is your modoboa standard installation conceptionated? Could it take a load of 100 mails per day? 1000 or 10000?
@isaaclepes commented on GitHub (Oct 25, 2022):
Just encountered this issue after updating to latest Modoboa.
Applied the recommended changes and will see how it goes.
@Spitfireap commented on GitHub (Oct 25, 2022):
A fix has just been pushed to the installer so the default
$max_serversvalue is 2. Hopefully it will fix future issues :)@isaaclepes commented on GitHub (Oct 26, 2022):
Awesome!
Just to report-back.. not only does it stop the crashing, it seems like emails are going through a bit quicker too. That could just be my imagination, though.
@stale[bot] commented on GitHub (Dec 26, 2022):
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
@PatTheMav commented on GitHub (Jan 3, 2023):
For posterity's sake: The reason why
$max_serversset to1became an issue is because of howamavis' Debian package is set up:Systemd starts the service via
ExecStart=/usr/sbin/amavisd-new foreground, which implicitly makes the service setdaemonizeto0and with only 1 server configuredamaviswill not fork but handle everything "in process" until the maximum amount of requests is reached and then quit.Running with just 1 child process (which is forked anew after maximum amount of mails have been processed) requires the service to be run as a daemon, which Debian's service configuration does not do by default.
@michaelNGV commented on GitHub (Feb 16, 2023):
The identifier is
$max_serversin the plural. If you're going to comment, please check what you're writing, as others will rely on it.I did have more rant but I'll leave it there.
@isaaclepes commented on GitHub (Sep 2, 2024):
Just encountered the same issue. Can we get a note added to the Wiki for installation/upgrades that warns of this?