mirror of
https://github.com/axllent/mailpit.git
synced 2026-04-26 00:35:51 +03:00
[GH-ISSUE #522] New metrics functionality spamming api.github.com #336
Labels
No labels
awaiting feedback
bug
docker
documentation
enhancement
github_actions
invalid
pull-request
question
stale
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/mailpit#336
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @avesst on GitHub (Jun 16, 2025).
Original GitHub issue: https://github.com/axllent/mailpit/issues/522
Hi,
The new Prometheus metrics endpoint introduced in #505 causes Mailpit to poll api.github.com to check for new versions of Mailpit two times every second minute making our organisation hit the Github rate limit for unauthenticated users.
This causes issues for other parts in our organisation using Githubs API (for instance, Packer and Terraform plugins are fetched from Github).
From what I've gathered, this is caused by prometheus.UpdateMetrics() calling stats.Load() which in turn calls updater.GithubLatest(), though I haven't been able to figure out why it's causing two requests each iteration as seen in our firewall logs.
@axllent commented on GitHub (Jun 16, 2025):
Oh no! This should not be happening, the latest version should be cached for 15 minutes. But... if there is an error getting the latest version it will keep retrying on every request which is not good in situations like this.
In the meantime please turn off your Prometheus metrics as there is clearly an issue. I will look into this within the next day or two. If you could give me any more info that would be great, thanks.
@grimne commented on GitHub (Jun 16, 2025):
Hi, I work with @avesst. Thank you for the reply
Happy to help and/or assist in any way possible.
@axllent commented on GitHub (Jun 16, 2025):
Thank you for the information. It should not be checking for a new version more than once per 15 minutes, so there might be a bug somewhere. It might be related to the two requests you are seeing every minute at the same time
I do know that if there is an error response (from GitHub) then it will check every time, and this needs to change too.
I expect to have a fix in the next few days and a new release this weekend. I apologise for any inconvenience this may have caused you.
@avesst commented on GitHub (Jun 16, 2025):
No worries, there were no significant impact for us. :)
@Tigger2014 commented on GitHub (Jun 16, 2025):
I've worked this out, or rather, possibly why it shows up so much more with metrics enabled. It was likely happening without Prometheus enabled, but would only happen every 15 mins, assuming it's always a failure as
Load()would be infrequently called. Prometheus would callLoad()every 15 seconds.I've taken a shot at resolving this all here
github.com/axllent/mailpit@9b08f73155This adds both a read and write mutex over the version, as that could cause it to race if two things call
Load()at once.I've also added exponential backoff calling github, maxing out at 30 minutes.
@axllent commented on GitHub (Jun 17, 2025):
Thanks (again!) @Tigger2014. Could you please submit that as a new PR (I have no idea where that commit is currently, but I suspect somewhere in your fork - even GitHub doesn't quite know lol)? While you're at it, could you please add docblocks (comments) to the
versionCachestruct andgetBackoff()function, thank you.@axllent commented on GitHub (Jun 17, 2025):
Sorry, one more thing @Tigger2014 - when an "error" is cached, the version cached is "error" (if I'm correct)? If so, then the logic in
server/ui-src/components/AboutMailpit.vue(line 94) needs changing, so maybev-if="mailbox.appInfo.LatestVersion == '' || mailbox.appInfo.LatestVersion == 'error'"(untested).@Tigger2014 commented on GitHub (Jun 17, 2025):
No problem @axllent , Sorry the link got lost, it seems GitHub was a bit smart and managed to reference only the commit!
I've submitted the PR with your improvements, except the update
AboutMailpit.vueas it made more sense to keep the error as an empty version as it already was previously, so I've amended my changes to reflect that instead.@axllent commented on GitHub (Jun 17, 2025):
Thank you so much @Tigger2014 - I will review both PR's in the next couple of days and aim for a new release this weekend.
@axllent commented on GitHub (Jun 18, 2025):
Great work again @Tigger2014 - I think your fix should work nicely. I suspect the double requests may have been a factor as to why this resulted in an error originally (impossible to tell), but as I mentioned earlier, when there was a persistent failure (for whatever reason) then Mailpit would try on every request. This new logic should handle that much better (effectively caching the error, with an increasing "backoff" period during extended failures).
@avesst & @grimne With this fix (once released this weekend), successful update checks should happen once per 15 minutes (which is how it was working), and with these changes failed checks will only test again after one minute, then then after 2 minutes from the last check, and then after three minutes from the last check etc etc. This will increase up to once every 30 minutes if the failures continue. I have also added console logging when an update check fails, so at least there will be a log of sorts if things go wrong in the future so we can try get better insight.
I'll update this ticket once the new version is released this weekend. If you wanted to test in the meantime and you're either using Docker or self compiling, then the
axllent/mailpit:edge(Docker) version or thedevelopbranch (self-compiling) have the latest changes. If you prefer to wait then not a problem either, I'm pretty sure this will at least prevent the spamming to api.github.com if things fail for whatever reason.@avesst commented on GitHub (Jun 18, 2025):
We have a short work week this week due to holidays in Sweden, so we'll await the release for FreeBSD and test ASAP next week.
Thanks for impeccable maintenance of this repo @axllent and thanks for feature implementation and quick fix @Tigger2014 :)
@axllent commented on GitHub (Jun 21, 2025):
This fix has now been released in v1.26.2. 👍 Can you please confirm this works from you @avesst & @grimne ? Thank you, and thanks again @Tigger2014 for the fix.
@axllent commented on GitHub (Jun 28, 2025):
I'm closing this issue, but feel free to re-open if it is still an issue.
@avesst commented on GitHub (Jun 28, 2025):
@axllent We got the patch yesterday and I just enabled it for 30 minutes and had a look. It seems the interval is more and more sparse, but it still seems it makes two requests every occasion?
@Tigger2014 commented on GitHub (Jun 28, 2025):
I'm not sure what's causing the doubles myself, Ive been unable to reproduce it. Hopefully @axllent will have some ideas.
You could also add
--disable-version-check / MP_DISABLE_VERSION_CHECKto disable the calls to GitHub altogether as a workaround in the meantime, I'm sure you're keen to benefit from the prom metrics.@axllent commented on GitHub (Jun 28, 2025):
Thanks for the feedback guys. There are a few options here worth exploring.
1 - I still think it is highly likely there are two requests hitting Mailpit each time as Mailpit does not immediately retry a failed update check. @avesst Is there any way you can establish if that is the case? Possibilities could include two separate "profiles" (?) set up in Prometheus for Mailpit, or alternatively a load-balanced Prometheus server (so there are actually two). Maybe you have one for the Prometheus endpoint, and another check still set up for
/api/v1/info(which you were originally testing in #497)? Either way, it looks to me like there are two separate requests hitting Mailpit every time.2 - The timings of these GitHub requests is all over the place. Forgetting briefly the "double requests", the time between them goes from 2 - 1 - 3 - 9 - 15 seconds. This makes no sense to me. With two requests each time it is likely the delay would be increasing by 2 minutes each time instead of one, but those delays logged by your firewall makes no sense to me. Are you able to rule out that Mailpit isn't crashing and restarting (or just being restarted) regularly? You should be able to see this from the Mailpit log files.
3 - There is a notable difference in the difference in the size of the data received between the latest (two) @ 4KB, and all the others at 36KB. The version check is polling
https://api.github.com/repos/axllent/mailpit/releases- which is 36KB when the HTTP request is gzipped, so those last two in your log appear to be errors. So it looks to me like those 36KB requests are in fact successful checks, meaning it should be caching values those for 15 minutes, so so there should be at least 15 minutes between each of those requests to GiHub (unless Mailpit is restarting). Can you please check the current Mailpit version in the web UI to ensure it is v1.26.2 or greater?4 - In addition to @Tigger2014's backoff changes, I also added error logging into Mailpit when it fails to do an update check. Have you looked at the Mailpit logs to look for errors getting the latest version?
@avesst commented on GitHub (Jul 7, 2025):
Hmm, I missed your reply @axllent, don't think I received an e-mail about this, perhaps because it's closed? (I don't seem to have permission to re-open btw)
Anyway; There is infact nothing at all hitting our Mailpit, I only enabled the Prometheus endpoint of Mailpit. We don't actually have a Prometheus server in house, we use VictoriaMetrics which has support for ingesting Prometheus data but works a little different in that it can't pull data, you have to push data to it.
Normally we have something like Telegraf or OpenTelemetry Collector running alongside that is doing the scraping of the exposed
/metricsendpoint, and sends it to VictoriaMetrics.I haven't looked at @Tigger2014's code, but as far as I know there shouldn't be possible to trigger something by scraping the endpoint itself.
The interval for scraping the data is completely arbitrary, but I think once every 15 seconds is probably the most common. It depends on how granular you need your timeseries data to be.
It might be as you suggest, that some requests are erros, due to us hitting GitHubs rate limit - there might be other services hitting GitHub making us randomly hit their rate limit which we haven't noticed.
I've now had Mailpit running with Prometheus enabled for just over 80 minutes with no hiccups. I will try to keep it enabled over the night to gather more data, but this is so far.
@Tigger2014 commented on GitHub (Jul 7, 2025):
Yes, simply having Prometheus enabled will cause us to check for the latest version as it kicks off an internal timer to refresh the stats every 15 seconds, which in turn will attempt to update the latest version if its backoff has elapsed.
There was a feature added to also disable version checking too
--disable-version-check/MP_DISABLE_VERSION_CHECK@avesst commented on GitHub (Jul 10, 2025):
I've just checked our logs again. Mailpit have now been running for >3 days with Prometheus activated, and from what I can tell - atleast for the past 1,5 days (that's how long back I scrolled), it seems to be back to polling every two minutes.
Last day tomorrow before summer vacation, so probably won't be available for a couple of weeks.
@axllent commented on GitHub (Jul 10, 2025):
Thanks @avesst . Do you mean Prometheus is polling Mailpit every 2 minutes, or Mailpit is polling GitHub every 2 minutes?
@avesst commented on GitHub (Jul 10, 2025):
Sorry,
Mailpit is polling Github. We don't have anything scraping of the endpoint setup.
-------- Original Message --------
On 10/07/2025 21:43, Ralph Slooten wrote:
axllent left a comment (axllent/mailpit#522)
Thanks @avesst . Do you mean Prometheus is polling Mailpit every 2 minutes, or Mailpit is polling GitHub every 2 minutes?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>
@axllent commented on GitHub (Jul 10, 2025):
Thanks. It sounds to me like the version caching may not working as expected as it should be caching for 15 minutes ~ unless the version check is failing in which case it should increase caching up to 15 minutes of the error. I will investigate this further.
@axllent commented on GitHub (Jul 25, 2025):
Sorry for the delay, I have been very busy recently. I have discovered that when Prometheus is enabled, Mailpit would check for new releases automatically (unless
--disable-version-check/MP_DISABLE_VERSION_CHECKwas set), even though the latest release information isn't actually used in the Prometheus data.I have pushed a fix for this this which will be released this weekend.
@axllent commented on GitHub (Jul 27, 2025):
This issue should now be resolved in v1.27.2. Prometheus statistics will no longer check the Github API for latest releases.
@avesst commented on GitHub (Aug 6, 2025):
Seems to be fixed, thanks!