[GH-ISSUE #402] Testing SMTP Server Error Responses

kerem commented

2026-03-15 13:31:28 +03:00

Owner

Originally created by @IBlasterus on GitHub (Dec 9, 2024).
Original GitHub issue: https://github.com/axllent/mailpit/issues/402

Sometimes during development you need to write logic for error responses from the SMTP server.
Can you add this feature to Mailpit Test Server?
There could be some startup keys that make Mailpit respond with an error.
For example, 451.

Originally created by @IBlasterus on GitHub (Dec 9, 2024). Original GitHub issue: https://github.com/axllent/mailpit/issues/402 Sometimes during development you need to write logic for error responses from the SMTP server. Can you add this feature to Mailpit Test Server? There could be some startup keys that make Mailpit respond with an error. For example, 451.

kerem

2026-03-15 13:31:28 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-15 13:31:44 +03:00

Author

Owner

@axllent commented on GitHub (Dec 9, 2024):

Previously a "chaos monkey" feature was requested (see #110, #144 & #268) to return SMTP errors at random. That proposal was rejected at the time because it was impossible to implement (properly) due to the fact that Mailpit was reliant a third party library for SMTP. Whilst Mailpit is still using a version of that library, it is now within the Mailpit codebase meaning it is now technically possible.

What you are asking for however is something different, so to help me understand your requirements:

Are you wanting a 100% chance of failure in your tests, or at random?
Is there a specific type of failure in the SMTP transaction, or is any SMTP error OK?
Should the error result in a SMTP server disconnection, or...?
Anything else you can think of...

The example given previously was MailHog's Jim feature which is what many/all of these previous requests were based on, and which may help provide some suggestions and ideas for this discussion.

Just to be clear, I'm not saying (yet) whether I will add this option, I'm just a lot more open to it now because it is now possible, it just needs to be implemented properly which takes a solid understanding of the requirements, and of course planning. It is also worth mentioning that I am having shoulder surgery in 2 days, so if I am to add this option it would likely only be in January or February next year, which gives me plenty of time to consider the options before then.

@axllent commented on GitHub (Dec 9, 2024): Previously a "[chaos monkey](https://github.com/Netflix/chaosmonkey)" feature was requested (see #110, #144 & #268) to return SMTP errors at random. That proposal was rejected at the time because it was impossible to implement (properly) due to the fact that Mailpit was reliant a third party library for SMTP. Whilst Mailpit is still using a version of that library, it is now within the Mailpit codebase meaning it is now technically possible. What you are asking for however is something different, so to help me understand your requirements: 1. Are you wanting a 100% chance of failure in your tests, or at random? 2. Is there a specific type of failure in the SMTP transaction, or is any SMTP error OK? 3. Should the error result in a SMTP server disconnection, or...? 4. Anything else you can think of... The example given previously was MailHog's [Jim](https://github.com/mailhog/MailHog/blob/master/docs/JIM.md) feature which is what many/all of these previous requests were based on, and which may help provide some suggestions and ideas for this discussion. Just to be clear, I'm not saying (yet) whether I will add this option, I'm just a lot more open to it now because it is now possible, it just needs to be implemented properly which takes a solid understanding of the requirements, and of course planning. It is also worth mentioning that I am having shoulder surgery in 2 days, so if I am to add this option it would likely only be in January or February next year, which gives me plenty of time to consider the options before then.

kerem commented

2026-03-15 13:31:49 +03:00

Author

Owner

@IBlasterus commented on GitHub (Dec 9, 2024):

Thank you very much for taking the time to answer my question.
I wish you a speedy recovery.

Are you wanting a 100% chance of failure in your tests, or at random?

Especially for me need 100% chance of failure.

Is there a specific type of failure in the SMTP transaction, or is any SMTP error OK?

Need specific type of failure. For example, for development backend reaction on 451 error.

Should the error result in a SMTP server disconnection, or...?

Maybe. My Backend connect to SMTP, send E-mail and disconnect.

Anything else you can think of...

Nothing else comes to mind.

@IBlasterus commented on GitHub (Dec 9, 2024): Thank you very much for taking the time to answer my question. I wish you a speedy recovery. > 1. Are you wanting a 100% chance of failure in your tests, or at random? Especially for me need 100% chance of failure. > 2. Is there a specific type of failure in the SMTP transaction, or is any SMTP error OK? Need specific type of failure. For example, for development backend reaction on 451 error. > 3. Should the error result in a SMTP server disconnection, or...? Maybe. My Backend connect to SMTP, send E-mail and disconnect. > 4. Anything else you can think of... Nothing else comes to mind.

kerem commented

2026-03-15 13:31:54 +03:00

Author

Owner

@ThomasLandauer commented on GitHub (Dec 11, 2024):

Just an idea on how to implement this:
Hard-code an email address like _mailpit_smtp_reply_code_451_after_rcpt@example.com, and if it's used as sender/receiver, return that error.
Any more fancy way (like sending some special keywords in the SMTP session) won't work, since most users probably can't modify that.

@ThomasLandauer commented on GitHub (Dec 11, 2024): Just an idea on how to implement this: Hard-code an email address like `_mailpit_smtp_reply_code_451_after_rcpt@example.com`, and if it's used as sender/receiver, return that error. Any more fancy way (like sending some special keywords in the SMTP session) won't work, since most users probably can't modify that.

kerem commented

2026-03-15 13:31:59 +03:00

Author

Owner

@axllent commented on GitHub (Dec 11, 2024):

Now that's an interesting approach, thanks for the idea @ThomasLandauer 👍

Edit: After some thought and as nice as this idea sounds, it is very limited in that the email would need to have already been processed in order to process the addresses, meaning the only failure one could trigger this way would be after a successful transaction.

@axllent commented on GitHub (Dec 11, 2024): Now that's an interesting approach, thanks for the idea @ThomasLandauer 👍 Edit: After some thought and as nice as this idea sounds, it is very limited in that the email would need to have already been processed in order to process the addresses, meaning the only failure one could trigger this way would be after a successful transaction.

kerem commented

2026-03-15 13:32:04 +03:00

Author

Owner

@axllent commented on GitHub (Jan 26, 2025):

I have just released v1.22.0 which adds new "Chaos" functionality (see release notes and links for documentation). The configuration is fairly flexible to allow you to set the error code and probability (which determines whether you want it to trigger never, randomly, or always), and can be set / changed via CLI flags (or environment variables), and if enabled, can be modified via both the web UI and API.

Please let me know how it works for you? Thanks.

@axllent commented on GitHub (Jan 26, 2025): I have just released [v1.22.0](https://github.com/axllent/mailpit/releases/tag/v1.22.0) which adds new "Chaos" functionality (see release notes and links for documentation). The configuration is fairly flexible to allow you to set the error code and probability (which determines whether you want it to trigger never, randomly, or always), and can be set / changed via CLI flags (or environment variables), and if enabled, can be modified via both the web UI and API. Please let me know how it works for you? Thanks.

kerem commented

2026-03-15 13:32:09 +03:00

Author

Owner

@ThomasLandauer commented on GitHub (Jan 26, 2025):

I've read those issues where people are requesting that chaos feature, but - to be honest - I don't understand their use case, (i.e. how adding unpredictability helps ;-)

Anyway, my use case is: I have a test suite that I run before deploying. The test suite contains a test for a successful connection to my mailserver (represented by Mailpit), and one test for an unsuccessful connection. The current implementation (if I understand it right) would require me to create a separate test suite for the unsuccessful test, and restart Mailpit when switching between theses test suites. This feels even clumsier than the hack I currently have (see https://github.com/axllent/mailpit/pull/405#issuecomment-2539983417)

So what I'd like to have would be a way to "request" an error from within the test. So my idea would be to either hard-code some "special" email address (give_error@example.com) or subject ("Please answer with error") or any other part of the email itself (X-Mailpit-Give-Error: Yes). Or add some API call (that can be triggered with cURL) which causes an error in the next SMTP session.

@ThomasLandauer commented on GitHub (Jan 26, 2025): I've read those issues where people are requesting that chaos feature, but - to be honest - I don't understand their use case, (i.e. how adding unpredictability helps ;-) Anyway, my use case is: I have a test suite that I run before deploying. The test suite contains a test for a successful connection to my mailserver (represented by Mailpit), and one test for an unsuccessful connection. The current implementation (if I understand it right) would require me to create a separate test suite for the unsuccessful test, and restart Mailpit when switching between theses test suites. This feels even clumsier than the hack I currently have (see https://github.com/axllent/mailpit/pull/405#issuecomment-2539983417) So what I'd like to have would be a way to "request" an error *from within the test.* So my idea would be to either hard-code some "special" email address (give_error@example.com) or subject ("Please answer with error") or any other part of the email itself (X-Mailpit-Give-Error: Yes). Or add some API call (that can be triggered with cURL) which causes an error in the *next* SMTP session.

kerem commented

2026-03-15 13:32:14 +03:00

Author

Owner

@axllent commented on GitHub (Jan 26, 2025):

Thanks for the feedback @ThomasLandauer. Chaos (engineering) revolves around handling unpredictability (ie: failures at random), which is different to you r case where you want to predict the failure. Predictability however can be achieved with the same functionality by setting the failure trigger to 100(%) probability, guaranteeing it will fail.

If I understand correctly, your test suite spins up an instance of Mailpit when it runs the test. You do not need to spin up a separate instance of Mailpit or do a completely separate test to get a failure, you just need to:

Spin up the initial instance of Mailpit with the --enable-chaos flag (or MP_ENABLE_CHAOS=true env)
Run your successful test
Do a PUT request to the API to set chaos trigger you want to fail and the failure code
Run your failure test

This matches your last point exactly, so either I'm confused, or maybe the instructions are not clear?

@axllent commented on GitHub (Jan 26, 2025): Thanks for the feedback @ThomasLandauer. Chaos (engineering) revolves around handling **unpredictability** (ie: failures at random), which is different to you r case where you want to predict the failure. Predictability however can be achieved with the same functionality by setting the failure trigger to 100(%) probability, guaranteeing it will fail. If I understand correctly, your test suite spins up an instance of Mailpit when it runs the test. You do not need to spin up a separate instance of Mailpit or do a completely separate test to get a failure, you just need to: 1. Spin up the initial instance of Mailpit with the `--enable-chaos` flag (or `MP_ENABLE_CHAOS=true` env) 2. Run your successful test 3. Do a `PUT` [request](https://mailpit.axllent.org/docs/api-v1/view.html#put-/api/v1/chaos) to the API to set chaos trigger you want to fail and the failure code 4. Run your failure test This matches your last point exactly, so either I'm confused, or maybe the instructions are not clear?

kerem commented

2026-03-15 13:32:20 +03:00

Author

Owner

@ThomasLandauer commented on GitHub (Jan 27, 2025):

Thanks, this works! :-)

Some suggestions (somewhat unordered - sorry):

If I hadn't followed this issue, I never would have found this feature under the heading "Chaos" => Please make this use case more prominent (don't have enough overview to suggest a place). Maybe even add a copy-pastable PUT request somewhere: 'http://127.0.0.1:8025/api/v1/chaos', '{"Sender": {"ErrorCode": 500, "Probability": 100 }}'
At https://mailpit.axllent.org/docs/api-v1/view.html#overview, add the base URL. At https://mailpit.axllent.org/docs/api-v1/, you're mentioning http://0.0.0.0:8025/ - is this a placeholder or the actual URL? I'm now using http://127.0.0.1:8025/api/v1/chaos
Since the entire chaos feature is probably only relevant for people who know SMTP, I would rename the triggers to use the actual SMTP command names (and also change their order in the examples):
```
{
"AUTH": {
"ErrorCode": 451,
"Probability": 5
},
"MAIL": {
"ErrorCode": 451,
"Probability": 5
},
"RCPT": {
"ErrorCode": 451,
"Probability": 5
}
}
```

Does the response to PUT contain a trailing \n? I'm getting:

- Expected | + Actual
@@ @@
-'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}'
+'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}
+'

@ThomasLandauer commented on GitHub (Jan 27, 2025): Thanks, this works! :-) Some suggestions (somewhat unordered - sorry): * If I hadn't followed this issue, I **never** would have found this feature under the heading "Chaos" => Please make this use case more prominent (don't have enough overview to suggest a place). Maybe even add a copy-pastable PUT request somewhere: `'http://127.0.0.1:8025/api/v1/chaos', '{"Sender": {"ErrorCode": 500, "Probability": 100 }}'` * At https://mailpit.axllent.org/docs/api-v1/view.html#overview, add the base URL. At https://mailpit.axllent.org/docs/api-v1/, you're mentioning `http://0.0.0.0:8025/` - is this a placeholder or the actual URL? I'm now using `http://127.0.0.1:8025/api/v1/chaos` * Since the entire chaos feature is probably only relevant for people who know SMTP, I would rename the triggers to use the actual SMTP command names (and also change their order in the examples): ```json { "AUTH": { "ErrorCode": 451, "Probability": 5 }, "MAIL": { "ErrorCode": 451, "Probability": 5 }, "RCPT": { "ErrorCode": 451, "Probability": 5 } } ``` * Does the response to PUT contain a trailing `\n`? I'm getting: ``` - Expected | + Actual @@ @@ -'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}' +'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}} +' ```

kerem commented

2026-03-15 13:32:25 +03:00

Author

Owner

@axllent commented on GitHub (Jan 27, 2025):

Thanks, this works! :-)

Great!

Some suggestions (somewhat unordered - sorry):

* If I hadn't followed this issue, I **never** would have found this feature under the heading "Chaos" => Please make this use case more prominent (don't have enough overview to suggest a place).

It's currently described & linked to from the project's README, Mailpit's home page, the features page, the configuration page , and the runtime options pages, so I really don't think it can be any more prominent than that, especially considering it's an advanced option which most users will never need or use. ;-)

Maybe even add a copy-pastable PUT request somewhere: `'http://127.0.0.1:8025/api/v1/chaos', '{"Sender": {"ErrorCode": 500, "Probability": 100 }}'`

Whilst I get your point, the online API documentation differs somewhat from the bundled (in Mailpit itself) API documentation which does - there are actually clear curl examples in the bundled documentation. The reason for the difference is that the website cannot know what IP your Mailpit instance is actually using, it's webroot, or whether it's even running, and I'm concerned that the average user will start posting Github issues because "it's not working" for them.

~~I will make the information about the bundled API documentation clearer on the website and explain this.~~

Edit: I have updated the online API to include curl examples. They cannot be interactive because Mailpit is typically hosted via HTTP and the website is HTTPS (your browser will block you), but the examples are now there.

* At https://mailpit.axllent.org/docs/api-v1/view.html#overview, add the base URL. At https://mailpit.axllent.org/docs/api-v1/, you're mentioning `http://0.0.0.0:8025/` - is this a placeholder or the actual URL? I'm now using `http://127.0.0.1:8025/api/v1/chaos`

0.0.0.0 is actually the same as 127.0.0.1, which is the same as localhost. ~~As above, I will try make this clearer and more consistent.~~

Edit: I have updated the API documentation to use localhost, even though this is a guess as I cannot know how Mailpit is configured ~ localhost is the default displayed in the console through.

* Since the entire chaos feature is probably only relevant for people who know SMTP, I would rename the triggers to use the actual SMTP command names (and also change their order in the examples):
  {
  "AUTH": {
  "ErrorCode": 451,
  "Probability": 5
  },
  "MAIL": {
  "ErrorCode": 451,
  "Probability": 5
  },
  "RCPT": {
  "ErrorCode": 451,
  "Probability": 5
  }
  }

Ironically, I had it as AUTH, MAIL and RCPT right up until the last minute before committing - but figured it may cause confusion (plus I needed to then explain each step in the web UI). In the end I went with the more human-readable object keys. Too late to change now, the feature is in the API and changing that would be a breaking change.

* Does the response to PUT contain a trailing `\n`? I'm getting:
  ```
  - Expected | + Actual
  @@ @@
  -'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}'
  +'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}
  +'
  ```

Quite possibly - I don't format the HTTP response manually, it's using Go's built-in JSON encoding to turn a struct (of the current config) into a JSON string. On that note, JSON should never be validated as a 1:1 string, as it it is prone to formatting differences (eg: "pretty"), and the fact that the order of keys can differ per implementation (Go will set them alphabetically, but PHP for instance won't). A JSON string should be converted to JSON and then parsed, and a JSON string with or without trailing lines is valid.

@axllent commented on GitHub (Jan 27, 2025): > Thanks, this works! :-) Great! > Some suggestions (somewhat unordered - sorry): > > * If I hadn't followed this issue, I **never** would have found this feature under the heading "Chaos" => Please make this use case more prominent (don't have enough overview to suggest a place). It's currently described & linked to from the project's README, Mailpit's home page, the features page, the configuration page , and the runtime options pages, so I really don't think it can be any more prominent than that, especially considering it's an advanced option which most users will never need or use. ;-) > Maybe even add a copy-pastable PUT request somewhere: `'http://127.0.0.1:8025/api/v1/chaos', '{"Sender": {"ErrorCode": 500, "Probability": 100 }}'` Whilst I get your point, the online API documentation differs somewhat from the bundled (in Mailpit itself) API documentation which does - there are actually clear curl examples in the [bundled documentation](http://127.0.0.1:8025/api/v1/#put-/api/v1/chaos). The reason for the difference is that the website cannot know what IP your Mailpit instance is actually using, it's webroot, or whether it's even running, and I'm concerned that the average user will start posting Github issues because "it's not working" for them. ~~I will make the information about the bundled API documentation clearer on the website and explain this.~~ Edit: I have updated the online API to include curl examples. They cannot be interactive because Mailpit is typically hosted via HTTP and the website is HTTPS (your browser will block you), but the examples are now there. > * At https://mailpit.axllent.org/docs/api-v1/view.html#overview, add the base URL. At https://mailpit.axllent.org/docs/api-v1/, you're mentioning `http://0.0.0.0:8025/` - is this a placeholder or the actual URL? I'm now using `http://127.0.0.1:8025/api/v1/chaos` `0.0.0.0` is actually the same as `127.0.0.1`, which is the same as `localhost`. ~~As above, I will try make this clearer and more consistent.~~ Edit: I have updated the API documentation to use `localhost`, even though this is a guess as I cannot know how Mailpit is configured ~ localhost is the default displayed in the console through. > * Since the entire chaos feature is probably only relevant for people who know SMTP, I would rename the triggers to use the actual SMTP command names (and also change their order in the examples): > { > "AUTH": { > "ErrorCode": 451, > "Probability": 5 > }, > "MAIL": { > "ErrorCode": 451, > "Probability": 5 > }, > "RCPT": { > "ErrorCode": 451, > "Probability": 5 > } > } Ironically, I had it as `AUTH`, `MAIL` and `RCPT` right up until the last minute before committing - but figured it may cause confusion (plus I needed to then explain each step in the web UI). In the end I went with the more human-readable object keys. Too late to change now, the feature is in the API and changing that would be a breaking change. > * Does the response to PUT contain a trailing `\n`? I'm getting: > ``` > - Expected | + Actual > @@ @@ > -'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}}' > +'{"Sender":{"ErrorCode":451,"Probability":0},"Recipient":{"ErrorCode":451,"Probability":0},"Authentication":{"ErrorCode":535,"Probability":0}} > +' > ``` Quite possibly - I don't format the HTTP response manually, it's using Go's built-in JSON encoding to turn a struct (of the current config) into a JSON string. On that note, JSON should never be validated as a 1:1 string, as it it is prone to formatting differences (eg: "pretty"), and the fact that the order of keys can differ per implementation (Go will set them alphabetically, but PHP for instance won't). A JSON string should be converted to JSON and then parsed, and a JSON string with or without trailing lines is valid.

kerem commented

2026-03-15 13:32:30 +03:00

Author

Owner

@phm46 commented on GitHub (Jan 29, 2025):

Hey guys,
it was a pleasure to read your polite and constructive discussion. It’s great to see this advanced and well-thought-out feature integrated in Mailpit - thank you for that!

When reading documentation for it, I’ve just noticed one small typo.

When an error is triggered, Mailpit’s SMTP server will return a <code> Chaos <trigger> error, where code is the configured error code, and stage is the stage which triggered the error, for example 451 Chaos recipient error.

Everywhere you are using the word “trigger”, but in this text you’ve replaced it with “stage”. I’ve got it, but it may be confusing for other users.
Thank you again for bringing this feature alive!

@phm46 commented on GitHub (Jan 29, 2025): Hey guys, it was a pleasure to read your polite and constructive discussion. It’s great to see this advanced and well-thought-out feature integrated in Mailpit - thank you for that! When reading documentation for it, I’ve just noticed one small typo. ``` When an error is triggered, Mailpit’s SMTP server will return a <code> Chaos <trigger> error, where code is the configured error code, and stage is the stage which triggered the error, for example 451 Chaos recipient error. ``` Everywhere you are using the word “trigger”, but in this text you’ve replaced it with “stage”. I’ve got it, but it may be confusing for other users. Thank you again for bringing this feature alive!

kerem commented

2026-03-15 13:32:35 +03:00

Author

Owner

@axllent commented on GitHub (Jan 29, 2025):

Thank you for the heads-up @phm46, I have fixed that typo in the documentation. Hopefully the documentation now makes more sense to you, and of course that you are able to use Chaos for what you need.

@axllent commented on GitHub (Jan 29, 2025): Thank you for the heads-up @phm46, I have fixed that typo in the documentation. Hopefully the documentation now makes more sense to you, and of course that you are able to use Chaos for what you need.

Rows
Columns

[GH-ISSUE #402] Testing SMTP Server Error Responses #263