[GH-ISSUE #1033] SMTP with custom CA certificate #717

Closed
opened 2026-02-25 23:43:21 +03:00 by kerem · 9 comments
Owner

Originally created by @jlssmt on GitHub (Jul 25, 2024).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/1033

Hello,

I want to self-host healthchecks.
I get the

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

error.
Can anyone please help me to set up healthchecks with a custom CA?
Is it possible or do I need django-ca?

My docker-compose looks like this:

services:
  healthchecks:
    image: healthchecks/healthchecks:latest
    container_name: healthchecks
    environment:
      - DB=sqlite
      - DB_NAME=/data/hc.sqlite
      - DEBUG=True
      - DEFAULT_FROM_EMAIL=XXXXXXXXX
      - EMAIL_HOST=XXXXXXXXX
      - EMAIL_HOST_PASSWORD=XXXXXXXXX
      - EMAIL_HOST_USER=XXXXXXXXX
      - EMAIL_PORT=25
      - EMAIL_USE_TLS=True
      - EMAIL_SSL_CERTFILE=/certificates/test.pem
      - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
      - SECRET_KEY=XXXXXXXXX
      - SITE_ROOT=http://localhost:8000
    ports:
      - 8000:8000
    volumes:
      - ./data:/data
      - ./certificates:/certificates
    restart: unless-stopped

Thanks in advance

Originally created by @jlssmt on GitHub (Jul 25, 2024). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/1033 Hello, I want to self-host healthchecks. I get the ```python ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000) ``` error. Can anyone please help me to set up healthchecks with a custom CA? Is it possible or do I need django-ca? My docker-compose looks like this: ```yaml services: healthchecks: image: healthchecks/healthchecks:latest container_name: healthchecks environment: - DB=sqlite - DB_NAME=/data/hc.sqlite - DEBUG=True - DEFAULT_FROM_EMAIL=XXXXXXXXX - EMAIL_HOST=XXXXXXXXX - EMAIL_HOST_PASSWORD=XXXXXXXXX - EMAIL_HOST_USER=XXXXXXXXX - EMAIL_PORT=25 - EMAIL_USE_TLS=True - EMAIL_SSL_CERTFILE=/certificates/test.pem - REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt - SECRET_KEY=XXXXXXXXX - SITE_ROOT=http://localhost:8000 ports: - 8000:8000 volumes: - ./data:/data - ./certificates:/certificates restart: unless-stopped ``` Thanks in advance
kerem closed this issue 2026-02-25 23:43:21 +03:00
Author
Owner

@cuu508 commented on GitHub (Jul 26, 2024):

Hello @jlssmt, what are you doing when you get this error?

Where does this error appear?

Is there additional context around it (a traceback perhaps?)?

<!-- gh-comment-id:2252654374 --> @cuu508 commented on GitHub (Jul 26, 2024): Hello @jlssmt, what are you doing when you get this error? Where does this error appear? Is there additional context around it (a traceback perhaps?)?
Author
Owner

@jlssmt commented on GitHub (Jul 26, 2024):

I'm trying to sign up via mail.
Error happens when my instance is trying to send a mail.

stacktrace:

healthchecks  | [pid: 9|app: 0|req: 1/1] 10.123.123.115 () {34 vars in 777 bytes} [Fri Jul 26 12:26:46 2024] GET / => generated 0 bytes in 225 msecs (HTTP/1.1 302) 8 headers in 250 bytes (1 switches on core 0)
healthchecks  | [pid: 10|app: 0|req: 1/2] 10.123.123.115 () {34 vars in 807 bytes} [Fri Jul 26 12:26:46 2024] GET /accounts/login/ => generated 2010 bytes in 245 msecs (HTTP/1.1 200) 9 headers in 399 bytes (1 switches on core 0)
healthchecks  | [pid: 11|app: 0|req: 1/3] 10.123.123.115 () {36 vars in 818 bytes} [Fri Jul 26 12:26:47 2024] GET /static/img/logo.png => generated 0 bytes in 2 msecs (HTTP/1.1 304) 5 headers in 190 bytes (0 switches on core 0)
healthchecks  | [pid: 12|app: 0|req: 1/4] 10.123.123.115 () {34 vars in 706 bytes} [Fri Jul 26 12:26:55 2024] GET /accounts/signup/csrf/ => generated 64 bytes in 166 msecs (HTTP/1.1 200) 8 headers in 356 bytes (1 switches on core 0)
healthchecks  | [pid: 9|app: 0|req: 2/5] 10.123.123.115 () {40 vars in 839 bytes} [Fri Jul 26 12:26:55 2024] POST /accounts/signup/ => generated 54 bytes in 588 msecs (HTTP/1.1 200) 8 headers in 331 bytes (1 switches on core 0)
healthchecks  | Exception in thread Thread-1:
healthchecks  | Traceback (most recent call last):
healthchecks  |   File "/usr/local/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
healthchecks  |     self.run()
healthchecks  |   File "/opt/healthchecks/hc/lib/emails.py", line 26, in run
healthchecks  |     self.message.send()
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/core/mail/message.py", line 301, in send
healthchecks  |     return self.get_connection(fail_silently).send_messages([self])
healthchecks  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
healthchecks  |     new_conn_created = self.open()
healthchecks  |                        ^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 93, in open
healthchecks  |     self.connection.starttls(context=self.ssl_context)
healthchecks  |   File "/usr/local/lib/python3.12/smtplib.py", line 779, in starttls
healthchecks  |     self.sock = context.wrap_socket(self.sock,
healthchecks  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
healthchecks  |     return self.sslsocket_class._create(
healthchecks  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
healthchecks  |     self.do_handshake()
healthchecks  |   File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
healthchecks  |     self._sslobj.do_handshake()
healthchecks  | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)
<!-- gh-comment-id:2252662638 --> @jlssmt commented on GitHub (Jul 26, 2024): I'm trying to sign up via mail. Error happens when my instance is trying to send a mail. stacktrace: ```bash healthchecks | [pid: 9|app: 0|req: 1/1] 10.123.123.115 () {34 vars in 777 bytes} [Fri Jul 26 12:26:46 2024] GET / => generated 0 bytes in 225 msecs (HTTP/1.1 302) 8 headers in 250 bytes (1 switches on core 0) healthchecks | [pid: 10|app: 0|req: 1/2] 10.123.123.115 () {34 vars in 807 bytes} [Fri Jul 26 12:26:46 2024] GET /accounts/login/ => generated 2010 bytes in 245 msecs (HTTP/1.1 200) 9 headers in 399 bytes (1 switches on core 0) healthchecks | [pid: 11|app: 0|req: 1/3] 10.123.123.115 () {36 vars in 818 bytes} [Fri Jul 26 12:26:47 2024] GET /static/img/logo.png => generated 0 bytes in 2 msecs (HTTP/1.1 304) 5 headers in 190 bytes (0 switches on core 0) healthchecks | [pid: 12|app: 0|req: 1/4] 10.123.123.115 () {34 vars in 706 bytes} [Fri Jul 26 12:26:55 2024] GET /accounts/signup/csrf/ => generated 64 bytes in 166 msecs (HTTP/1.1 200) 8 headers in 356 bytes (1 switches on core 0) healthchecks | [pid: 9|app: 0|req: 2/5] 10.123.123.115 () {40 vars in 839 bytes} [Fri Jul 26 12:26:55 2024] POST /accounts/signup/ => generated 54 bytes in 588 msecs (HTTP/1.1 200) 8 headers in 331 bytes (1 switches on core 0) healthchecks | Exception in thread Thread-1: healthchecks | Traceback (most recent call last): healthchecks | File "/usr/local/lib/python3.12/threading.py", line 1073, in _bootstrap_inner healthchecks | self.run() healthchecks | File "/opt/healthchecks/hc/lib/emails.py", line 26, in run healthchecks | self.message.send() healthchecks | File "/usr/local/lib/python3.12/site-packages/django/core/mail/message.py", line 301, in send healthchecks | return self.get_connection(fail_silently).send_messages([self]) healthchecks | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages healthchecks | new_conn_created = self.open() healthchecks | ^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 93, in open healthchecks | self.connection.starttls(context=self.ssl_context) healthchecks | File "/usr/local/lib/python3.12/smtplib.py", line 779, in starttls healthchecks | self.sock = context.wrap_socket(self.sock, healthchecks | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket healthchecks | return self.sslsocket_class._create( healthchecks | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create healthchecks | self.do_handshake() healthchecks | File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake healthchecks | self._sslobj.do_handshake() healthchecks | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000) ```
Author
Owner

@cuu508 commented on GitHub (Jul 26, 2024):

Your docker-compose file sets EMAIL_SSL_CERTFILE env var, but I don't think it gets passed through to Django.

The env vars are loaded into Django settings in the settings.py file here, it goes through a fixed set of env vars and EMAIL_SSL_CERTFILE is not one of them.

I can add EMAIL_SSL_CERTFILE in settings.py if we know it does fix the issue. As an experiment, could you try the following:

  1. on the host system, create a file local_settings.py with contents:
EMAIL_SSL_CERTFILE=/certificates/test.pem
  1. in docker-compose.yml, under volumes add:
- path/to/local_settings.py:/opt/healthchecks/hc/local_settings.py
<!-- gh-comment-id:2252683951 --> @cuu508 commented on GitHub (Jul 26, 2024): Your docker-compose file sets `EMAIL_SSL_CERTFILE` env var, but I don't think it gets passed through to Django. The env vars are loaded into Django settings [in the settings.py file here](https://github.com/healthchecks/healthchecks/blob/master/hc/settings.py#L228), it goes through a fixed set of env vars and `EMAIL_SSL_CERTFILE` is not one of them. I can add `EMAIL_SSL_CERTFILE` in settings.py if we know it does fix the issue. As an experiment, could you try the following: 1. on the host system, create a file `local_settings.py` with contents: ``` EMAIL_SSL_CERTFILE=/certificates/test.pem ``` 2. in docker-compose.yml, under volumes add: ``` - path/to/local_settings.py:/opt/healthchecks/hc/local_settings.py ```
Author
Owner

@jlssmt commented on GitHub (Jul 26, 2024):

i tried
EMAIL_SSL_CERTFILE = "/certificates/test.pem"

i think it's working. but i get a different error now. but i don't know if this is related to the healthchecks app...

healthchecks  | [pid: 9|app: 0|req: 1/1] 10.123.123.115 () {34 vars in 720 bytes} [Fri Jul 26 12:51:23 2024] GET /accounts/signup/csrf/ => generated 64 bytes in 192 msecs (HTTP/1.1 200) 8 headers in 356 bytes (1 switches on core 0)
healthchecks  | [pid: 10|app: 0|req: 1/2] 10.123.123.115 () {40 vars in 853 bytes} [Fri Jul 26 12:51:23 2024] POST /accounts/signup/ => generated 54 bytes in 728 msecs (HTTP/1.1 200) 8 headers in 331 bytes (1 switches on core 0)
healthchecks  | Exception in thread Thread-1:
healthchecks  | Traceback (most recent call last):
healthchecks  |   File "/usr/local/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
healthchecks  |     self.run()
healthchecks  |   File "/opt/healthchecks/hc/lib/emails.py", line 26, in run
healthchecks  |     self.message.send()
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/core/mail/message.py", line 301, in send
healthchecks  |     return self.get_connection(fail_silently).send_messages([self])
healthchecks  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
healthchecks  |     new_conn_created = self.open()
healthchecks  |                        ^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 93, in open
healthchecks  |     self.connection.starttls(context=self.ssl_context)
healthchecks  |                                      ^^^^^^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/utils/functional.py", line 47, in __get__
healthchecks  |     res = instance.__dict__[self.name] = self.func(instance)
healthchecks  |                                          ^^^^^^^^^^^^^^^^^^^
healthchecks  |   File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 63, in ssl_context
healthchecks  |     ssl_context.load_cert_chain(self.ssl_certfile, self.ssl_keyfile)
healthchecks  | ssl.SSLError: [SSL] PEM lib (_ssl.c:3916)
<!-- gh-comment-id:2252705381 --> @jlssmt commented on GitHub (Jul 26, 2024): i tried `EMAIL_SSL_CERTFILE = "/certificates/test.pem"` i think it's working. but i get a different error now. but i don't know if this is related to the healthchecks app... ```bash healthchecks | [pid: 9|app: 0|req: 1/1] 10.123.123.115 () {34 vars in 720 bytes} [Fri Jul 26 12:51:23 2024] GET /accounts/signup/csrf/ => generated 64 bytes in 192 msecs (HTTP/1.1 200) 8 headers in 356 bytes (1 switches on core 0) healthchecks | [pid: 10|app: 0|req: 1/2] 10.123.123.115 () {40 vars in 853 bytes} [Fri Jul 26 12:51:23 2024] POST /accounts/signup/ => generated 54 bytes in 728 msecs (HTTP/1.1 200) 8 headers in 331 bytes (1 switches on core 0) healthchecks | Exception in thread Thread-1: healthchecks | Traceback (most recent call last): healthchecks | File "/usr/local/lib/python3.12/threading.py", line 1073, in _bootstrap_inner healthchecks | self.run() healthchecks | File "/opt/healthchecks/hc/lib/emails.py", line 26, in run healthchecks | self.message.send() healthchecks | File "/usr/local/lib/python3.12/site-packages/django/core/mail/message.py", line 301, in send healthchecks | return self.get_connection(fail_silently).send_messages([self]) healthchecks | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages healthchecks | new_conn_created = self.open() healthchecks | ^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 93, in open healthchecks | self.connection.starttls(context=self.ssl_context) healthchecks | ^^^^^^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/site-packages/django/utils/functional.py", line 47, in __get__ healthchecks | res = instance.__dict__[self.name] = self.func(instance) healthchecks | ^^^^^^^^^^^^^^^^^^^ healthchecks | File "/usr/local/lib/python3.12/site-packages/django/core/mail/backends/smtp.py", line 63, in ssl_context healthchecks | ssl_context.load_cert_chain(self.ssl_certfile, self.ssl_keyfile) healthchecks | ssl.SSLError: [SSL] PEM lib (_ssl.c:3916) ```
Author
Owner

@jlssmt commented on GitHub (Jul 26, 2024):

I don't understand it 😆 😢
It seems for me that Django cannot verify my passed public cert and I don't know why...

I'm passing a public cert in pem format.
I do not have a private cert because the public cert is signed from our company CA and not self signed.
So I cannot pass EMAIL_SSL_KEYFILE.

But Django seems to accept also certfile without keyfile:
github.com/django/django@1b277b45cc/django/core/mail/backends/smtp.py (L61)

maybe django-ca is needed for my setup?

<!-- gh-comment-id:2252783125 --> @jlssmt commented on GitHub (Jul 26, 2024): I don't understand it 😆 😢 It seems for me that Django cannot verify my passed public cert and I don't know why... I'm passing a public cert in pem format. I do not have a private cert because the public cert is signed from our company CA and not self signed. So I cannot pass EMAIL_SSL_KEYFILE. But Django seems to accept also certfile without keyfile: https://github.com/django/django/blob/1b277b45cc4059760072095f3bd6e8a4e4c4d406/django/core/mail/backends/smtp.py#L61 maybe django-ca is needed for my setup?
Author
Owner

@cuu508 commented on GitHub (Jul 26, 2024):

I'm not sure, but I think EMAIL_SSL_KEYFILE is for specifying client certificate, and so requires keyfile.

Django accepts certfile because the certificate and the key can be combined in one file:

https://docs.python.org/3/library/ssl.html#combined-key-and-certificate

Often the private key is stored in the same file as the certificate; in this case, only the certfile parameter to SSLContext.load_cert_chain() needs to be passed. If the private key is stored with the certificate, it should come before the first certificate in the certificate chain

What you need to do instead is to specify certificates that the client is willing to trust and the ssl module has load_verify_locations method for that:

https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations

Load a set of “certification authority” (CA) certificates used to validate other peers’ certificates when verify_mode is other than CERT_NONE. At least one of cafile or capath must be specified.

But it doesn't look like Django's EmailBackend has a way to use that.

<!-- gh-comment-id:2252895575 --> @cuu508 commented on GitHub (Jul 26, 2024): I'm not sure, but I *think* `EMAIL_SSL_KEYFILE` is for specifying client certificate, and so requires keyfile. Django accepts certfile because the certificate and the key can be combined in one file: https://docs.python.org/3/library/ssl.html#combined-key-and-certificate > Often the private key is stored in the same file as the certificate; in this case, only the certfile parameter to [SSLContext.load_cert_chain()](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_cert_chain) needs to be passed. If the private key is stored with the certificate, it should come before the first certificate in the certificate chain What you need to do instead is to specify certificates that the client is willing to trust and the ssl module has `load_verify_locations` method for that: https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations > Load a set of “certification authority” (CA) certificates used to validate other peers’ certificates when [verify_mode](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.verify_mode) is other than [CERT_NONE](https://docs.python.org/3/library/ssl.html#ssl.CERT_NONE). At least one of cafile or capath must be specified. But it doesn't look like Django's EmailBackend has a way to use that.
Author
Owner

@cuu508 commented on GitHub (Jul 26, 2024):

Not sure if it would work but perhaps you could prepare a custom ca-certificates.crt file which includes your company CA cert, and then tell docker-compose to mount it in /etc/ssl/certs/ca-certificates.crt. Django will then use system's default CA certificates, but they will include your certificate too.

<!-- gh-comment-id:2252922161 --> @cuu508 commented on GitHub (Jul 26, 2024): Not sure if it would work but perhaps you could prepare a custom `ca-certificates.crt` file which includes your company CA cert, and then tell docker-compose to mount it in `/etc/ssl/certs/ca-certificates.crt`. Django will then use system's default CA certificates, but they will include your certificate too.
Author
Owner

@jlssmt commented on GitHub (Jul 29, 2024):

Nailed it

Mount:
- ./certificates/custom-CA.pem:/usr/local/share/ca-certificates/custom-CA.pem.crt:ro

Run:
update-ca-certificates

<!-- gh-comment-id:2256107411 --> @jlssmt commented on GitHub (Jul 29, 2024): Nailed it Mount: `- ./certificates/custom-CA.pem:/usr/local/share/ca-certificates/custom-CA.pem.crt:ro` Run: `update-ca-certificates`
Author
Owner

@cuu508 commented on GitHub (Aug 2, 2024):

Good stuff! Marking this as resolved.

<!-- gh-comment-id:2264750177 --> @cuu508 commented on GitHub (Aug 2, 2024): Good stuff! Marking this as resolved.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/healthchecks#717
No description provided.