[GH-ISSUE #11] Unable to get local issuer certificate when the server is using Letsencrypt #12

Closed
opened 2026-02-27 21:01:11 +03:00 by kerem · 23 comments
Owner

Originally created by @Naolador on GitHub (Feb 23, 2021).
Original GitHub issue: https://github.com/sg4r/proxmox-backup-client/issues/11

OS: CentOS 7
Client version: 1.0.6

When I was trying to run the backup, it returns the following error:

Error: error trying to connect: the handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate

I think therer's something wrong in the statem_clnt.c file when trying to read the certificate. Here's the file:


# less /root/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-src-111.14.0+1.1.1j/openssl/ssl/statem/statem_clnt.c
...
   1898     /*
   1899      * The documented interface is that SSL_VERIFY_PEER should be set in order
   1900      * for client side verification of the server certificate to take place.
   1901      * However, historically the code has only checked that *any* flag is set
   1902      * to cause server verification to take place. Use of the other flags makes
   1903      * no sense in client mode. An attempt to clean up the semantics was
   1904      * reverted because at least one application *only* set
   1905      * SSL_VERIFY_FAIL_IF_NO_PEER_CERT. Prior to the clean up this still caused
   1906      * server verification to take place, after the clean up it silently did
   1907      * nothing. SSL_CTX_set_verify()/SSL_set_verify() cannot validate the flags
   1908      * sent to them because they are void functions. Therefore, we now use the
   1909      * (less clean) historic behaviour of performing validation if any flag is
   1910      * set. The *documented* interface remains the same.
   1911      */
   1912     if (s->verify_mode != SSL_VERIFY_NONE && i <= 0) {
   1913         SSLfatal(s, ssl_x509err2alert(s->verify_result),
   1914                  SSL_F_TLS_PROCESS_SERVER_CERTIFICATE,
   1915                  SSL_R_CERTIFICATE_VERIFY_FAILED);
   1916         goto err;
   1917     }
   1918     ERR_clear_error();          /* but we keep s->verify_result */
...

I tried to comment those lines out, but still giving the same error. I think it needs recompile?

Originally created by @Naolador on GitHub (Feb 23, 2021). Original GitHub issue: https://github.com/sg4r/proxmox-backup-client/issues/11 OS: CentOS 7 Client version: 1.0.6 When I was trying to run the backup, it returns the following error: ``` Error: error trying to connect: the handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate ``` I think therer's something wrong in the statem_clnt.c file when trying to read the certificate. Here's the file: ``` # less /root/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-src-111.14.0+1.1.1j/openssl/ssl/statem/statem_clnt.c ... 1898 /* 1899 * The documented interface is that SSL_VERIFY_PEER should be set in order 1900 * for client side verification of the server certificate to take place. 1901 * However, historically the code has only checked that *any* flag is set 1902 * to cause server verification to take place. Use of the other flags makes 1903 * no sense in client mode. An attempt to clean up the semantics was 1904 * reverted because at least one application *only* set 1905 * SSL_VERIFY_FAIL_IF_NO_PEER_CERT. Prior to the clean up this still caused 1906 * server verification to take place, after the clean up it silently did 1907 * nothing. SSL_CTX_set_verify()/SSL_set_verify() cannot validate the flags 1908 * sent to them because they are void functions. Therefore, we now use the 1909 * (less clean) historic behaviour of performing validation if any flag is 1910 * set. The *documented* interface remains the same. 1911 */ 1912 if (s->verify_mode != SSL_VERIFY_NONE && i <= 0) { 1913 SSLfatal(s, ssl_x509err2alert(s->verify_result), 1914 SSL_F_TLS_PROCESS_SERVER_CERTIFICATE, 1915 SSL_R_CERTIFICATE_VERIFY_FAILED); 1916 goto err; 1917 } 1918 ERR_clear_error(); /* but we keep s->verify_result */ ... ``` I tried to comment those lines out, but still giving the same error. I think it needs recompile?
kerem closed this issue 2026-02-27 21:01:11 +03:00
Author
Owner

@sg4r commented on GitHub (Feb 23, 2021):

hi, you can use my first rpm release for Centos7 or Centos8
https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.6-beta
can you check it ?
the rpm file are in Assets from release

<!-- gh-comment-id:784287560 --> @sg4r commented on GitHub (Feb 23, 2021): hi, you can use my first rpm release for Centos7 or Centos8 https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.6-beta can you check it ? the rpm file are in Assets from release
Author
Owner

@Naolador commented on GitHub (Feb 23, 2021):

Hi @sg4r I'm still getting the same error. I'm using fullchain on the pbs server for the cert file as suggested by proxmox. Is that working on your machine?

<!-- gh-comment-id:784500443 --> @Naolador commented on GitHub (Feb 23, 2021): Hi @sg4r I'm still getting the same error. I'm using fullchain on the pbs server for the cert file as suggested by proxmox. Is that working on your machine?
Author
Owner

@sg4r commented on GitHub (Feb 24, 2021):

hi, sorry, I am not using Letsencrypt certificates.
we should check if the problem also exists from a debian client, and if so, report the problem to the pbs developer.
if the problem is only for the CentOS client, it would be necessary to review the part of the code to fix it.

<!-- gh-comment-id:784826245 --> @sg4r commented on GitHub (Feb 24, 2021): hi, sorry, I am not using Letsencrypt certificates. we should check if the problem also exists from a debian client, and if so, report the problem to the pbs developer. if the problem is only for the CentOS client, it would be necessary to review the part of the code to fix it.
Author
Owner

@Naolador commented on GitHub (Feb 25, 2021):

It works on Debian10, so I think it's the client code issue.

<!-- gh-comment-id:785687873 --> @Naolador commented on GitHub (Feb 25, 2021): It works on Debian10, so I think it's the client code issue.
Author
Owner

@sg4r commented on GitHub (Feb 25, 2021):

Thank you for the feedback.
is it possible to indicate the version of the client under debian ?

<!-- gh-comment-id:785694504 --> @sg4r commented on GitHub (Feb 25, 2021): Thank you for the feedback. is it possible to indicate the version of the client under debian ?
Author
Owner

@Naolador commented on GitHub (Feb 25, 2021):

1.0.8

<!-- gh-comment-id:785698465 --> @Naolador commented on GitHub (Feb 25, 2021): 1.0.8
Author
Owner

@sg4r commented on GitHub (Feb 25, 2021):

ok, i will look to support version 1.0.8 or higher

<!-- gh-comment-id:785702568 --> @sg4r commented on GitHub (Feb 25, 2021): ok, i will look to support version 1.0.8 or higher
Author
Owner

@Naolador commented on GitHub (Feb 25, 2021):

Much appreciated!

<!-- gh-comment-id:785707398 --> @Naolador commented on GitHub (Feb 25, 2021): Much appreciated!
Author
Owner

@sg4r commented on GitHub (Apr 5, 2021):

hi, it's corrected.
can you test build and client version v1.0.11 please

<!-- gh-comment-id:813548146 --> @sg4r commented on GitHub (Apr 5, 2021): hi, it's corrected. can you test build and client version v1.0.11 please
Author
Owner

@Naolador commented on GitHub (Apr 6, 2021):

Hi @sg4r the rpm can be installed, however, it's complaining glibc 2.18 missing due to CentOS7 doesn't support this version.
Tried this way https://serverfault.com/questions/894625/safely-upgrade-glibc-on-centos-7 doesn't work, even by adding the variable before the command instead of export

<!-- gh-comment-id:813782771 --> @Naolador commented on GitHub (Apr 6, 2021): Hi @sg4r the rpm can be installed, however, it's complaining glibc 2.18 missing due to CentOS7 doesn't support this version. Tried this way https://serverfault.com/questions/894625/safely-upgrade-glibc-on-centos-7 doesn't work, even by adding the variable before the command instead of export
Author
Owner

@sg4r commented on GitHub (Apr 6, 2021):

hi,
Centos8 client can be found at https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.11
for Centos7, i must add support for OpenSSL 1.1.0
https://docs.rs/openssl/0.9.24/openssl/pkcs5/fn.scrypt.html
by default Centos7 have version 1.0 only
[centos@pbs proxmox-backup]$ openssl version -a
OpenSSL 1.0.2k-fips 26 Jan 2017
i need to check if it works with this https://cloudwafer.com/blog/installing-openssl-on-centos-7/

<!-- gh-comment-id:813865546 --> @sg4r commented on GitHub (Apr 6, 2021): hi, Centos8 client can be found at https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.11 for Centos7, i must add support for OpenSSL 1.1.0 https://docs.rs/openssl/0.9.24/openssl/pkcs5/fn.scrypt.html by default Centos7 have version 1.0 only [centos@pbs proxmox-backup]$ openssl version -a OpenSSL 1.0.2k-fips 26 Jan 2017 i need to check if it works with this https://cloudwafer.com/blog/installing-openssl-on-centos-7/
Author
Owner

@sg4r commented on GitHub (Apr 7, 2021):

hi, i have build version for centos7. rpm package are at https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.11
can you test client version v1.0.11 please ?

<!-- gh-comment-id:815037605 --> @sg4r commented on GitHub (Apr 7, 2021): hi, i have build version for centos7. rpm package are at https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.11 can you test client version v1.0.11 please ?
Author
Owner

@Naolador commented on GitHub (Apr 8, 2021):

hi @sg4r I replaced the new package you provided and it's not complaining glibc version mismatch anymore. However, I'm still facing the same SSL error as I posted at the beginning ☹
I'm wondering if it works on your testing environment with Let's encrypt enabled?

<!-- gh-comment-id:815362866 --> @Naolador commented on GitHub (Apr 8, 2021): hi @sg4r I replaced the new package you provided and it's not complaining glibc version mismatch anymore. However, I'm still facing the same SSL error as I posted at the beginning ☹ I'm wondering if it works on your testing environment with Let's encrypt enabled?
Author
Owner

@sg4r commented on GitHub (Apr 8, 2021):

hi, is it possible to indicate the recipe of the commands to use so that I can test this operation on my side.
on my platform I use the backup from clients under centos7 and proxmox-backup-client backup home.pxar:/home/ --repository root@pam@pbsserver:datastore
is it possible to tell me the procedure from the pbs documentation
Thank you

<!-- gh-comment-id:815453231 --> @sg4r commented on GitHub (Apr 8, 2021): hi, is it possible to indicate the recipe of the commands to use so that I can test this operation on my side. on my platform I use the backup from clients under centos7 and proxmox-backup-client backup home.pxar:/home/ --repository root@pam@pbsserver:datastore is it possible to tell me the procedure from the pbs documentation Thank you
Author
Owner

@Naolador commented on GitHub (Apr 9, 2021):

Hi, @sg4r I'm using the API token and trigger the backup with a script in the client. The backup script is something looks like this:

#!/bin/bash
export PBS_REPOSITORY=backup@pbs\!backup@proxbackup.example.com:my_backups
export PBS_PASSWORD=xxx-xxx-xxx-xxx-xxx
proxmox-backup-client backup root.pxar:/
<!-- gh-comment-id:816338585 --> @Naolador commented on GitHub (Apr 9, 2021): Hi, @sg4r I'm using the API token and trigger the backup with a script in the client. The backup script is something looks like this: ``` #!/bin/bash export PBS_REPOSITORY=backup@pbs\!backup@proxbackup.example.com:my_backups export PBS_PASSWORD=xxx-xxx-xxx-xxx-xxx proxmox-backup-client backup root.pxar:/ ```
Author
Owner

@eduardpalomo commented on GitHub (Apr 15, 2021):

hi, i have build version for centos7. rpm package are at https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.11
can you test client version v1.0.11 please ?

Hi @sg4r, I also have the same problem, before implementing the letsencrypt ssl everything worked fine, but when I started using letsencrypt on the server, the backups of the physical hosts stopped working with the TLS error

<!-- gh-comment-id:820391415 --> @eduardpalomo commented on GitHub (Apr 15, 2021): > hi, i have build version for centos7. rpm package are at https://github.com/sg4r/proxmox-backup-client/releases/tag/v1.0.11 > can you test client version v1.0.11 please ? Hi @sg4r, I also have the same problem, before implementing the letsencrypt ssl everything worked fine, but when I started using letsencrypt on the server, the backups of the physical hosts stopped working with the TLS error
Author
Owner

@wmorgue commented on GitHub (May 5, 2021):

CentOS 7.
Installed a https://github.com/sg4r/proxmox-backup-client/releases/download/v1.0.11/proxmox-backup-1.0.11-1.2.x86_64.el8.rpm and got error:

could not connect to server - error trying to connect:
error:1416F086:SSL routines:tls_process_server_certificate:
certificate verify failed:ssl/statem/statem_clnt.c:1915
openssl version -a
OpenSSL 1.0.2k-fips  26 Jan 2017
<!-- gh-comment-id:832548184 --> @wmorgue commented on GitHub (May 5, 2021): CentOS 7. Installed a `https://github.com/sg4r/proxmox-backup-client/releases/download/v1.0.11/proxmox-backup-1.0.11-1.2.x86_64.el8.rpm` and got error: ```bash could not connect to server - error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate: certificate verify failed:ssl/statem/statem_clnt.c:1915 ``` ```bash openssl version -a OpenSSL 1.0.2k-fips 26 Jan 2017 ```
Author
Owner

@Tim-NL commented on GitHub (May 5, 2021):

I installed this on CloudLinux 8 (RHEL 8 based) - So 1:1 compatible with CentOS 8 as well. And it worked great. My backup server uses Let's Encrypt for its certificates and I had no issues with connecting to it. All comments are about CentOS 7, but the latest builds did resolve the issue for me. I was not able to use it with Let's Encrypt on CentOS 8 in the past (a few weeks ago) but now it works great. So the latest changes did resolve some issues.

-Tim

<!-- gh-comment-id:832551239 --> @Tim-NL commented on GitHub (May 5, 2021): I installed this on CloudLinux 8 (RHEL 8 based) - So 1:1 compatible with CentOS 8 as well. And it worked great. My backup server uses Let's Encrypt for its certificates and I had no issues with connecting to it. All comments are about CentOS 7, but the latest builds did resolve the issue for me. I was not able to use it with Let's Encrypt on CentOS 8 in the past (a few weeks ago) but now it works great. So the latest changes did resolve some issues. -Tim
Author
Owner

@wmorgue commented on GitHub (May 5, 2021):

What version did you install for CentOS 7?
@Tim-NL

<!-- gh-comment-id:832558390 --> @wmorgue commented on GitHub (May 5, 2021): What version did you install for CentOS 7? @Tim-NL
Author
Owner

@Tim-NL commented on GitHub (May 5, 2021):

What version did you install for CentOS 7?
@Tim-NL

I did not use/try CentOS 7, only CentOS 8 (and CloudLinux 8/RHEL 8). - I just replied to let the author know that their changes did resolve the issue for me on CentOS 8.

-Tim

<!-- gh-comment-id:832570410 --> @Tim-NL commented on GitHub (May 5, 2021): > What version did you install for CentOS 7? > @Tim-NL I did not use/try CentOS 7, only CentOS 8 (and CloudLinux 8/RHEL 8). - I just replied to let the author know that their changes did resolve the issue for me on CentOS 8. -Tim
Author
Owner

@sg4r commented on GitHub (May 10, 2021):

hi , thanks for the feedback on this point. I'm not sure it's a code issue, but openssl certificate recognition
for REDHAT 7.x ou CentOS 7.x you can try yum reinstall ca-certificates or add the ca certificate in the simple PEM or DER file formats to the list of CAs trusted on the system:
Download Active PEM certificate from: https://letsencrypt.org/certificates/ to the /etc/pki/ca-trust/source/anchors/ subdirectory, and run the update-ca-trust command.

<!-- gh-comment-id:836260026 --> @sg4r commented on GitHub (May 10, 2021): hi , thanks for the feedback on this point. I'm not sure it's a code issue, but openssl certificate recognition for REDHAT 7.x ou CentOS 7.x you can try yum reinstall ca-certificates or add the ca certificate in the simple PEM or DER file formats to the list of CAs trusted on the system: Download Active PEM certificate from: https://letsencrypt.org/certificates/ to the /etc/pki/ca-trust/source/anchors/ subdirectory, and run the update-ca-trust command.
Author
Owner

@francescor commented on GitHub (Jan 15, 2022):

I tried the above, but lost a lot of time... the fix is the suggestion in https://github.com/sg4r/proxmox-backup-client/issues/16

<!-- gh-comment-id:1013660378 --> @francescor commented on GitHub (Jan 15, 2022): I tried the above, but lost a lot of time... the fix is the suggestion in https://github.com/sg4r/proxmox-backup-client/issues/16
Author
Owner

@sg4r commented on GitHub (Jan 23, 2022):

hi, you can find centos7 proxmox-backup-client version 2.1.2 rpm at https://github.com/sg4r/proxmox-backup-client/releases/tag/v2.1.2
you can check it ?

<!-- gh-comment-id:1019547814 --> @sg4r commented on GitHub (Jan 23, 2022): hi, you can find centos7 proxmox-backup-client version 2.1.2 rpm at https://github.com/sg4r/proxmox-backup-client/releases/tag/v2.1.2 you can check it ?
Sign in to join this conversation.
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/proxmox-backup-client#12
No description provided.