[PR #1839] [CLOSED] Openssl v3 compability issues with pkcs8 key #2643

New issue

Closed

opened 2026-03-16 11:00:18 +03:00 by kerem · 0 comments

kerem commented

2026-03-16 11:00:18 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hickory-dns/hickory-dns/pull/1839
Author: @darnuria
Created: 11/26/2022
Status: ❌ Closed

Base: main ← Head: opensslv3/compat/pkcs8

📝 Commits (2)

3d22cdd Add Pkcs8 for reading RSA key in Pem format.
d4103fb Document explicitly format accepted byrustls side read_key.

📊 Changes

1 file changed (+27 additions, -8 deletions)

View changed files

📝 crates/proto/src/rustls/tls_server.rs (+27 -8)

📄 Description

Review note: I am really outside of my comfort zone on this one, I am not by any mean an openSSL expert, it's just what I found by poking around error in CI and new OpenSSL v3 on my computer.

Hello I investigated failure In CI and tried to regenerate test certificate with openSSL v3 OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022 on my Linux.

So I did this changes to script/gen_certs.sh since just don't worked with openssl v3.

What it does?

removed -verify because It generated no sense error of unknown algorithm (not sure if it was wise)
use genpkey since genrsa is marked deprecated

 echo "----> Generating CA <----"
-${OPENSSL:?} genrsa -out ca.key 4096
-${OPENSSL:?} req -x509 -new -nodes -key ca.key -days 365 -out ca.pem -verify -config /tmp/ca.conf
+${OPENSSL:?} genpkey -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ca.key
+${OPENSSL:?} req -x509 -new -nodes -key ca.key -days 365 -out ca.pem -config /tmp/ca.conf

And later:

 echo "----> Generating CERT  <----"
-${OPENSSL:?} genrsa -out cert-key.pem 4096
+${OPENSSL:?} genpkey -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out cert-key.pem

And get this error:

cargo make all-features
[...]
failures:

---- quic::tests::test_quic_stream stdout ----
using server src path: /home/darnuria/programmation/trust-dns
thread 'quic::tests::test_quic_stream' panicked at 'called `Result::unwrap()` on an `Err` value: ProtoError { kind: Msg("No RSA keys in file: /home/darnuria/programmation/trust-dns/tests/test-data/cert-key.pem"), backtrack: None }', crates/proto/src/quic/tests.rs:68:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    quic::tests::test_quic_stream

After investigation I found that openssl V3 genrsa and genpkey issue now pkcs8 pem RSA pem key there is an option to revert that from openssl-genrsa man:

       openssl-genrsa - generate an RSA private key
[...]

       This command has been deprecated.  The openssl-genpkey(1) command should be used instead.

       This command generates an RSA private key.
[...]
       -traditional
           Write the key using the traditional PKCS#1 format instead of the PKCS#8 format.

So I went down to the function causing problems with the CI after regenerating certificates rustls side.

First I wondered that it was problem rustls_pemfile side so I opened first a PR there: https://github.com/rustls/pemfile/pull/10 but, afterthought I closed it in favor of this PR.

CI ISSUEs still not solved:

For Openssl side I suppose the problem is that the cert/key created with an older Openssl and read with a newer cause some issue I have to dig on this point, just regenerating cert/key/ is not satisfying.

And rustls side it's something to be 'future' proof by accepting pkcs8 key (what this PR try to do)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hickory-dns/hickory-dns/pull/1839 **Author:** [@darnuria](https://github.com/darnuria) **Created:** 11/26/2022 **Status:** ❌ Closed **Base:** `main` ← **Head:** `opensslv3/compat/pkcs8` --- ### 📝 Commits (2) - [`3d22cdd`](https://github.com/hickory-dns/hickory-dns/commit/3d22cdd0d5750ca9300b008653789b51c2329efa) Add Pkcs8 for reading RSA key in Pem format. - [`d4103fb`](https://github.com/hickory-dns/hickory-dns/commit/d4103fb39fdb097424311af25952e36a9e665572) Document explicitly format accepted byrustls side read_key. ### 📊 Changes **1 file changed** (+27 additions, -8 deletions) <details> <summary>View changed files</summary> 📝 `crates/proto/src/rustls/tls_server.rs` (+27 -8) </details> ### 📄 Description Review note: I am really outside of my comfort zone on this one, I am not by any mean an openSSL expert, it's just what I found by poking around error in CI and new OpenSSL v3 on my computer. Hello I investigated failure In CI and tried to regenerate test certificate with openSSL v3 `OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022` on my Linux. So I did this changes to `script/gen_certs.sh` since just don't worked with openssl v3. What it does? - removed `-verify` because It generated no sense error of unknown algorithm (not sure if it was wise) - use `genpkey` since `genrsa` is marked deprecated ```diff echo "----> Generating CA <----" -${OPENSSL:?} genrsa -out ca.key 4096 -${OPENSSL:?} req -x509 -new -nodes -key ca.key -days 365 -out ca.pem -verify -config /tmp/ca.conf +${OPENSSL:?} genpkey -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ca.key +${OPENSSL:?} req -x509 -new -nodes -key ca.key -days 365 -out ca.pem -config /tmp/ca.conf ``` And later: ```diff echo "----> Generating CERT <----" -${OPENSSL:?} genrsa -out cert-key.pem 4096 +${OPENSSL:?} genpkey -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out cert-key.pem ``` And get this error: ``` cargo make all-features [...] failures: ---- quic::tests::test_quic_stream stdout ---- using server src path: /home/darnuria/programmation/trust-dns thread 'quic::tests::test_quic_stream' panicked at 'called `Result::unwrap()` on an `Err` value: ProtoError { kind: Msg("No RSA keys in file: /home/darnuria/programmation/trust-dns/tests/test-data/cert-key.pem"), backtrack: None }', crates/proto/src/quic/tests.rs:68:6 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: quic::tests::test_quic_stream ``` After investigation I found that openssl V3 `genrsa` and `genpkey` issue now pkcs8 pem RSA pem key there is an option to revert that from `openssl-genrsa` man: ``` openssl-genrsa - generate an RSA private key [...] This command has been deprecated. The openssl-genpkey(1) command should be used instead. This command generates an RSA private key. [...] -traditional Write the key using the traditional PKCS#1 format instead of the PKCS#8 format. ``` So I went down to the function causing problems with the CI after regenerating certificates rustls side. First I wondered that it was problem `rustls_pemfile` side so I opened first a PR there: https://github.com/rustls/pemfile/pull/10 but, afterthought I closed it in favor of this PR. ## CI ISSUEs still not solved: For Openssl side I suppose the problem is that the cert/key created with an older Openssl and read with a newer cause some issue I have to dig on this point, just regenerating cert/key/ is not satisfying. And rustls side it's something to be 'future' proof by accepting pkcs8 key (what this PR try to do) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>