[GH-ISSUE #6416 ] Deleted Items stop login working #2427

Author

Owner

@tessus commented on GitHub (Oct 31, 2025):

Is this issue only a problem with the container image? After reading all the different issues starting with weiznich's link above it seems that this is always a problem when the mariadb connector lib is greater than 3.3.15.
e.g. Fedora 42 and 43 use mariadb-connector-c-3.4.5

So what is the solution? I compile vaultwarden on Fedora since my server runs Fedora.

@tessus commented on GitHub (Oct 31, 2025): Is this issue only a problem with the container image? After reading all the different issues starting with weiznich's link above it seems that this is always a problem when the mariadb connector lib is greater than 3.3.15. e.g. Fedora 42 and 43 use mariadb-connector-c-3.4.5 So what is the solution? I compile vaultwarden on Fedora since my server runs Fedora.

kerem commented

Author

Owner

@weiznich commented on GitHub (Nov 3, 2025):

@tessus There is no solution you as an end user can apply here other than not using the distro provided libmariadb version as it is not compatible with any diesel based application. You might want to notify the package maintainers of your distribution about this problem as well.

See https://github.com/diesel-rs/diesel/discussions/4806#discussioncomment-14857450 for more steps you can take.

@weiznich commented on GitHub (Nov 3, 2025): @tessus There is no solution you as an end user can apply here other than not using the distro provided libmariadb version as it is not compatible with any diesel based application. You might want to notify the package maintainers of your distribution about this problem as well. See https://github.com/diesel-rs/diesel/discussions/4806#discussioncomment-14857450 for more steps you can take.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 3, 2025):

@weiznich thanks for the reply. Although I am a bit fuzzy on the situation. Version 3.3.15 is 7 months old. Most distributions have already moved on to 3.4.z, thus - as you mentioned - the only 2 options are: 1) for distros to use a patch or 2) for the libmariadb devs to make the lib compatible with mysql again.

A fix for diesel seems complex, since one would then also have to check for library versions. Hmm. This might be a maintenance burden in the future.

I will also check, whether I can just use the mysql client lib or connector or whatever is required instead of mariadb-connector-c-devel to compile vw.

I can certainly open an issue with the Fedora packagers and bug the mariadb devs, but I doubt that the latter will have any effect on the betterment of the situation. In the ticket you referenced they were rather determined to ignore the truth and live with the inconsistencies in their code. You cannot convince people like that nor have a constructive discussion with them.

@tessus commented on GitHub (Nov 3, 2025): @weiznich thanks for the reply. Although I am a bit fuzzy on the situation. Version 3.3.15 is 7 months old. Most distributions have already moved on to 3.4.z, thus - as you mentioned - the only 2 options are: 1) for distros to use a patch or 2) for the libmariadb devs to make the lib compatible with mysql again. A fix for diesel seems complex, since one would then also have to check for library versions. Hmm. This might be a maintenance burden in the future. I will also check, whether I can just use the mysql client lib or connector or whatever is required instead of mariadb-connector-c-devel to compile vw. I can certainly open an issue with the Fedora packagers and bug the mariadb devs, but I doubt that the latter will have any effect on the betterment of the situation. In the ticket you referenced they were rather determined to ignore the truth and live with the inconsistencies in their code. You cannot convince people like that nor have a constructive discussion with them.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 4, 2025):

So... before trying to replace mariadb-connector-c with mysql-community-libs, I built the current main HEAD 9017ca265a on Fedora with mariadb-connector-c-3.4.5-1. I checked, they are not using any patches, so I should have received the same error as described in this issue.
On my test VM with a previous version of vw, I deleted a cipher and it was moved to the trash. Then I installed the new vw version, but was able to login. No errors in the log, no panic, no endless spinner in the webui.
Running select uuid, deleted_at from ciphers where deleted_at is not null confirmed that I had in fact 5 such entries in my database.

So now I am even more confused. I should have experienced the same issue as the OP. 🤔

@tessus commented on GitHub (Nov 4, 2025): So... before trying to replace `mariadb-connector-c` with `mysql-community-libs`, I built the current main HEAD 9017ca265a28ff1370e6047b8e85c0a6c3327306 on Fedora with `mariadb-connector-c-3.4.5-1`. I checked, they are not using any patches, so I should have received the same error as described in this issue. On my test VM with a previous version of vw, I deleted a cipher and it was moved to the trash. Then I installed the new vw version, but was able to login. No errors in the log, no panic, no endless spinner in the webui. Running `select uuid, deleted_at from ciphers where deleted_at is not null` confirmed that I had in fact 5 such entries in my database. So now I am even more confused. I should have experienced the same issue as the OP. 🤔

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 4, 2025):

@tessus, 3.4.5 is not affected. Only versions after that.

@BlackDex commented on GitHub (Nov 4, 2025): @tessus, 3.4.5 is not affected. Only versions after that.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 4, 2025):

@BlackDex thanks for the info. I guess I misunderstood https://github.com/diesel-rs/diesel/discussions/4806#discussioncomment-14814651 then, which mentioned downgrading to 3.3.15.

So I think I will work on trying to use the mysql lib instead. The problem is that mysql is not part of Fedora, which means creating a package that requires a lib from a non-Fedora repo makes it less portable/usable. This really sucks. On the other side, I am probably the only one using my Fedora vaultwarden package anyway. Well, first I have to make it work...

@tessus commented on GitHub (Nov 4, 2025): @BlackDex thanks for the info. I guess I misunderstood https://github.com/diesel-rs/diesel/discussions/4806#discussioncomment-14814651 then, which mentioned downgrading to 3.3.15. So I think I will work on trying to use the mysql lib instead. The problem is that mysql is not part of Fedora, which means creating a package that requires a lib from a non-Fedora repo makes it less portable/usable. This really sucks. On the other side, I am probably the only one using my Fedora vaultwarden package anyway. Well, first I have to make it work...

kerem commented

Author

Owner

@weiznich commented on GitHub (Nov 4, 2025):

A possible solution would be to explicitly add the mysqlclient-sys crate to the vaultwarden dependency list. Then you should be able to use cargo install vaultwarden -F mysql -F mysqlclient-sys/bundled to build and statically link a version of libmysqlclient that is known to work with diesel.

@weiznich commented on GitHub (Nov 4, 2025): A possible solution would be to explicitly add the `mysqlclient-sys` crate to the `vaultwarden` dependency list. Then you should be able to use `cargo install vaultwarden -F mysql -F mysqlclient-sys/bundled` to build and statically link a version of `libmysqlclient` that is known to work with diesel.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 4, 2025):

A possible solution would be to explicitly add the mysqlclient-sys crate to the vaultwarden dependency list. Then you should be able to use cargo install vaultwarden -F mysql -F mysqlclient-sys/bundled to build and statically link a version of libmysqlclient that is known to work with diesel.

That is what I'm thinking about as trying to install older packages is also not really nice and easy. It will make compiling take longer though, at least the first time and as long the cache doesn't gets busted.

@BlackDex commented on GitHub (Nov 4, 2025): > A possible solution would be to explicitly add the `mysqlclient-sys` crate to the `vaultwarden` dependency list. Then you should be able to use `cargo install vaultwarden -F mysql -F mysqlclient-sys/bundled` to build and statically link a version of `libmysqlclient` that is known to work with diesel. That is what I'm thinking about as trying to install older packages is also not really nice and easy. It will make compiling take longer though, at least the first time and as long the cache doesn't gets busted.

kerem commented

Author

Owner

@weiznich commented on GitHub (Nov 4, 2025):

I would suggest exposing this as optional feature so users could decide to use this or not. There are reasons to use one or the other variant. Using the system libraries allows you to easily update these libraries if necessary and overall better integrates in how linux distributions want to have their packages. On the other hand that means supporting several different versions of the library which can cause problems if the underlying is not carefully about changes itself. For deployment setups (docker/etc) is often easier to just statically link everything into one binary by using the bundled feature. Also as you already wrote: Buildtimes obviously suffer from this.

For building libmysqlclient-sys via the bundled feature it's required to have cmake + a working c/c++ compiler.

@weiznich commented on GitHub (Nov 4, 2025): I would suggest exposing this as optional feature so users could decide to use this or not. There are reasons to use one or the other variant. Using the system libraries allows you to easily update these libraries if necessary and overall better integrates in how linux distributions want to have their packages. On the other hand that means supporting several different versions of the library which can cause problems if the underlying is not carefully about changes itself. For deployment setups (docker/etc) is often easier to just statically link everything into one binary by using the bundled feature. Also as you already wrote: Buildtimes obviously suffer from this. For building `libmysqlclient-sys` via the bundled feature it's required to have `cmake` + a working c/c++ compiler.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 4, 2025):

My main problem currently is that we use Debian Trixie, which uses a faulty version, and now causes these issues for our Debian based containers.

@BlackDex commented on GitHub (Nov 4, 2025): My main problem currently is that we use Debian Trixie, which uses a faulty version, and now causes these issues for our Debian based containers.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 4, 2025):

A possible solution would be to explicitly add the mysqlclient-sys crate to the vaultwarden dependency list. Then you should be able to use cargo install vaultwarden -F mysql -F mysqlclient-sys/bundled to build and statically link a version of libmysqlclient that is known to work with diesel.

I can try that tomorrow. I just patch the Cargo.toml before building. Nice.

As for the build time: Mathijs' last improvement PR pushed it down from 18 minutes to 14 minutes on my build VMs. Also, I am not building every 5 minutes, but when there are interesting commits in main. And even if I were building a package every day, I wouldn't care whether it took 14 minutes or 60 minutes. As long as it didn't take more than 24h. ;-) hahaha.

That is what I'm thinking about as trying to install older packages is also not really nice and easy.

My thoughts exactly. I want to find a proper solution now instead of 6 months from now when older versions of the mariadb-connector are no longer available. That would require me to build my own connector and either statically link that one or build another connector package. I reckon I'd rather go with statically linking the mysql lib....

I will still open a ticket with the Fedora mariadb-connector packager.

@tessus commented on GitHub (Nov 4, 2025): > A possible solution would be to explicitly add the mysqlclient-sys crate to the vaultwarden dependency list. Then you should be able to use cargo install vaultwarden -F mysql -F mysqlclient-sys/bundled to build and statically link a version of libmysqlclient that is known to work with diesel. I can try that tomorrow. I just patch the Cargo.toml before building. Nice. As for the build time: Mathijs' last improvement PR pushed it down from 18 minutes to 14 minutes on my build VMs. Also, I am not building every 5 minutes, but when there are interesting commits in main. And even if I were building a package every day, I wouldn't care whether it took 14 minutes or 60 minutes. As long as it didn't take more than 24h. ;-) hahaha. > That is what I'm thinking about as trying to install older packages is also not really nice and easy. My thoughts exactly. I want to find a proper solution now instead of 6 months from now when older versions of the mariadb-connector are no longer available. That would require me to build my own connector and either statically link that one or build another connector package. I reckon I'd rather go with statically linking the mysql lib.... I will still open a ticket with the Fedora mariadb-connector packager.

kerem commented

Author

Owner

@weiznich commented on GitHub (Nov 4, 2025):

Such an feature could also be enabled by "default" with some way to disable it. Eg. by having the vaultwarden mysql feature enable the bundled feature, but also provide a mysql-unbundled feature that doesn't bundle the lib.

Other than that I see the following options:

Someone needs to submit a fix/revert for the broken behaviour to libmariadb and see what is happening then
I still think it might be worth to notify distro maintainers of that package about this problem, possibly we can get at least a fix/revert there?
Someone needs to dig through and workaround the issue there. Possibly by changing this construction to use the known size for statically sized types instead. Such a change would need to be rather careful to verify that the buffer has the correct size and is correctly initialized. Additionally I'm still skeptical to include such a workaround given that libmariadb did already ship such a breaking change once and they don't really seam to care about that. That doesn't increase the trust in them not doing something like that again.

@weiznich commented on GitHub (Nov 4, 2025): Such an feature could also be enabled by "default" with some way to disable it. Eg. by having the `vaultwarden` `mysql` feature enable the `bundled` feature, but also provide a `mysql-unbundled` feature that doesn't bundle the lib. Other than that I see the following options: * Someone needs to submit a fix/revert for the broken behaviour to libmariadb and see what is happening then * I still think it might be worth to notify distro maintainers of that package about this problem, possibly we can get at least a fix/revert there? * Someone needs to dig through and workaround the issue there. Possibly by changing [this](https://github.com/diesel-rs/diesel/blob/143b254c5b2cc49e2d7d52da69f68eec67a46d86/diesel/src/mysql/connection/bind.rs#L419) construction to use the known size for statically sized types instead. Such a change would need to be rather careful to verify that the buffer has the correct size and is correctly initialized. Additionally I'm still skeptical to include such a workaround given that libmariadb did already ship such a breaking change once and they don't really seam to care about that. That doesn't increase the trust in them not doing something like that again.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 4, 2025):

Here's my feedback. It worked mostly great, but interestingly I had to install 3 packages: libtirpc-devel, perl-Time-Piece, and ncurses-devel. (The cargo build process failed 3 times.) While I understand the first 2, ncurses makes no sense for a mysql lib nor for the build process. But it's not an issue, just strange.

Then it compiled successfully, but spat out a warning:

warning: the following packages contain code that will be rejected by a future version of Rust: num-bigint-dig v0.8.4

I updated with cargo update num-bigint-dig, but 0.8.5 yielded the same warning. It seems an additional fix has been added after 0.8.5 was released, so that should be solved soon.

The size of the binary has increased from 38 MB to 52 MB.

Digging a bit deeper it seems that -F mysqlclient-sys/bundled not only statically linked mysql, but also openssl. Hmmm. This is rather annoying and makes me slightly unhappy. But I guess I will have to live with it.

Another strange thing is that I removed all mariadb* packages before building vw with -F mysqlclient-sys/bundled and there was no libmysqlclient library (static or dynamic) on the system, because I forgot to install it after removing the mariadb packages. I ran a find on / and there was definitely not another one hiding somewhere else. Yet, I did not encounter a build error. Both crates mysqlclient-sys and mysqlclient-src explicitly state that the libmysqlclient library is required for building them.

I tested the binary and it works. I don't understand why and how, but it does.

The only explanation I could come up with is that mysqlclient-sys/bundled builds a static library even without the mysql development and library packages installed. But for the default feature set, a dynamic lib from the system is used.

Anyhoo, all in all it was successful. Thanks for the mysqlclient-sys/bundled suggestion. Now I'll have to decide whether I accept the fact that openssl is statically linked or that the resulting binary depends on a non-Fedora package.
Decisions, decisions. (crap, there is no pulling-out-my-hair emoji)

For reference:

ldd before

linux-vdso.so.1 (0x00007f81b9e03000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007f81b7b28000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f81b7600000)
libmariadb.so.3 => /lib64/libmariadb.so.3 (0x00007f81b9d9a000)
libpq.so.5 => /lib64/libpq.so.5 (0x00007f81b7ad5000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f81b75d4000)
libm.so.6 => /lib64/libm.so.6 (0x00007f81b74e6000)
libc.so.6 => /lib64/libc.so.6 (0x00007f81b72f4000)
libz.so.1 => /lib64/libz.so.1 (0x00007f81b72d1000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f81b727b000)
libldap.so.2 => /lib64/libldap.so.2 (0x00007f81b7215000)
/lib64/ld-linux-x86-64.so.2 (0x00007f81b9e05000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f81b714b000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f81b9d7f000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f81b9d78000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f81b7ac5000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f81b7abe000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f81b7139000)
liblber.so.2 => /lib64/liblber.so.2 (0x00007f81b7128000)
libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007f81b70d0000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f81b70b1000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f81b7080000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f81b704b000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f81b6fa0000)

ldd after

linux-vdso.so.1 (0x00007f3c7c9d9000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f3c79600000)
libpq.so.5 => /lib64/libpq.so.5 (0x00007f3c7c962000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3c7c936000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3c7990c000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3c7940c000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007f3c79321000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f3c78c00000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f3c7c8de000)
libldap.so.2 => /lib64/libldap.so.2 (0x00007f3c798a5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3c7c9db000)
libz.so.1 => /lib64/libz.so.1 (0x00007f3c79882000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f3c79257000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f3c79240000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f3c7c8d6000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f3c79230000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3c7987c000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3c7921e000)
liblber.so.2 => /lib64/liblber.so.2 (0x00007f3c7920d000)
libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007f3c791b6000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f3c79197000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f3c79166000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f3c78bcc000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f3c78b21000)

P.S.: I made a mistake. openssl is not linked statically.

@tessus commented on GitHub (Nov 4, 2025): Here's my feedback. It worked mostly great, but interestingly I had to install 3 packages: `libtirpc-devel`, `perl-Time-Piece`, and `ncurses-devel`. (The cargo build process failed 3 times.) While I understand the first 2, ncurses makes no sense for a mysql lib nor for the build process. But it's not an issue, just strange. Then it compiled successfully, but spat out a warning: ``` warning: the following packages contain code that will be rejected by a future version of Rust: num-bigint-dig v0.8.4 ``` I updated with `cargo update num-bigint-dig`, but 0.8.5 yielded the same warning. It seems an additional fix has been added after 0.8.5 was released, so that should be solved soon. The size of the binary has increased from 38 MB to 52 MB. ~~Digging a bit deeper it seems that `-F mysqlclient-sys/bundled` not only statically linked mysql, but also openssl. Hmmm. This is rather annoying and makes me slightly unhappy. But I guess I will have to live with it.~~ Another strange thing is that I removed all mariadb* packages before building vw with `-F mysqlclient-sys/bundled` and there was no `libmysqlclient` library (static or dynamic) on the system, because I forgot to install it after removing the mariadb packages. I ran a find on / and there was definitely not another one hiding somewhere else. Yet, I did not encounter a build error. Both crates `mysqlclient-sys` and `mysqlclient-src` explicitly state that the `libmysqlclient` library is required for building them. I tested the binary and it works. I don't understand why and how, but it does. The only explanation I could come up with is that `mysqlclient-sys/bundled` builds a static library even without the mysql development and library packages installed. But for the default feature set, a dynamic lib from the system is used. Anyhoo, all in all it was successful. Thanks for the `mysqlclient-sys/bundled` suggestion. ~~Now I'll have to decide whether I accept the fact that openssl is statically linked or that the resulting binary depends on a non-Fedora package. Decisions, decisions. (crap, there is no pulling-out-my-hair emoji)~~ ---- For reference: <details><summary>ldd before</summary> <p> ``` linux-vdso.so.1 (0x00007f81b9e03000) libssl.so.3 => /lib64/libssl.so.3 (0x00007f81b7b28000) libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f81b7600000) libmariadb.so.3 => /lib64/libmariadb.so.3 (0x00007f81b9d9a000) libpq.so.5 => /lib64/libpq.so.5 (0x00007f81b7ad5000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f81b75d4000) libm.so.6 => /lib64/libm.so.6 (0x00007f81b74e6000) libc.so.6 => /lib64/libc.so.6 (0x00007f81b72f4000) libz.so.1 => /lib64/libz.so.1 (0x00007f81b72d1000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f81b727b000) libldap.so.2 => /lib64/libldap.so.2 (0x00007f81b7215000) /lib64/ld-linux-x86-64.so.2 (0x00007f81b9e05000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f81b714b000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f81b9d7f000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f81b9d78000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f81b7ac5000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f81b7abe000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f81b7139000) liblber.so.2 => /lib64/liblber.so.2 (0x00007f81b7128000) libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007f81b70d0000) libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f81b70b1000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f81b7080000) libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f81b704b000) libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f81b6fa0000) ``` </p> </details> <details><summary>ldd after</summary> <p> ``` linux-vdso.so.1 (0x00007f3c7c9d9000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f3c79600000) libpq.so.5 => /lib64/libpq.so.5 (0x00007f3c7c962000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3c7c936000) libm.so.6 => /lib64/libm.so.6 (0x00007f3c7990c000) libc.so.6 => /lib64/libc.so.6 (0x00007f3c7940c000) libssl.so.3 => /lib64/libssl.so.3 (0x00007f3c79321000) libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f3c78c00000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f3c7c8de000) libldap.so.2 => /lib64/libldap.so.2 (0x00007f3c798a5000) /lib64/ld-linux-x86-64.so.2 (0x00007f3c7c9db000) libz.so.1 => /lib64/libz.so.1 (0x00007f3c79882000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f3c79257000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f3c79240000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f3c7c8d6000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f3c79230000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3c7987c000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3c7921e000) liblber.so.2 => /lib64/liblber.so.2 (0x00007f3c7920d000) libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007f3c791b6000) libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f3c79197000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f3c79166000) libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f3c78bcc000) libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f3c78b21000) ``` </p> </details> P.S.: I made a mistake. openssl is not linked statically.

kerem commented

Author

Owner

@weiznich commented on GitHub (Nov 5, 2025):

Here's my feedback. It worked mostly great, but interestingly I had to install 3 packages: libtirpc-devel, perl-Time-Piece, and ncurses-devel. (The cargo build process failed 3 times.)

These are required by mysql itself. There is unfortunately no way to run their configure script only for libmysqlclient, so you need to have all dependencies for all parts of the code on your system.

Digging a bit deeper it seems that -F mysqlclient-sys/bundled not only statically linked mysql, but also openssl. Hmmm. This is rather annoying and makes me slightly unhappy. But I guess I will have to live with it.

As far as I can tell this is correct, but might be something that could be changed. github.com/sgrif/mysqlclient-sys@65d2f05886/mysqlclient-src/Cargo.toml (L203) specifies that it uses the bundled openssl-sys build. It would need to be something more along this https://github.com/sgrif/pq-sys/blob/master/pq-src/Cargo.toml#L206 line. This likely requires some additional tweaking of the build parameters passed to cmake. Contributions for that are certainly welcome. With a solution like that implemented in pq-sys you would be able to dynamically link openssl or statically link it as well by manually enabling the openssl/vendored feature from the top level crate.

Another strange thing is that I removed all mariadb* packages before building vw with -F mysqlclient-sys/bundled and there was no libmysqlclient library (static or dynamic) on the system, because I forgot to install it after removing the mariadb packages. I ran a find on / and there was definitely not another one hiding somewhere else. Yet, I did not encounter a build error. Both crates mysqlclient-sys and mysqlclient-src explicitly state that the libmysqlclient library is required for building them.

The mysqlclient-src crate brings it's own copy of the mysql source code. https://docs.rs/crate/mysqlclient-src/latest/source/source/. So yes it still requires libmysqlclient but it brings its own version to satisfy that requirement.

@weiznich commented on GitHub (Nov 5, 2025): > Here's my feedback. It worked mostly great, but interestingly I had to install 3 packages: libtirpc-devel, perl-Time-Piece, and ncurses-devel. (The cargo build process failed 3 times.) These are required by mysql itself. There is unfortunately no way to run their configure script only for libmysqlclient, so you need to have all dependencies for all parts of the code on your system. > Digging a bit deeper it seems that -F mysqlclient-sys/bundled not only statically linked mysql, but also openssl. Hmmm. This is rather annoying and makes me slightly unhappy. But I guess I will have to live with it. As far as I can tell this is correct, but might be something that could be changed. https://github.com/sgrif/mysqlclient-sys/blob/65d2f058864ec58fcb868cabeab85b908ab85f31/mysqlclient-src/Cargo.toml#L203 specifies that it uses the bundled openssl-sys build. It would need to be something more along this https://github.com/sgrif/pq-sys/blob/master/pq-src/Cargo.toml#L206 line. This likely requires some additional tweaking of the build parameters passed to cmake. Contributions for that are certainly welcome. With a solution like that implemented in pq-sys you would be able to dynamically link openssl or statically link it as well by manually enabling the openssl/vendored feature from the top level crate. > Another strange thing is that I removed all mariadb* packages before building vw with -F mysqlclient-sys/bundled and there was no libmysqlclient library (static or dynamic) on the system, because I forgot to install it after removing the mariadb packages. I ran a find on / and there was definitely not another one hiding somewhere else. Yet, I did not encounter a build error. Both crates mysqlclient-sys and mysqlclient-src explicitly state that the libmysqlclient library is required for building them. The `mysqlclient-src` crate brings it's own copy of the mysql source code. https://docs.rs/crate/mysqlclient-src/latest/source/source/. So yes it still requires libmysqlclient but it brings its own version to satisfy that requirement.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 5, 2025):

@weiznich thx for your reply. I made a mistake and it seems that the openssl is not statically linked.

I also found out that Fedora apparently now has mysql in its default repo as well. I don't know when that happened, but it seems it is no longer necessary to use the oracle mysql repo. I recall a rather nasty fight about licensing and whatnot, but I guess this was resolved at one point.

This means I can choose to statically link the mysql client or not. I like to have options, so that's great.

@tessus commented on GitHub (Nov 5, 2025): @weiznich thx for your reply. I made a mistake and it seems that the openssl is not statically linked. I also found out that Fedora apparently now has mysql in its default repo as well. I don't know when that happened, but it seems it is no longer necessary to use the oracle mysql repo. I recall a rather nasty fight about licensing and whatnot, but I guess this was resolved at one point. This means I can choose to statically link the mysql client or not. I like to have options, so that's great.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

@weiznich thx for your reply. I made a mistake and it seems that the openssl is not statically linked.

I think it is, and to make it worse, it's statically linked, and dynamically.
The MySQL Client library uses the statically linked version, and all other parts of Vaultwarden will use the dynamically linked version.

You can verify this by running this command on the resulted binary.

strings ./target/debug/vaultwarden | grep -iE "OpenSSL 3"
# Or for release
strings ./target/release/vaultwarden | grep -iE "OpenSSL 3"

I also found out that Fedora apparently now has mysql in its default repo as well. I don't know when that happened, but it seems it is no longer necessary to use the oracle mysql repo. I recall a rather nasty fight about licensing and whatnot, but I guess this was resolved at one point.

This means I can choose to statically link the mysql client or not. I like to have options, so that's great.

I'm not sure you can fully statically link a glibc library externally provided, but i could be wrong of course. Also, if you build against the Oracle MySQL and only provide the binary, and someone has the MariaDB client installed, that could cause weird issues.

@BlackDex commented on GitHub (Nov 5, 2025): > [@weiznich](https://github.com/weiznich) thx for your reply. I made a mistake and it seems that the openssl is not statically linked. > I think it is, and to make it worse, it's statically linked, and dynamically. The MySQL Client library uses the statically linked version, and all other parts of Vaultwarden will use the dynamically linked version. You can verify this by running this command on the resulted binary. ```bash strings ./target/debug/vaultwarden | grep -iE "OpenSSL 3" # Or for release strings ./target/release/vaultwarden | grep -iE "OpenSSL 3" ``` > I also found out that Fedora apparently now has mysql in its default repo as well. I don't know when that happened, but it seems it is no longer necessary to use the oracle mysql repo. I recall a rather nasty fight about licensing and whatnot, but I guess this was resolved at one point. > > This means I can choose to statically link the mysql client or not. I like to have options, so that's great. I'm not sure you can fully statically link a glibc library externally provided, but i could be wrong of course. Also, if you build against the Oracle MySQL and only provide the binary, and someone has the MariaDB client installed, that could cause weird issues.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

I tried to use the OPENSSL_NO_VENDOR=true option, but that is causing race-condition and always fails to build, no matter if i point specifically to the correct libraries. So, probably the only way to fix that is by adding a no-bundled-openssl feature to mysqlclient-sys/src, or make bundled-openssl the default, as Weiznich mentioned

@BlackDex commented on GitHub (Nov 5, 2025): I tried to use the `OPENSSL_NO_VENDOR=true` option, but that is causing race-condition and always fails to build, no matter if i point specifically to the correct libraries. So, probably the only way to fix that is by adding a `no-bundled-openssl` feature to mysqlclient-sys/src, or make bundled-openssl the default, as Weiznich mentioned

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

I think we need to stop building Debian images, and only use Alpine from now on.
There we (I my self) control which version is used, and i might even be able to switch to Oracle's version, but that is probably not the best idea either.

@BlackDex commented on GitHub (Nov 5, 2025): I think we need to stop building Debian images, and only use Alpine from now on. There we (I my self) control which version is used, and i might even be able to switch to Oracle's version, but that is probably not the best idea either.

kerem commented

Author

Owner

@weiznich commented on GitHub (Nov 5, 2025):

Just to mention that as that would be another option to sidestep this mess: Diesel-async doesn't use the c client libraries for mysql/postgres. You could likely use AsyncConnectionWrapper<AsyncMysqlConnection> as drop in replacement of diesel::MysqlConnection.

Switching to diesel_async for "everything" is likely more work as it would break e.g. #[derive(MultiConnection)]

@weiznich commented on GitHub (Nov 5, 2025): Just to mention that as that would be another option to sidestep this mess: Diesel-async doesn't use the c client libraries for mysql/postgres. You could likely use `AsyncConnectionWrapper<AsyncMysqlConnection>` as drop in replacement of `diesel::MysqlConnection`. Switching to diesel_async for "everything" is likely more work as it would break e.g. `#[derive(MultiConnection)]`

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

Using diesel_async and those libraries also brings in aws-lc or we need to use ring. I'm not sure how that will break all users which have a secure connection. Only using Alpine/Static build libraries will at least work as a drop-in replacement (for the most part). So i think for now, to at least not causes issues with a new version we want to release soon, is stop building Debian for now.

The Async stuff would be nice in the future, but If that now might cause breaking or different way of working for the just recently merged MultiConnection, that would be a pain.

@BlackDex commented on GitHub (Nov 5, 2025): Using diesel_async and those libraries also brings in aws-lc or we need to use ring. I'm not sure how that will break all users which have a secure connection. Only using Alpine/Static build libraries will at least work as a drop-in replacement (for the most part). So i think for now, to at least not causes issues with a new version we want to release soon, is stop building Debian for now. The Async stuff would be nice in the future, but If that now might cause breaking or different way of working for the just recently merged MultiConnection, that would be a pain.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

~~I Think i will do something else, ill switch to Ubuntu, that has Oracle's MySQL version in the repo.~~
Scrap that, Ubuntu doesn't support all architectures we provide currently.

@BlackDex commented on GitHub (Nov 5, 2025): ~I Think i will do something else, ill switch to Ubuntu, that has Oracle's MySQL version in the repo.~ Scrap that, Ubuntu doesn't support all architectures we provide currently.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

FYI: I created a PR to stop building Debian images #6440

@BlackDex commented on GitHub (Nov 5, 2025): FYI: I created a PR to stop building Debian images #6440

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 5, 2025):

I think it is, and to make it worse, it's statically linked, and dynamically.
The MySQL Client library uses the statically linked version, and all other parts of Vaultwarden will use the dynamically linked version.

I agree, it is not optimal, but in reality not a problem. And unfortunately this happens more often than you might think.

But let's play it through: The statically linked OpenSSL is the same as the one currently installed on the system. (in my case 3.5.4)
So database operations are handled by this version of OpenSSL. Everything works.
The rest of vaultwaden uses the dynamically linked version of OpenSSL, which is currently also 3.5.4. Everyhing works.
In the future, OpenSSL will be updated by the Linux distro. Please note that distros make sure that updates to OpenSSL are ABI compatible. In rare cases when this is not the case, the "old" shared lib is still kept in place. So even though the new version might be 3.5.z or 3.6.z, the binaries using the shared lib will still work. So will the database operations via the statically linked code.

The reason why I am unhappy, is 3-fold: 1) it's a waste of space 2) new features that might be possible with newer versions of OpenSSL would not be available for the database operations 3) security updates. The latter 2 are more or less philosophical, since it takes a while to propagate stuff from OpenSSL, through crates, to vw code. TLS is used by the mysql client for establishing a connection - that's it. No new feature will be that important, especially since a connection will still be established either way. Security updates are important, but by that time I'll have already created a new package.

To circle back, it is not optimal, but it is not an operational showstopper.

Also, if you build against the Oracle MySQL and only provide the binary, and someone has the MariaDB client installed, that could cause weird issues.

No, this wouldn't be the case. There are a bunch of safeguards in place:

The mariadb library has a different name from the mysql library. Even if it were installed on the target system it wouldn't be used.
The Oracle MySQL and Fedora MySQL packages use the same library name, but different package names. However, these packages are pretty much the same - and 100% ABI compatible.

Thus a system could have either one installed and vaultwarden would run just fine.

Adittionally, RPMs allow you to specify dependencies. I could require a specific package (one of the 2), make it conflicting with the other, or I could allow both, or even specify the full path to the library (which is IMO a bad idea).

Thus, if I were to build against Oracle MySQL and allow the target system to use the Fedora MySQL client package, all would be good.

@tessus commented on GitHub (Nov 5, 2025): > I think it is, and to make it worse, it's statically linked, and dynamically. The MySQL Client library uses the statically linked version, and all other parts of Vaultwarden will use the dynamically linked version. I agree, it is not optimal, but in reality not a problem. And unfortunately this happens more often than you might think. But let's play it through: The statically linked OpenSSL is the same as the one currently installed on the system. (in my case 3.5.4) So database operations are handled by this version of OpenSSL. Everything works. The rest of vaultwaden uses the dynamically linked version of OpenSSL, which is currently also 3.5.4. Everyhing works. In the future, OpenSSL will be updated by the Linux distro. Please note that distros make sure that updates to OpenSSL are ABI compatible. In rare cases when this is not the case, the "old" shared lib is still kept in place. So even though the new version might be 3.5.z or 3.6.z, the binaries using the shared lib will still work. So will the database operations via the statically linked code. The reason why I am unhappy, is 3-fold: 1) it's a waste of space 2) new features that might be possible with newer versions of OpenSSL would not be available for the database operations 3) security updates. The latter 2 are more or less philosophical, since it takes a while to propagate stuff from OpenSSL, through crates, to vw code. TLS is used by the mysql client for establishing a connection - that's it. No new feature will be that important, especially since a connection will still be established either way. Security updates are important, but by that time I'll have already created a new package. To circle back, it is not optimal, but it is not an operational showstopper. > Also, if you build against the Oracle MySQL and only provide the binary, and someone has the MariaDB client installed, that could cause weird issues. No, this wouldn't be the case. There are a bunch of safeguards in place: The mariadb library has a different name from the mysql library. Even if it were installed on the target system it wouldn't be used. The Oracle MySQL and Fedora MySQL packages use the same library name, but different package names. However, these packages are pretty much the same - and 100% ABI compatible. Thus a system could have either one installed and vaultwarden would run just fine. Adittionally, RPMs allow you to specify dependencies. I could require a specific package (one of the 2), make it conflicting with the other, or I could allow both, or even specify the full path to the library (which is IMO a bad idea). Thus, if I were to build against Oracle MySQL and allow the target system to use the Fedora MySQL client package, all would be good.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 5, 2025):

I created a PR to stop building Debian images

@BlackDex why not just use the oracle apt repo? It's quite straightforward to setup. Adding this to the build process should be an easy endeavor.

I am not a fan of Oracle (I was in IBM DB2 development for a long time), but if it is an easy fix, why not use it?

@tessus commented on GitHub (Nov 5, 2025): > I created a PR to stop building Debian images @BlackDex why not just use the oracle apt repo? It's quite straightforward to [setup](https://dev.mysql.com/doc/refman/8.4/en/linux-installation-apt-repo.html). Adding this to the build process should be an easy endeavor. I am not a fan of Oracle (I was in IBM DB2 development for a long time), but if it is an easy fix, why not use it?

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

I created a PR to stop building Debian images

@BlackDex why not just use the oracle apt repo? It's quite straightforward to setup. Adding this to the build process should be an easy endeavor.

I am not a fan of Oracle (I was in IBM DB2 development for a long time), but if it is an easy fix, why not use it?

Oracle only builds amd64 and arm64, so not all architectures we support are available. Kinda the same as with Ubuntu. Only armhf is available, and we currently also provide arm (no hf).

While I'm not sure if that is still used, but i think it is, older NAS devices run on this.

@BlackDex commented on GitHub (Nov 5, 2025): > > I created a PR to stop building Debian images > > [@BlackDex](https://github.com/BlackDex) why not just use the oracle apt repo? It's quite straightforward to [setup](https://dev.mysql.com/doc/refman/8.4/en/linux-installation-apt-repo.html). Adding this to the build process should be an easy endeavor. > > I am not a fan of Oracle (I was in IBM DB2 development for a long time), but if it is an easy fix, why not use it? Oracle only builds amd64 and arm64, so not all architectures we support are available. Kinda the same as with Ubuntu. Only armhf is available, and we currently also provide arm (no hf). While I'm not sure if that is still used, but i think it is, older NAS devices run on this.

kerem commented

Author

Owner

@tessus commented on GitHub (Nov 5, 2025):

@BlackDex Thanks, this makes sense. I haven't thought of that. On the other hand, it doesn't hurt to just provide what is possible. There is no rule that states you have to provide containers/packages for all supported platforms. ;-)
I'm sure that people who are currently using your Debian containers on {amd,arm}64 will appreciate it.

@tessus commented on GitHub (Nov 5, 2025): @BlackDex Thanks, this makes sense. I haven't thought of that. On the other hand, it doesn't hurt to just provide what is possible. There is no rule that states you have to provide containers/packages for all supported platforms. ;-) I'm sure that people who are currently using your Debian containers on {amd,arm}64 will appreciate it.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 5, 2025):

There is also no main reason to provide all these different container images if we need to jump through hoops.

The static binaries work fine, Alpine has 0 security issues according to trivy, and Debian, 51 and Ubuntu 11, so that is also a win from my point of view.

@BlackDex commented on GitHub (Nov 5, 2025): There is also no main reason to provide all these different container images if we need to jump through hoops. The static binaries work fine, Alpine has 0 security issues according to trivy, and Debian, 51 and Ubuntu 11, so that is also a win from my point of view.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 9, 2025):

Ok.. I found something strange, and please if someone can correct me if I'm wrong here.
But i tried ArchLinux version of Vaultwarden, and that seems to work just fine, though is seems to be using the 12.0.2 server version, which uses the 3.4.7 connecter-c version. And there are no issue there.

I see no patches done on the client, so, something most be different?
Maybe @polyzen could help here to clarify this maybe?

@BlackDex commented on GitHub (Nov 9, 2025): Ok.. I found something strange, and please if someone can correct me if I'm wrong here. But i tried ArchLinux version of Vaultwarden, and that seems to work just fine, though is seems to be using the `12.0.2` server version, which uses the `3.4.7` connecter-c version. And there are no issue there. I see no patches done on the client, so, something most be different? Maybe @polyzen could help here to clarify this maybe?

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 9, 2025):

Ok... So.... I tried the Alpine package of Vaultwarden too, and that seems to work also...
This doesn't make it less strange.

@BlackDex commented on GitHub (Nov 9, 2025): Ok... So.... I tried the Alpine package of Vaultwarden too, and that seems to work also... This doesn't make it less strange.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 9, 2025):

@weiznich, I'm not sure what exactly has changed in v2.3.x of Diesel which could cause this behavior.

But When testing the Alpine/Arch binaries, they worked just fine, then looking at the dependencies, it was still using Diesel v2.2.12.

Now i reverted Diesel back to v2.2.12 in my testing environment, and i needed to make some changes to get it to work like remove BigDecimal (Because that was fixed in a newer version) and some other small stuff, but it builds fine, and also doesn't cause this bug to happen.

Isn't there still a chance something broke in Diesel which caused a regression to happen?

@BlackDex commented on GitHub (Nov 9, 2025): @weiznich, I'm not sure what exactly has changed in v2.3.x of Diesel which could cause this behavior. But When testing the Alpine/Arch binaries, they worked just fine, then looking at the dependencies, it was still using Diesel v2.2.12. Now i reverted Diesel back to v2.2.12 in my testing environment, and i needed to make some changes to get it to work like remove BigDecimal (Because that was fixed in a newer version) and some other small stuff, but it builds fine, and also doesn't cause this bug to happen. Isn't there still a chance something broke in Diesel which caused a regression to happen?

kerem commented

Author

Owner

@weiznich commented on GitHub (Nov 9, 2025):

It "works" with older diesel versions because the check that generates this error was only introduced recently with https://github.com/diesel-rs/diesel/pull/4756

These checks were introduced as reaction to a code audit. See the report here.

Without this check the relevant code could and technically does perform a out of bounds read.

Am I 100% sure that this is bug free now? No, but I'm certain that this is now more correct than before.

https://github.com/diesel-rs/diesel/pull/4851 might help to actually address the problem, it still need some time from me to understand if that is sound.

@weiznich commented on GitHub (Nov 9, 2025): It "works" with older diesel versions because the check that generates this error was only introduced recently with https://github.com/diesel-rs/diesel/pull/4756 These checks were introduced as reaction to a code audit. See the report [here](http://diesel.rs/assets/NGICore%20Diesel%20penetration%20test%20report%202025%201.0.pdf). Without this check the relevant code could and technically does perform a out of bounds read. Am I 100% sure that this is bug free now? No, but I'm certain that this is now more correct than before. https://github.com/diesel-rs/diesel/pull/4851 might help to actually address the problem, it still need some time from me to understand if that is sound.

kerem commented