[GH-ISSUE #4693] Very slow multiple add/delete operations

kerem commented

2026-03-03 02:13:48 +03:00

Owner

Originally created by @epiniguin on GitHub (Jul 2, 2024).
Original GitHub issue: https://github.com/dani-garcia/vaultwarden/issues/4693

Subject of the issue

I installed Vaultwarden and Bitwarden Unified Beta for testing and comparing.
Both are installed as Docker containers on the same Debian VM(It is 8 cores Intel Xeon with 4GB RAM. So it is not a slow server).
Both vaults use a shared Postgresql installed on the same VM as Docker container.

Both vaults are empty before testing.
Both are behind the same Nginx proxy installed on the router.

I use the web vault, the Firefox official Bitwarden extension and the Android official Bitwarden app.

The clients and the servers are in the same LAN.

For a testing purpose I exported 427 passwords from Firefox to CSV.
I imported and after some pause deleted these items. After that I emptied the recycle bin.
I tried to import from web vaults and a Firefox extension - the results were the same, so I guess it is not a client issue.
Deleting and recycle bin emptying are possible only from the web vaults.

The results are in the table below.
When testing Vaultwarden with enabled push notifications it was difficult to measure the times precise because of web vault 60s timeout errors. Besides, after these timeouts the data in the Vaultwarden web vault stopped to refresh and it needed to renew the page and to unlock the vault.

Operation	Vaultwarden(disabled push)	Vaultwarden(enabled push)	Bitwarden(enabled push)
Import 427 items	24s	24s	12s
Delete 427 items	14s	130-150s	3s
Empty recycle bin(427 items)	41s	130-150s	3s

Deployment environment

Your environment (Generated via diagnostics page)

Vaultwarden version: v1.30.5
Web-vault version: v2024.1.2b
OS/Arch: linux/x86_64
Running within a container: true (Base: Debian)
Environment settings overridden: false
Uses a reverse proxy: true
IP Header check: true (X-Real-IP)
Internet access: true
Internet access via a proxy: false
DNS Check: true
Browser/Server Time Check: true
Server/NTP Time Check: true
Domain Configuration Check: true
HTTPS Check: true
Database type: PostgreSQL
Database version: PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
Clients used:
Reverse proxy and version:
Other relevant information:

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden:

{
  "_duo_akey": null,
  "_enable_duo": true,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "auth_request_purge_schedule": "30 * * * * *",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "**********://**********************************************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://***************",
  "domain_origin": "*****://***************",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_attempts_limit": 3,
  "email_change_allowed": true,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": true,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "experimental_client_feature_flags": "fido2-vault-credentials",
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "Vaultwarden",
  "invitations_allowed": true,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": "/data/vaultwarden.log",
  "log_level": "Info",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "",
  "org_events_enabled": false,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 600000,
  "push_enabled": true,
  "push_identity_uri": "https://identity.bitwarden.com",
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": true,
  "signups_domains_whitelist": "",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": null,
  "smtp_from": "*************",
  "smtp_from_name": "Vaultwarden",
  "smtp_host": "*******************",
  "smtp_password": "***",
  "smtp_port": 587,
  "smtp_security": "starttls",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": "*************",
  "templates_folder": "data/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": null,
  "user_send_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": false,
  "websocket_port": 3012,
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

vaultwarden version: 1.30.5

Install method: Docker
Clients used:
web vault, firefox extension
Reverse proxy and version:
nginx/1.26.1 installed on the router
MySQL/MariaDB or PostgreSQL version:
PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit

Steps to reproduce

Import 300-500 items from CSV.
Delete these items.
Empty recycle bin.

Expected behaviour

Vaultwarden and Bitwarden should work at approximately the same speed.

Actual behaviour

Vaultwarden is much slower than Bitwarden particularly with enabled push notifications.

Troubleshooting data

Originally created by @epiniguin on GitHub (Jul 2, 2024). Original GitHub issue: https://github.com/dani-garcia/vaultwarden/issues/4693   ### Subject of the issue  I installed Vaultwarden and Bitwarden Unified Beta for testing and comparing. Both are installed as Docker containers on the same Debian VM(It is 8 cores Intel Xeon with 4GB RAM. So it is not a slow server). Both vaults use a shared Postgresql installed on the same VM as Docker container. Both vaults are empty before testing. Both are behind the same Nginx proxy installed on the router. I use the web vault, the Firefox official Bitwarden extension and the Android official Bitwarden app. The clients and the servers are in the same LAN. For a testing purpose I exported 427 passwords from Firefox to CSV. I imported and after some pause deleted these items. After that I emptied the recycle bin. I tried to import from web vaults and a Firefox extension - the results were the same, so I guess it is not a client issue. Deleting and recycle bin emptying are possible only from the web vaults. The results are in the table below. When testing Vaultwarden with enabled push notifications it was difficult to measure the times precise because of web vault 60s timeout errors. Besides, after these timeouts the data in the Vaultwarden web vault stopped to refresh and it needed to renew the page and to unlock the vault. |Operation|Vaultwarden(disabled push)|Vaultwarden(enabled push)|Bitwarden(enabled push)| |---|---|---|---| |Import 427 items|24s|24s|12s| |Delete 427 items|14s|130-150s|3s| |Empty recycle bin(427 items)|41s|130-150s|3s| ### Deployment environment ### Your environment (Generated via diagnostics page) * Vaultwarden version: v1.30.5 * Web-vault version: v2024.1.2b * OS/Arch: linux/x86_64 * Running within a container: true (Base: Debian) * Environment settings overridden: false * Uses a reverse proxy: true * IP Header check: true (X-Real-IP) * Internet access: true * Internet access via a proxy: false * DNS Check: true * Browser/Server Time Check: true * Server/NTP Time Check: true * Domain Configuration Check: true * HTTPS Check: true * Database type: PostgreSQL * Database version: PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit * Clients used: * Reverse proxy and version: * Other relevant information: ### Config (Generated via diagnostics page) <details><summary>Show Running Config</summary> **Environment settings which are overridden:** ```json { "_duo_akey": null, "_enable_duo": true, "_enable_email_2fa": true, "_enable_smtp": true, "_enable_yubico": true, "_icon_service_csp": "", "_icon_service_url": "", "_ip_header_enabled": true, "_smtp_img_src": "cid:", "admin_ratelimit_max_burst": 3, "admin_ratelimit_seconds": 300, "admin_session_lifetime": 20, "admin_token": "***", "allowed_iframe_ancestors": "", "attachments_folder": "data/attachments", "auth_request_purge_schedule": "30 * * * * *", "authenticator_disable_time_drift": false, "data_folder": "data", "database_conn_init": "", "database_max_conns": 10, "database_timeout": 30, "database_url": "**********://**********************************************", "db_connection_retries": 15, "disable_2fa_remember": false, "disable_admin_token": false, "disable_icon_download": false, "domain": "*****://***************", "domain_origin": "*****://***************", "domain_path": "", "domain_set": true, "duo_host": null, "duo_ikey": null, "duo_skey": null, "email_attempts_limit": 3, "email_change_allowed": true, "email_expiration_time": 600, "email_token_size": 6, "emergency_access_allowed": true, "emergency_notification_reminder_schedule": "0 3 * * * *", "emergency_request_timeout_schedule": "0 7 * * * *", "enable_db_wal": true, "event_cleanup_schedule": "0 10 0 * * *", "events_days_retain": null, "experimental_client_feature_flags": "fido2-vault-credentials", "extended_logging": true, "helo_name": null, "hibp_api_key": null, "icon_blacklist_non_global_ips": true, "icon_blacklist_regex": null, "icon_cache_folder": "data/icon_cache", "icon_cache_negttl": 259200, "icon_cache_ttl": 2592000, "icon_download_timeout": 10, "icon_redirect_code": 302, "icon_service": "internal", "incomplete_2fa_schedule": "30 * * * * *", "incomplete_2fa_time_limit": 3, "invitation_expiration_hours": 120, "invitation_org_name": "Vaultwarden", "invitations_allowed": true, "ip_header": "X-Real-IP", "job_poll_interval_ms": 30000, "log_file": "/data/vaultwarden.log", "log_level": "Info", "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f", "login_ratelimit_max_burst": 10, "login_ratelimit_seconds": 60, "org_attachment_limit": null, "org_creation_users": "", "org_events_enabled": false, "org_groups_enabled": false, "password_hints_allowed": true, "password_iterations": 600000, "push_enabled": true, "push_identity_uri": "https://identity.bitwarden.com", "push_installation_id": "***", "push_installation_key": "***", "push_relay_uri": "https://push.bitwarden.com", "reload_templates": false, "require_device_email": false, "rsa_key_filename": "data/rsa_key", "send_purge_schedule": "0 5 * * * *", "sendmail_command": null, "sends_allowed": true, "sends_folder": "data/sends", "show_password_hint": false, "signups_allowed": true, "signups_domains_whitelist": "", "signups_verify": false, "signups_verify_resend_limit": 6, "signups_verify_resend_time": 3600, "smtp_accept_invalid_certs": false, "smtp_accept_invalid_hostnames": false, "smtp_auth_mechanism": null, "smtp_debug": false, "smtp_embed_images": true, "smtp_explicit_tls": null, "smtp_from": "*************", "smtp_from_name": "Vaultwarden", "smtp_host": "*******************", "smtp_password": "***", "smtp_port": 587, "smtp_security": "starttls", "smtp_ssl": null, "smtp_timeout": 15, "smtp_username": "*************", "templates_folder": "data/templates", "tmp_folder": "data/tmp", "trash_auto_delete_days": null, "trash_purge_schedule": "0 5 0 * * *", "use_sendmail": false, "use_syslog": false, "user_attachment_limit": null, "user_send_limit": null, "web_vault_enabled": true, "web_vault_folder": "web-vault/", "websocket_address": "0.0.0.0", "websocket_enabled": false, "websocket_port": 3012, "yubico_client_id": null, "yubico_secret_key": null, "yubico_server": null } ``` </details>     * vaultwarden version: 1.30.5  * Install method: Docker * Clients used:  web vault, firefox extension * Reverse proxy and version:  nginx/1.26.1 installed on the router * MySQL/MariaDB or PostgreSQL version:  PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit  ### Steps to reproduce  1. Import 300-500 items from CSV. 2. Delete these items. 3. Empty recycle bin. ### Expected behaviour  Vaultwarden and Bitwarden should work at approximately the same speed. ### Actual behaviour  Vaultwarden is much slower than Bitwarden particularly with enabled push notifications. ### Troubleshooting data

kerem

2026-03-03 02:13:48 +03:00

closed this issue
added the
enhancement

low priority
labels

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@BlackDex commented on GitHub (Jul 2, 2024):

Hmm, i thought we fixed that already in testing tagged images, but I'm not sure. Because we do not push on import, but we shouldn't do this on bulk delete/trash-clean too. But we should see if we can trigger a refresh or something else for the other clients, not sure if that is possible currently.

@BlackDex commented on GitHub (Jul 2, 2024): Hmm, i thought we fixed that already in `testing` tagged images, but I'm not sure. Because we do not push on import, but we shouldn't do this on bulk delete/trash-clean too. But we should see if we can trigger a refresh or something else for the other clients, not sure if that is possible currently.

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@BlackDex commented on GitHub (Jul 2, 2024):

Also, there is a similar issue #4669.
The only way i can think of is running the notifications in there own thread, and handle the pushes outside of the web/main threads.

@BlackDex commented on GitHub (Jul 2, 2024): Also, there is a similar issue #4669. The only way i can think of is running the notifications in there own thread, and handle the pushes outside of the web/main threads.

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@epiniguin commented on GitHub (Jul 2, 2024):

It is slower without enabled push also.
I guess that 427 items is not a problem at all for Postgresql. That's why Bitwarden deletes them for 3s(with https delays).

@epiniguin commented on GitHub (Jul 2, 2024): It is slower without enabled push also. I guess that 427 items is not a problem at all for Postgresql. That's why Bitwarden deletes them for 3s(with https delays).

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@epiniguin commented on GitHub (Jul 2, 2024):

Also, there is a similar issue #4669.

In my case I don't have any delay for single operations. Only for multiple.

@epiniguin commented on GitHub (Jul 2, 2024): > Also, there is a similar issue #4669. In my case I don't have any delay for single operations. Only for multiple.

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@stefan0xC commented on GitHub (Jul 3, 2024):

I can't really reproduce your findings (with the vaultwarden/server:latest image and also using postgres 16). With 330 items each operation (in the web-vault) took about 1-1.5 seconds with Vaultwarden.

[2024-07-03 06:16:15.476][request][INFO] POST /api/ciphers/import
[2024-07-03 06:16:16.599][response][INFO] (post_ciphers_import) POST /api/ciphers/import => 200 OK
...
[2024-07-03 06:16:41.966][request][INFO] PUT /api/ciphers/delete
[2024-07-03 06:16:42.961][response][INFO] (delete_cipher_selected_put) PUT /api/ciphers/delete => 200 OK
...
[2024-07-03 06:16:53.331][request][INFO] DELETE /api/ciphers
[2024-07-03 06:16:54.417][response][INFO] (delete_cipher_selected) DELETE /api/ciphers => 200 OK

edit: even with 499 items it's 1.7-2 seconds for these requests.

@stefan0xC commented on GitHub (Jul 3, 2024): I can't really reproduce your findings (with the `vaultwarden/server:latest` image and also using postgres 16). With 330 items each operation (in the web-vault) took about 1-1.5 seconds with Vaultwarden. ``` [2024-07-03 06:16:15.476][request][INFO] POST /api/ciphers/import [2024-07-03 06:16:16.599][response][INFO] (post_ciphers_import) POST /api/ciphers/import => 200 OK ... [2024-07-03 06:16:41.966][request][INFO] PUT /api/ciphers/delete [2024-07-03 06:16:42.961][response][INFO] (delete_cipher_selected_put) PUT /api/ciphers/delete => 200 OK ... [2024-07-03 06:16:53.331][request][INFO] DELETE /api/ciphers [2024-07-03 06:16:54.417][response][INFO] (delete_cipher_selected) DELETE /api/ciphers => 200 OK ``` edit: even with 499 items it's 1.7-2 seconds for these requests.

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@epiniguin commented on GitHub (Jul 3, 2024):

It's my log.
I don't know what is the difference between our systems.
But as I wrote Bitwarden works fast on the same infrastructure.

What diagnostic could I do?

[2024-07-03 09:40:19.431][request][INFO] POST /api/ciphers/import
[2024-07-03 09:40:43.737][response][INFO] (post_ciphers_import) POST /api/ciphers/import => 200 OK
...
[2024-07-03 09:43:04.312][request][INFO] PUT /api/ciphers/delete
[2024-07-03 09:44:38.549][response][INFO] (delete_cipher_selected_put) PUT /api/ciphers/delete => 200 OK
....
[2024-07-03 09:46:49.204][request][INFO] DELETE /api/ciphers
[2024-07-03 09:48:26.149][response][INFO] (delete_cipher_selected) DELETE /api/ciphers => 200 OK

@epiniguin commented on GitHub (Jul 3, 2024): It's my log. I don't know what is the difference between our systems. But as I wrote Bitwarden works fast on the same infrastructure. What diagnostic could I do? ``` [2024-07-03 09:40:19.431][request][INFO] POST /api/ciphers/import [2024-07-03 09:40:43.737][response][INFO] (post_ciphers_import) POST /api/ciphers/import => 200 OK ... [2024-07-03 09:43:04.312][request][INFO] PUT /api/ciphers/delete [2024-07-03 09:44:38.549][response][INFO] (delete_cipher_selected_put) PUT /api/ciphers/delete => 200 OK .... [2024-07-03 09:46:49.204][request][INFO] DELETE /api/ciphers [2024-07-03 09:48:26.149][response][INFO] (delete_cipher_selected) DELETE /api/ciphers => 200 OK ```

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@BlackDex commented on GitHub (Jul 3, 2024):

Try to enable LOG_LEVEL=debug and see if that provides more (and enough) details, else set it to trace that should give a lot details.

@BlackDex commented on GitHub (Jul 3, 2024): Try to enable `LOG_LEVEL=debug` and see if that provides more (and enough) details, else set it to `trace` that should give a lot details.

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@stefan0xC commented on GitHub (Jul 3, 2024):

Okay, I can only reproduce the issue if I enable push.

@stefan0xC commented on GitHub (Jul 3, 2024): Okay, I can only reproduce the issue if I enable push.

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@epiniguin commented on GitHub (Jul 3, 2024):

Okay, I can only reproduce the issue if I enable push.

Attached delete logs with enabled push.

delete_with_push.txt
bin_with_push.txt

@epiniguin commented on GitHub (Jul 3, 2024): > Okay, I can only reproduce the issue if I enable push. Attached delete logs with enabled push. [delete_with_push.txt](https://github.com/user-attachments/files/16085399/delete_with_push.txt) [bin_with_push.txt](https://github.com/user-attachments/files/16085438/bin_with_push.txt)

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@epiniguin commented on GitHub (Jul 3, 2024):

This is emptying bin trace with disabled push.
I don't see anything bad, but my timing is 23s, not 1.5s as yours.

bin_wo_push_trace.txt

@epiniguin commented on GitHub (Jul 3, 2024): This is emptying bin trace with disabled push. I don't see anything bad, but my timing is 23s, not 1.5s as yours. [bin_wo_push_trace.txt](https://github.com/user-attachments/files/16085641/bin_wo_push_trace.txt)

kerem commented

2026-03-03 02:13:49 +03:00

Author

Owner

@gabrielecabrini commented on GitHub (Jun 12, 2025):

I've also noticed the same issue even when deleting single items taking more than 3 seconds each (disabled push)

@gabrielecabrini commented on GitHub (Jun 12, 2025): I've also noticed the same issue even when deleting single items taking more than 3 seconds each (disabled push)

kerem commented

2026-03-03 02:13:50 +03:00

Author

Owner

@BlackDex commented on GitHub (Jun 12, 2025):

I've also noticed the same issue even when deleting single items taking more than 3 seconds each (disabled push)

If you have push disabled, then it's something totally different. Might even be a database latency, or disk I/O issue.
Try to enable debug logging and see if you find anything useful in the logs.

@BlackDex commented on GitHub (Jun 12, 2025): > I've also noticed the same issue even when deleting single items taking more than 3 seconds each (disabled push) If you have push disabled, then it's something totally different. Might even be a database latency, or disk I/O issue. Try to enable debug logging and see if you find anything useful in the logs.

kerem commented

2026-03-03 02:13:50 +03:00

Author

Owner

@BlackDex commented on GitHub (Aug 7, 2025):

Ok, i took a better look the past few days, and we were sending out a push on every delete.
This overloaded both Vaultwarden and the Bitwarden Push servers, and caused the slow-down.

While i think we could still improve upon import and delete of multiple ciphers, the main issues here should be fixed via #6144, and trying to optimize import and delete of multiple items is a different task from my point of view.

@BlackDex commented on GitHub (Aug 7, 2025): Ok, i took a better look the past few days, and we were sending out a push on every delete. This overloaded both Vaultwarden and the Bitwarden Push servers, and caused the slow-down. While i think we could still improve upon import and delete of multiple ciphers, the main issues here should be fixed via #6144, and trying to optimize import and delete of multiple items is a different task from my point of view.

Rows
Columns

[GH-ISSUE #4693] Very slow multiple add/delete operations #1958

Subject of the issue

Deployment environment

Your environment (Generated via diagnostics page)

Config (Generated via diagnostics page)

Steps to reproduce

Expected behaviour

Actual behaviour

Troubleshooting data