[GH-ISSUE #822] It gets unresponsive sometime after starting the container #575

Closed
opened 2026-03-03 01:30:42 +03:00 by kerem · 10 comments
Owner

Originally created by @gerroon on GitHub (Jan 19, 2020).
Original GitHub issue: https://github.com/dani-garcia/vaultwarden/issues/822

Subject of the issue

Hi

I started getting a strange issue. The BW_RS container becomes unresponsive sometime after starting, when I sya unresponsive I mean unresponsive to browsing via clients like Firefox. I can neither access the local ip nor the reverse proxied url. The Docker logs shows nothing, it looks as if it is running normally.

Here last couple lines from docker logs -f bitwarden

[2020-01-19 03:49:10][request][INFO] POST /notifications/hub/negotiate
[2020-01-19 03:49:10][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK

Your environment

Debian Testing, Docker

docker run -d --name bitwarden -v /media/bitwarden:/data/ -p 4080:80 -p 3012:3012 --restart=always bitwardenrs/server:latest

  • Bitwarden_rs version: 1.13.1

  • Install method: Docker

  • Clients used: Firefox, Android

  • Reverse proxy and version: Apache

  • Other relevant information:

Steps to reproduce

Start the Docker container wand wait about half an hour, the web page is unaccessible.

Expected behaviour

I should be able to access the site nomatter how long the container was up and running.

Actual behaviour

The site is not reachable. The site is initally reachable for a while, but after sometime it is not reachable via all the IPs the server has or the reverse proxy (via Apache2)

Relevant logs

curl -v http://localhost:80/alive

 Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0)    
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)    
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
* Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 150000 ms for 3 (transfer 0x55d21c2f0dc0)
* Expire in 200 ms for 4 (transfer 0x55d21c2f0dc0)
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /alive HTTP/1.1
> Host: localhost
> User-Agent: curl/7.64.0
> Accept: */*
> 


root@9af1c2f0025e:/# bash healthcheck.sh
No results shown

Originally created by @gerroon on GitHub (Jan 19, 2020). Original GitHub issue: https://github.com/dani-garcia/vaultwarden/issues/822 ### Subject of the issue Hi I started getting a strange issue. The BW_RS container becomes unresponsive sometime after starting, when I sya unresponsive I mean unresponsive to browsing via clients like Firefox. I can neither access the local ip nor the reverse proxied url. The Docker logs shows nothing, it looks as if it is running normally. Here last couple lines from `docker logs -f bitwarden` ``` [2020-01-19 03:49:10][request][INFO] POST /notifications/hub/negotiate [2020-01-19 03:49:10][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK ``` ### Your environment Debian Testing, Docker `docker run -d --name bitwarden -v /media/bitwarden:/data/ -p 4080:80 -p 3012:3012 --restart=always bitwardenrs/server:latest` * Bitwarden_rs version: 1.13.1 * Install method: Docker * Clients used: Firefox, Android * Reverse proxy and version: Apache * Other relevant information: ### Steps to reproduce Start the Docker container wand wait about half an hour, the web page is unaccessible. ### Expected behaviour I should be able to access the site nomatter how long the container was up and running. ### Actual behaviour The site is not reachable. The site is initally reachable for a while, but after sometime it is not reachable via all the IPs the server has or the reverse proxy (via Apache2) ### Relevant logs `curl -v http://localhost:80/alive` ``` Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 1 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Expire in 0 ms for 1 (transfer 0x55d21c2f0dc0) * Trying 127.0.0.1... * TCP_NODELAY set * Expire in 150000 ms for 3 (transfer 0x55d21c2f0dc0) * Expire in 200 ms for 4 (transfer 0x55d21c2f0dc0) * Connected to localhost (127.0.0.1) port 80 (#0) > GET /alive HTTP/1.1 > Host: localhost > User-Agent: curl/7.64.0 > Accept: */* > ``` `root@9af1c2f0025e:/# bash healthcheck.sh` No results shown
kerem closed this issue 2026-03-03 01:30:42 +03:00
Author
Owner

@gerroon commented on GitHub (Jan 19, 2020):

One thing I also see

docker exec -ti bitwarden curl -v http://localhost:80/alive

* Expire in 0 ms for 1 (transfer 0x5573c1386dc0)
* Expire in 0 ms for 1 (transfer 0x5573c1386dc0)
* Expire in 0 ms for 1 (transfer 0x5573c1386dc0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 150000 ms for 3 (transfer 0x5573c1386dc0)
* Expire in 200 ms for 4 (transfer 0x5573c1386dc0)
*   Trying ::1...
* TCP_NODELAY set

However this line below does not produce those errors, not sure if ipv6 causing any issues if so why it would do so

docker exec -ti bitwarden curl -v http://127.0.0.1:80/alive

docker exec -ti bitwarden curl -v http://127.0.0.1:80/alive
* Expire in 0 ms for 6 (transfer 0x56181be35dc0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x56181be35dc0)

I feel like this must have been introduced in last 2 release becasue I never had this issue before. I can restart the container and it is back for a while and the check command runs successfully and I can login to site

Here I restarted the unresponsive bitwarden container.

docker exec -ti bitwarden curl -v http://127.0.0.1:80/alive

* Expire in 0 ms for 6 (transfer 0x55d289f54dc0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55d289f54dc0)
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET /alive HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Server: Rocket
< Feature-Policy: accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; sync-xhr 'self' https://haveibeenpwned.com https://twofactorauth.org; usb 'none'; vr 'none'
< Referrer-Policy: same-origin
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Content-Security-Policy: frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb moz-extension://*;
< Cache-Control: no-cache, no-store, max-age=0
< Access-Control-Allow-Origin: 
< Content-Length: 29
< Date: Sun, 19 Jan 2020 19:48:34 GMT
< 
* Connection #0 to host 127.0.0.1 left intact

<!-- gh-comment-id:576038885 --> @gerroon commented on GitHub (Jan 19, 2020): One thing I also see `docker exec -ti bitwarden curl -v http://localhost:80/alive` ``` * Expire in 0 ms for 1 (transfer 0x5573c1386dc0) * Expire in 0 ms for 1 (transfer 0x5573c1386dc0) * Expire in 0 ms for 1 (transfer 0x5573c1386dc0) * Trying 127.0.0.1... * TCP_NODELAY set * Expire in 150000 ms for 3 (transfer 0x5573c1386dc0) * Expire in 200 ms for 4 (transfer 0x5573c1386dc0) * Trying ::1... * TCP_NODELAY set ``` However this line below does not produce those errors, not sure if ipv6 causing any issues if so why it would do so `docker exec -ti bitwarden curl -v http://127.0.0.1:80/alive` ``` docker exec -ti bitwarden curl -v http://127.0.0.1:80/alive * Expire in 0 ms for 6 (transfer 0x56181be35dc0) * Trying 127.0.0.1... * TCP_NODELAY set * Expire in 200 ms for 4 (transfer 0x56181be35dc0) ``` I feel like this must have been introduced in last 2 release becasue I never had this issue before. I can restart the container and it is back for a while and the check command runs successfully and I can login to site Here I restarted the unresponsive bitwarden container. ``` docker exec -ti bitwarden curl -v http://127.0.0.1:80/alive * Expire in 0 ms for 6 (transfer 0x55d289f54dc0) * Trying 127.0.0.1... * TCP_NODELAY set * Expire in 200 ms for 4 (transfer 0x55d289f54dc0) * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0) > GET /alive HTTP/1.1 > Host: 127.0.0.1 > User-Agent: curl/7.64.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Server: Rocket < Feature-Policy: accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; sync-xhr 'self' https://haveibeenpwned.com https://twofactorauth.org; usb 'none'; vr 'none' < Referrer-Policy: same-origin < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < X-XSS-Protection: 1; mode=block < Content-Security-Policy: frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb moz-extension://*; < Cache-Control: no-cache, no-store, max-age=0 < Access-Control-Allow-Origin: < Content-Length: 29 < Date: Sun, 19 Jan 2020 19:48:34 GMT < * Connection #0 to host 127.0.0.1 left intact ```
Author
Owner

@gerroon commented on GitHub (Feb 4, 2020):

I cant figure this out :( I am really thinking that either FF addon or the Android app is crashing this. Because crashes/unrespnsiveness is almost like random. Sometimes it takes horus sometimes 10 min after restrating the container.

Noone else has this issue?

Here app is totally unresponsive (cant login to site, or sometimes the paeg cant even be opened), the web page cant be opened, the addons cant sync. And I just had to restart the conrainer just 10 mins ago.

Also here is the top view of the processes running inside the container.
https://paste.debian.net/hidden/bb8cd5f5/

[2020-02-04 02:02:51][start][INFO] Rocket has launched from http://0.0.0.0:80
[2020-02-04 02:03:10][request][INFO] POST /identity/connect/token
[2020-02-04 02:03:11][response][INFO] POST /identity/connect/token (login) => 200 OK
[2020-02-04 02:03:11][request][INFO] POST /notifications/hub/negotiate
[2020-02-04 02:03:11][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK
[2020-02-04 02:03:40][request][INFO] GET /icons/10.2.0.10/icon.png
[2020-02-04 02:03:40][response][INFO] GET /icons/<domain>/icon.png (icon) => 200 OK
[2020-02-04 02:03:40][request][INFO] GET /icons/10.2.0.11/icon.png
[2020-02-04 02:03:40][response][INFO] GET /icons/<domain>/icon.png (icon) => 200 OK
[2020-02-04 02:03:52][request][INFO] PUT /api/ciphers/8fe7a9b2-736e-49d9-bd97-730d8db2bf00
[2020-02-04 02:03:53][response][INFO] PUT /api/ciphers/<uuid> (put_cipher) => 200 OK
[2020-02-04 02:04:10][request][INFO] POST /identity/connect/token
[2020-02-04 02:04:10][response][INFO] POST /identity/connect/token (login) => 200 OK
[2020-02-04 02:04:10][request][INFO] POST /notifications/hub/negotiate
[2020-02-04 02:04:10][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK
[2020-02-04 02:04:10][request][INFO] PUT /api/ciphers/162e2911-4305-4665-8c84-504daf929c55
[2020-02-04 02:04:11][response][INFO] PUT /api/ciphers/<uuid> (put_cipher) => 200 OK

Here is straight curl to the site

 curl -v IP:PORT
*   Trying IP:PORT...
* TCP_NODELAY set
* connect to IP port PORT failed: Connection timed out
* Failed to connect to IP port PORT: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to IP port PORT: Connection timed out


Docker stats

ba363b229dbb        bitwarden                 0.01%               115.4MiB / 11.76GiB   0.96%               36MB / 7.06MB       168MB / 494MB       182
<!-- gh-comment-id:581708856 --> @gerroon commented on GitHub (Feb 4, 2020): I cant figure this out :( I am really thinking that either FF addon or the Android app is crashing this. Because crashes/unrespnsiveness is almost like random. Sometimes it takes horus sometimes 10 min after restrating the container. Noone else has this issue? Here app is totally unresponsive (cant login to site, or sometimes the paeg cant even be opened), the web page cant be opened, the addons cant sync. And I just had to restart the conrainer just 10 mins ago. Also here is the `top` view of the processes running inside the container. https://paste.debian.net/hidden/bb8cd5f5/ ``` [2020-02-04 02:02:51][start][INFO] Rocket has launched from http://0.0.0.0:80 [2020-02-04 02:03:10][request][INFO] POST /identity/connect/token [2020-02-04 02:03:11][response][INFO] POST /identity/connect/token (login) => 200 OK [2020-02-04 02:03:11][request][INFO] POST /notifications/hub/negotiate [2020-02-04 02:03:11][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK [2020-02-04 02:03:40][request][INFO] GET /icons/10.2.0.10/icon.png [2020-02-04 02:03:40][response][INFO] GET /icons/<domain>/icon.png (icon) => 200 OK [2020-02-04 02:03:40][request][INFO] GET /icons/10.2.0.11/icon.png [2020-02-04 02:03:40][response][INFO] GET /icons/<domain>/icon.png (icon) => 200 OK [2020-02-04 02:03:52][request][INFO] PUT /api/ciphers/8fe7a9b2-736e-49d9-bd97-730d8db2bf00 [2020-02-04 02:03:53][response][INFO] PUT /api/ciphers/<uuid> (put_cipher) => 200 OK [2020-02-04 02:04:10][request][INFO] POST /identity/connect/token [2020-02-04 02:04:10][response][INFO] POST /identity/connect/token (login) => 200 OK [2020-02-04 02:04:10][request][INFO] POST /notifications/hub/negotiate [2020-02-04 02:04:10][response][INFO] POST /notifications/hub/negotiate (negotiate) => 200 OK [2020-02-04 02:04:10][request][INFO] PUT /api/ciphers/162e2911-4305-4665-8c84-504daf929c55 [2020-02-04 02:04:11][response][INFO] PUT /api/ciphers/<uuid> (put_cipher) => 200 OK ``` Here is straight curl to the site ``` curl -v IP:PORT * Trying IP:PORT... * TCP_NODELAY set * connect to IP port PORT failed: Connection timed out * Failed to connect to IP port PORT: Connection timed out * Closing connection 0 curl: (28) Failed to connect to IP port PORT: Connection timed out ``` Docker stats ``` ba363b229dbb bitwarden 0.01% 115.4MiB / 11.76GiB 0.96% 36MB / 7.06MB 168MB / 494MB 182 ```
Author
Owner

@mprasil commented on GitHub (Feb 6, 2020):

This is strange. Just looking at the stats it seems to consume more memory than I'm used to seeing. (might not be relevant) Do you have many users?

It seems that the service just runs out of workers for whatever reason, can you share some apache configuration, that might be relevant. Perhaps websocket configuration isn't correct and it ends up using all the workers with persistent connections? (try to turn off websockets maybe?)

Also just total stab in the dark, but you can increase the number of worker threads (the default is 10):

docker run -d --name bitwarden \
  -v /media/bitwarden:/data/ \
  -p 4080:80 \
  -p 3012:3012 \
  --restart=always \
  -e ROCKET_WORKERS=50 \
  bitwardenrs/server:latest
<!-- gh-comment-id:582800001 --> @mprasil commented on GitHub (Feb 6, 2020): This is strange. Just looking at the stats it seems to consume more memory than I'm used to seeing. (might not be relevant) Do you have many users? It seems that the service just runs out of workers for whatever reason, can you share some apache configuration, that might be relevant. Perhaps websocket configuration isn't correct and it ends up using all the workers with persistent connections? (try to turn off websockets maybe?) Also just total stab in the dark, but you can increase the number of worker threads ([the default is 10](https://github.com/dani-garcia/bitwarden_rs/blob/8867626de898bb8416ed8319806b1c220d57dcb1/docker/amd64/sqlite/Dockerfile#L72)): ```bash docker run -d --name bitwarden \ -v /media/bitwarden:/data/ \ -p 4080:80 \ -p 3012:3012 \ --restart=always \ -e ROCKET_WORKERS=50 \ bitwardenrs/server:latest ```
Author
Owner

@gerroon commented on GitHub (Feb 6, 2020):

Hi

I have total 3 users.

Here is the apache conf. The rest is just couple extra lines about the site and ssl

        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /notifications/hub(.*) ws://<SERVER>:3012/$1 [P,L]
        ProxyPass / http://127.0.0.1:4030/

        ProxyPreserveHost On
        ProxyRequests Off
        RequestHeader set X-Real-IP %{REMOTE_ADDR}s


I will try increasing the workers to see if I get any remedy. Thanks for the pointers.

<!-- gh-comment-id:582992433 --> @gerroon commented on GitHub (Feb 6, 2020): Hi I have total 3 users. Here is the apache conf. The rest is just couple extra lines about the site and ssl ``` RewriteEngine On RewriteCond %{HTTP:Upgrade} =websocket [NC] RewriteRule /notifications/hub(.*) ws://<SERVER>:3012/$1 [P,L] ProxyPass / http://127.0.0.1:4030/ ProxyPreserveHost On ProxyRequests Off RequestHeader set X-Real-IP %{REMOTE_ADDR}s ``` I will try increasing the workers to see if I get any remedy. Thanks for the pointers.
Author
Owner

@jjlin commented on GitHub (Feb 6, 2020):

You have a lot of hung calls to the healthcheck script for some reason. What OS version and Docker version are you running? Also, maybe try disabling the healthcheck (docker run --no-healthcheck ...) and see if that helps.

<!-- gh-comment-id:583110000 --> @jjlin commented on GitHub (Feb 6, 2020): You have a lot of hung calls to the healthcheck script for some reason. What OS version and Docker version are you running? Also, maybe try disabling the healthcheck (`docker run --no-healthcheck ...`) and see if that helps.
Author
Owner

@gerroon commented on GitHub (Feb 6, 2020):

Hi

I have no idea why health checks runs like that since I do not get involved with the container myself.

I am on Debian Testing, Docker version 19.03.5

I will also try your recommendation

<!-- gh-comment-id:583111717 --> @gerroon commented on GitHub (Feb 6, 2020): Hi I have no idea why health checks runs like that since I do not get involved with the container myself. I am on Debian Testing, Docker version 19.03.5 I will also try your recommendation
Author
Owner

@cmroanirgo commented on GitHub (Feb 17, 2020):

Just browsing this fault and I notice either a) the issue or (more likely) b) a typo in this issue.

You state you start with: docker run ... -p 4080:80 (note port 4080)
Your apache conf is incorrect in using: ProxyPass / http://127.0.0.1:4030/ (note port 4030)

Perhaps my comment is a red herring, or this is not the actual problem...? You do state that it works for a while...

<!-- gh-comment-id:586877535 --> @cmroanirgo commented on GitHub (Feb 17, 2020): Just browsing this fault and I notice either a) the issue or (more likely) b) a typo in this issue. You state you start with: `docker run ... -p 4080:80` (note port 4080) Your apache conf is incorrect in using: `ProxyPass / http://127.0.0.1:4030/` (note port 4030) Perhaps my comment is a red herring, or this is not the actual problem...? You *do* state that it works for a while...
Author
Owner

@gerroon commented on GitHub (Feb 17, 2020):

Hi

I tried to anonymize some info, might have mistyped. That is not the real issue, since I can always access it when it works.

Increasing number of workes seemed to help. I will test a bit more.

<!-- gh-comment-id:587055830 --> @gerroon commented on GitHub (Feb 17, 2020): Hi I tried to anonymize some info, might have mistyped. That is not the real issue, since I can always access it when it works. Increasing number of workes seemed to help. I will test a bit more.
Author
Owner

@jjlin commented on GitHub (May 13, 2020):

Potentially related: #950

<!-- gh-comment-id:628288560 --> @jjlin commented on GitHub (May 13, 2020): Potentially related: #950
Author
Owner

@BlackDex commented on GitHub (Nov 18, 2020):

Closing this ticket because of inactivity.
Feel free to re-open if the issue isn't resolved using the testing/master version and updated docker software.

<!-- gh-comment-id:729646150 --> @BlackDex commented on GitHub (Nov 18, 2020): Closing this ticket because of inactivity. Feel free to re-open if the issue isn't resolved using the `testing`/`master` version and updated docker software.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/vaultwarden#575
No description provided.