[GH-ISSUE #1202 ] bitwarden_rs crashes from some kind of panic #847

Author

Owner

@BlackDex commented on GitHub (Oct 30, 2020):

Seems to be an issue with the favicon download.
Don't know why exactly, but need to debug this.
Thx for reporting and providing the logs.

@BlackDex commented on GitHub (Oct 30, 2020): Seems to be an issue with the favicon download. Don't know why exactly, but need to debug this. Thx for reporting and providing the logs.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Oct 30, 2020):

Is there something I can do to work around it in the meantime? I have added a restart entry in the systemd unit file, but because I access my bitwarden server remotely for the android client, I really would like to avoid it crashing.

@GNY-001F2 commented on GitHub (Oct 30, 2020): Is there something I can do to work around it in the meantime? I have added a restart entry in the systemd unit file, but because I access my bitwarden server remotely for the android client, I really would like to avoid it crashing.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Oct 30, 2020):

Until we fixed the bug you could disable icon downloading.
github.com/dani-garcia/bitwarden_rs@ec920b5756/.env.template (L97)

@BlackDex commented on GitHub (Oct 30, 2020): Until we fixed the bug you could disable icon downloading. https://github.com/dani-garcia/bitwarden_rs/blob/ec920b5756fdae0b64edde328f634fbe2cd92ac0/.env.template#L97

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Oct 30, 2020):

Thanks, I will do that and inform you if I still notice any crashes.

@GNY-001F2 commented on GitHub (Oct 30, 2020): Thanks, I will do that and inform you if I still notice any crashes.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Oct 31, 2020):

May i ask why you build it your self?
And i wonder what happens if you try to use a docker image using the same config and data folder.
If you go and try that, please backup the whole data-folder.

Because i can't seem to reproduce this issue at all for some reason.

@BlackDex commented on GitHub (Oct 31, 2020): May i ask why you build it your self? And i wonder what happens if you try to use a docker image using the same config and data folder. If you go and try that, please backup the whole data-folder. Because i can't seem to reproduce this issue at all for some reason.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Oct 31, 2020):

Maybe you could enable the debug log_level.
That would provide some more information.

@BlackDex commented on GitHub (Oct 31, 2020): Maybe you could enable the debug log_level. That would provide some more information.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Oct 31, 2020):

I built the image myself for two reasons:

I don't know how to use docker properly and it seemed safer to stick to known quantities.
I already had an external-facing setup that I didn't want to reconfigure too much.

I will enable the debug log_level and report what comes up. I won't be trying the docker instance because I'd like to keep my configuration simple and easy to understand.

@GNY-001F2 commented on GitHub (Oct 31, 2020): I built the image myself for two reasons: 1. I don't know how to use docker properly and it seemed safer to stick to known quantities. 2. I already had an external-facing setup that I didn't want to reconfigure too much. I will enable the debug log_level and report what comes up. I won't be trying the docker instance because I'd like to keep my configuration simple and easy to understand.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 18, 2020):

@GNY-001F2 Do you have any more insights regarding this issue?

@BlackDex commented on GitHub (Nov 18, 2020): @GNY-001F2 Do you have any more insights regarding this issue?

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Nov 19, 2020):

Hi, sorry I've just been so busy I never got around to doing anything.

@GNY-001F2 commented on GitHub (Nov 19, 2020): Hi, sorry I've just been so busy I never got around to doing anything.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Nov 24, 2020):

@BlackDex Sorry for the delay. I've just gotten around to setting log_level to debug in .env.

For the testing I've re-enabled icons download as well.

Hopefully the log file should have something worth looking at after a couple of days.

Please let me know if I've missed anything or if there are any additional settings you would like me to add to the setup.

@GNY-001F2 commented on GitHub (Nov 24, 2020): @BlackDex Sorry for the delay. I've just gotten around to setting log_level to debug in `.env`. For the testing I've re-enabled icons download as well. Hopefully the log file should have something worth looking at after a couple of days. Please let me know if I've missed anything or if there are any additional settings you would like me to add to the setup.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Nov 24, 2020):

Nope, i just hope debug will provide enough info to pinpoint the issue. Else we would need trace, but that is going to generate a bit to much info for my taste.

@BlackDex commented on GitHub (Nov 24, 2020): Nope, i just hope debug will provide enough info to pinpoint the issue. Else we would need trace, but that is going to generate a bit to much info for my taste.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Dec 4, 2020):

@BlackDex
bitwarden_rs_cut.log

The file ended up becoming 158MiB in size, so I've cut to the last 100k lines. I hope this is sufficient.

@GNY-001F2 commented on GitHub (Dec 4, 2020): @BlackDex [bitwarden_rs_cut.log](https://github.com/dani-garcia/bitwarden_rs/files/5644410/bitwarden_rs_cut.log) The file ended up becoming 158MiB in size, so I've cut to the last 100k lines. I hope this is sufficient.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 4, 2020):

Ug, i had this my self also today while coding on the favicon downloading. And i ended up disabling the html5ever from output. I will disable that in a new pr. But for that i need to first finish my favicon download changes.

I will look at the log asap when not on a phone hehe, that is not working.

Thx for it, hopefully there is something useful in there 👍

@BlackDex commented on GitHub (Dec 4, 2020): Ug, i had this my self also today while coding on the favicon downloading. And i ended up disabling the html5ever from output. I will disable that in a new pr. But for that i need to first finish my favicon download changes. I will look at the log asap when not on a phone hehe, that is not working. Thx for it, hopefully there is something useful in there 👍

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 7, 2020):

I'm not seeing any Panic in here unfortunately.

@BlackDex commented on GitHub (Dec 7, 2020): I'm not seeing any Panic in here unfortunately.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Dec 7, 2020):

@BlackDex the full log is 158MB, does github support files that large?

@GNY-001F2 commented on GitHub (Dec 7, 2020): @BlackDex the full log is 158MB, does github support files that large?

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 7, 2020):

I actually have no idea. but a panic would crash/stop bitwarden_rs, and thus should be at the end of the file.
But there is nowhere in the file you posted earlier any string with panic in there.

But, there is only one way to try it out right, try and upload it.
But if you can see there is a panic somewhere, i would just need that part, including a lot of lines above that, lets say, 2000 lines or something?

@BlackDex commented on GitHub (Dec 7, 2020): I actually have no idea. but a panic would crash/stop bitwarden_rs, and thus should be at the end of the file. But there is nowhere in the file you posted earlier any string with `panic` in there. But, there is only one way to try it out right, try and upload it. But if you can see there is a panic somewhere, i would just need that part, including a lot of lines above that, lets say, 2000 lines or something?

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Dec 8, 2020):

There is no panic in the logs, just some ERRORs. Is it possible that the panic state being reached by reqwest-internal-sync-runtime (according to journalctl) isn't dumping the panic data to the log?

@GNY-001F2 commented on GitHub (Dec 8, 2020): There is no ```panic``` in the logs, just some ```ERROR```s. Is it possible that the panic state being reached by `reqwest-internal-sync-runtime` (according to journalctl) isn't dumping the panic data to the log?

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 8, 2020):

Well, i am working on the icon downloading the past few days when i have time. And i found some issues which could case panic's. I am fixing those so that it will not panic any more, but just returns a 404 if it really couldn't find an icon and some other fail-safes.

@BlackDex commented on GitHub (Dec 8, 2020): Well, i am working on the icon downloading the past few days when i have time. And i found some issues which could case panic's. I am fixing those so that it will not panic any more, but just returns a 404 if it really couldn't find an icon and some other fail-safes.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 8, 2020):

@GNY-001F2: The PR ( #1262 ) Which fixes hopefully most panics has been merged. It will take a while before a new image is available on docker, but i would suggest to tryout the testing image when it's done, and keep debug enabled (I have removed a lot of noise) to see if this happens again, and also other issues you might encounter.

@BlackDex commented on GitHub (Dec 8, 2020): @GNY-001F2: The PR ( #1262 ) Which fixes hopefully most panics has been merged. It will take a while before a new image is available on docker, but i would suggest to tryout the `testing` image when it's done, and keep debug enabled (I have removed a lot of noise) to see if this happens again, and also other issues you might encounter.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 12, 2020):

Panics are logged to the bitwarden.log file so they should be in there.
Errors can sometimes occur, and depending on what kind of error's there could be a miss-configuration or infrastructure error, but that all depends on the error message it self.

I'm going to close this ticket for now since there now have been several changes to the icon downloading to prevent panics.
Please do report any panic you get, because we should solve and prevent those.

@BlackDex commented on GitHub (Dec 12, 2020): Panics are logged to the bitwarden.log file so they should be in there. Errors can sometimes occur, and depending on what kind of error's there could be a miss-configuration or infrastructure error, but that all depends on the error message it self. I'm going to close this ticket for now since there now have been several changes to the icon downloading to prevent panics. Please do report any panic you get, because we should solve and prevent those.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Dec 14, 2020):

@BlackDex Sorry for the delay, I was on vacation. I am using native builds rather than docker images as I mentioned earlier. Docker is complicated for me to set up and run alongside all my native server programs, and I'd rather not experiment with it for something as critical as a password manager.

Is there a testing version I could run natively on my system? I see the release tags are only up to v1.17.

@GNY-001F2 commented on GitHub (Dec 14, 2020): @BlackDex Sorry for the delay, I was on vacation. I am using native builds rather than docker images as I mentioned earlier. Docker is complicated for me to set up and run alongside all my native server programs, and I'd rather not experiment with it for something as critical as a password manager. Is there a testing version I could run natively on my system? I see the release tags are only up to v1.17.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 14, 2020):

Ah nice vacation 😊. Hope it was a nice one.
Instead of using a tagged one, use the master branch, that matches the testing tagged docker image, and will include all the latest fixes not yet tagged. Keep in mind that you probably need the latest web build also with the latest patches.

@BlackDex commented on GitHub (Dec 14, 2020): Ah nice vacation 😊. Hope it was a nice one. Instead of using a tagged one, use the master branch, that matches the `testing` tagged docker image, and will include all the latest fixes not yet tagged. Keep in mind that you probably need the latest web build also with the latest patches.

kerem commented

Author

Owner

@jjlin commented on GitHub (Dec 14, 2020):

It's probably easiest to just extract the bitwarden_rs executable and web-vault dir from the bitwardenrs/server:testing-alpine image. The Alpine build for a given platform (e.g., x86_64) is statically linked and should work on any Linux distro for that platform.

If you can't or don't want to install Docker, https://github.com/jjlin/docker-image-extract provides a shell script that pulls and extracts an image from Docker Hub, e.g.

$ ./docker-image-extract bitwardenrs/server:testing-alpine

@jjlin commented on GitHub (Dec 14, 2020): It's probably easiest to just extract the `bitwarden_rs` executable and `web-vault` dir from the `bitwardenrs/server:testing-alpine` image. The Alpine build for a given platform (e.g., `x86_64`) is statically linked and should work on any Linux distro for that platform. If you can't or don't want to install Docker, https://github.com/jjlin/docker-image-extract provides a shell script that pulls and extracts an image from Docker Hub, e.g. $ ./docker-image-extract bitwardenrs/server:testing-alpine

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 17, 2020):

@jjlin I double checked your statement regarding the static build, and it seems it is only true for the amd64 build for some reason. The armv7 build isn't, and those are the only Alpine based images.
Not sure though why.

@BlackDex commented on GitHub (Dec 17, 2020): @jjlin I double checked your statement regarding the static build, and it seems it is only true for the amd64 build for some reason. The armv7 build isn't, and those are the only Alpine based images. Not sure though why.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 17, 2020):

Hmm... maybe i need to check the actual alpine image instead of the debian based image :)

@BlackDex commented on GitHub (Dec 17, 2020): Hmm... maybe i need to check the actual alpine image instead of the debian based image :)

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Dec 17, 2020):

@BlackDex:

So I compiled to the latest commit on master, but the panic still exists. Log mode was warn rather than debug, so the output is certainly cleaner. In case it is relevant, Web vault was updated to 2.17.1 and patched accordingly.

Please see bitwarden_rs_100_tail.log

I have enabled debug mode to further diagnose the problem, because now something is also interfering with the auto-restart with systemd - I get a resource temporarily unavailable error.

Gonna run a few system updates today and let the log get generated in debug mode for the next couple of days, see what I find.

Oh, and as for the vacation, I'd say it was pretty good, thanks :)

@GNY-001F2 commented on GitHub (Dec 17, 2020): @BlackDex: So I compiled to the latest commit on master, but the panic still exists. Log mode was warn rather than debug, so the output is certainly cleaner. In case it is relevant, Web vault was updated to 2.17.1 and patched accordingly. Please see [bitwarden_rs_100_tail.log](https://github.com/dani-garcia/bitwarden_rs/files/5711289/bitwarden_rs_100_tail.log) I have enabled debug mode to further diagnose the problem, because now something is also interfering with the auto-restart with systemd - I get a ```resource temporarily unavailable``` error. Gonna run a few system updates today and let the log get generated in debug mode for the next couple of days, see what I find. Oh, and as for the vacation, I'd say it was pretty good, thanks :)

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 17, 2020):

@GNY-001F2, how have you configured the icon_download_timeout, if you have that configured?

And if you could give a few lines (50) from above the panic that would be nice, since there something triggered it.

@BlackDex commented on GitHub (Dec 17, 2020): @GNY-001F2, how have you configured the icon_download_timeout, if you have that configured? And if you could give a few lines (50) from above the panic that would be nice, since there something triggered it.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 17, 2020):

Also, on what kind of hardware is it running? What are the specs.
It seems to fail in spawning a new thread. Are there maybe limitations set for bitwarden_rs either via systemd or via the host?

@BlackDex commented on GitHub (Dec 17, 2020): Also, on what kind of hardware is it running? What are the specs. It seems to fail in spawning a new thread. Are there maybe limitations set for bitwarden_rs either via systemd or via the host?

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 17, 2020):

But, i also do wonder what happens if you use a pre-compiled version like the musl version from the amd64 docker image.
If that also generates the same panic.

@BlackDex commented on GitHub (Dec 17, 2020): But, i also do wonder what happens if you use a pre-compiled version like the musl version from the amd64 docker image. If that also generates the same panic.

kerem commented

bitwarden_rs_tail_1000.log

Author

Owner

@GNY-001F2 commented on GitHub (Dec 18, 2020):

@BlackDex I did not manually configure icon_download_timeout.

As for my specs:

CPU: i7 4770k
MOBO: Asus Z87I-Deluxe
RAM: Patriot 1600Mhz 2x8GB DDR3
HDDs: 2x 1TB WD Blues in RAID 1 with all server programs and 160GB OS HDD from hitachi (which had low hours on it since I shucked it from my PS3 years ago)
OS: OpenSUSE tumbleweed, with rust nightly toolchain downloaded via rustup.

Looking at the log file, all I see is a bunch of panics+crashes with different timestamps, instead of any other logs. The logs were in warn mode when this happened.

I've uploaded a 1000-line log; maybe this helps.

I've started a new logfile since yesterday's crash with debug mode on, and once I have another panic I'll post the new dump.

Also for whatever reason I am suspecting that the panics are not getting recorded in debug mode, so I'll let the program run in debug mode for the next 48 hours, and if I have nothing, I'll run it again in warn mode for another 48 hours just to see if any panics are recorded. I suspect this because journalctl records some exits at 00:01 IST on 18th December, but the logs between 23:59 on 17th and 00:03 IST on 18th are missing.

It could have been me trying to restart bitwarden_rs via systemd manually at that time, but I honestly don't remember all the steps I took to get the software back online.

@GNY-001F2 commented on GitHub (Dec 18, 2020): @BlackDex I did not manually configure icon_download_timeout. As for my specs: 1. CPU: i7 4770k 2. MOBO: Asus Z87I-Deluxe 3. RAM: Patriot 1600Mhz 2x8GB DDR3 4. HDDs: 2x 1TB WD Blues in RAID 1 with all server programs and 160GB OS HDD from hitachi (which had low hours on it since I shucked it from my PS3 years ago) 5. OS: OpenSUSE tumbleweed, with rust nightly toolchain downloaded via rustup. Looking at the log file, all I see is a bunch of panics+crashes with different timestamps, instead of any other logs. The logs were in warn mode when this happened. I've uploaded a 1000-line log; maybe this helps. [bitwarden_rs_tail_1000.log](https://github.com/dani-garcia/bitwarden_rs/files/5716523/bitwarden_rs_tail_1000.log) I've started a new logfile since yesterday's crash with debug mode on, and once I have another panic I'll post the new dump. Also for whatever reason I am suspecting that the panics are not getting recorded in debug mode, so I'll let the program run in debug mode for the next 48 hours, and if I have nothing, I'll run it again in warn mode for another 48 hours just to see if any panics are recorded. I suspect this because journalctl records some exits at 00:01 IST on 18th December, but the logs between 23:59 on 17th and 00:03 IST on 18th are missing. It could have been me trying to restart bitwarden_rs via systemd manually at that time, but I honestly don't remember all the steps I took to get the software back online.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 18, 2020):

Could you share your systemd service file?
Maybe it has something to do with running it via systemd

@BlackDex commented on GitHub (Dec 18, 2020): Could you share your systemd service file? Maybe it has something to do with running it via systemd

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 18, 2020):

@GNY-001F2 I just tested this within an LXD which is using openSuse Tumbleweed.
I executed the following commands:

#!/usr/bin/env bash

zypper up
zypper install curl vim git sqlite3 gcc sqlite3-devel openssl-devel
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Changed the defaults to nightly kept the rest the same as it is.

git clone --depth=1 https://github.com/dani-garcia/bitwarden_rs.git
cd bitwarden_rs/
cargo test --features sqlite
cargo build --features sqlite

cp .env.template .env
# Disabled the web-vault and web-sockets

./target/debug/bitwarden_rs


# Using a different shell i did the following
curl -o icon.png "http://127.0.0.1:8000/icons/github.com/icon.png"

Which resulted in no issues, doesn't matter if i put logging on info, warn, debug or trace.
Also, keep in mind that you probably want to clear your icon-cache to make sure it triggers a new download.

@BlackDex commented on GitHub (Dec 18, 2020): @GNY-001F2 I just tested this within an LXD which is using openSuse Tumbleweed. I executed the following commands: ```bash #!/usr/bin/env bash zypper up zypper install curl vim git sqlite3 gcc sqlite3-devel openssl-devel curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Changed the defaults to nightly kept the rest the same as it is. git clone --depth=1 https://github.com/dani-garcia/bitwarden_rs.git cd bitwarden_rs/ cargo test --features sqlite cargo build --features sqlite cp .env.template .env # Disabled the web-vault and web-sockets ./target/debug/bitwarden_rs # Using a different shell i did the following curl -o icon.png "http://127.0.0.1:8000/icons/github.com/icon.png" ``` Which resulted in no issues, doesn't matter if i put logging on `info`, `warn`, `debug` or `trace`. Also, keep in mind that you probably want to clear your icon-cache to make sure it triggers a new download.

kerem commented

bitwarden_rs22_400tail.log

Author

Owner

@GNY-001F2 commented on GitHub (Dec 22, 2020):

@BlackDex So I can report another crash today afternoon, this time while it was running in debug mode. Last 400 lines included.

@GNY-001F2 commented on GitHub (Dec 22, 2020): @BlackDex So I can report another crash today afternoon, this time while it was running in debug mode. Last 400 lines included. [bitwarden_rs22_400tail.log](https://github.com/dani-garcia/bitwarden_rs/files/5731063/bitwarden_rs22_400tail.log)

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 22, 2020):

The error is very strange. Since that part of the code should just run only once during the application start/first time icon fetching.

So i do not understand why, after first providing icons it panics somewhere in between. I assume lanip is something you changed? What happens if you manually try to fetch the icon for that IP? Does it panic then also?

@BlackDex commented on GitHub (Dec 22, 2020): The error is very strange. Since that part of the code should just run only once during the application start/first time icon fetching. So i do not understand why, after first providing icons it panics somewhere in between. I assume lanip is something you changed? What happens if you manually try to fetch the icon for that IP? Does it panic then also?

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Dec 22, 2020):

lanip is literally the ip address of a system on my LAN.

How do I fetch an icon manually? So I can test this.

@GNY-001F2 commented on GitHub (Dec 22, 2020): lanip is literally the ip address of a system on my LAN. How do I fetch an icon manually? So I can test this.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Dec 22, 2020):

# Using a different shell i did the following
curl -o icon.png "http://127.0.0.1:8000/icons/github.com/icon.png"
Which resulted in no issues, doesn't matter if i put logging on info, warn, debug or trace.
Also, keep in mind that you probably want to clear your icon-cache to make sure it triggers a new download.

See my previous post :).
Replace github.com with lanip and 127.0.0.1:8000 with your own host of course.

@BlackDex commented on GitHub (Dec 22, 2020): > ```shell > # Using a different shell i did the following > curl -o icon.png "http://127.0.0.1:8000/icons/github.com/icon.png" > ``` > > Which resulted in no issues, doesn't matter if i put logging on `info`, `warn`, `debug` or `trace`. > Also, keep in mind that you probably want to clear your icon-cache to make sure it triggers a new download. See my previous post :). Replace github.com with lanip and 127.0.0.1:8000 with your own host of course.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Dec 23, 2020):

@BlackDex After yesterday's report, the program crashed today as well, but it was with a different domain. Also worth noting is that instead of crashing after a response message it crashed after a request message. So the underlying cause at least multiple domains, if it is not independent of the domain names.

I had set up some restrictions on the number of restarts in the bitwarden_rs.service file, and I found that it was failing to restart with the error: "Resource temporarily unavailable".

Dec 23 23:06:33 myhost systemd[12208]: bitwarden_rs.service: Failed at step EXEC spawning /path/to/bitwarden_rs/target/release/bitwarden_rs: Resource temporarily unavailable

The panic also seems to have this common starting point with all my previous panics:

[2020-12-23 08:56:10.293][panic][ERROR] thread 'unnamed' panicked at 'called Result::unwrap()on anErr value: reqwest::Error { kind: Builder, source: Os { code: 11, kind: WouldBlock, m essage: "Resource temporarily unavailable" } }': src/api/icons.rs:38

The systemd service file is:

bitwarden_rs.service.txt

See my previous post :).

So I tried downloading the icons from both within the bitwarden host and from an external machine, and I can report the following:

In the bitwarden host, the file downloaded is actually HTML with a 404
The external host is unable to route to my bitwarden server, probably because I have it firewalled, so I used the canonical domain settings. This means external requests were routed via my nginx reverse proxy also located on the same server.
After a few requests being sent to the bitwarden_rs host, the program crashed and failed to restart.

@GNY-001F2 commented on GitHub (Dec 23, 2020): @BlackDex After yesterday's report, the program crashed today as well, but it was with a different domain. Also worth noting is that instead of crashing after a response message it crashed after a request message. So the underlying cause at least multiple domains, if it is not independent of the domain names. I had set up some restrictions on the number of restarts in the bitwarden_rs.service file, and I found that it was failing to restart with the error: "Resource temporarily unavailable". `Dec 23 23:06:33 myhost systemd[12208]: bitwarden_rs.service: Failed at step EXEC spawning /path/to/bitwarden_rs/target/release/bitwarden_rs: Resource temporarily unavailable` The panic also seems to have this common starting point with all my previous panics: `[2020-12-23 08:56:10.293][panic][ERROR] thread 'unnamed' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Builder, source: Os { code: 11, kind: WouldBlock, m essage: "Resource temporarily unavailable" } }': src/api/icons.rs:38` The systemd service file is: [bitwarden_rs.service.txt](https://github.com/dani-garcia/bitwarden_rs/files/5736451/bitwarden_rs.service.txt) > See my previous post :). So I tried downloading the icons from both within the bitwarden host and from an external machine, and I can report the following: 1. In the bitwarden host, the file downloaded is actually HTML with a 404 2. The external host is unable to route to my bitwarden server, probably because I have it firewalled, so I used the canonical domain settings. This means external requests were routed via my nginx reverse proxy also located on the same server. 3. After a few requests being sent to the bitwarden_rs host, the program crashed and failed to restart.

kerem commented

bitwarden_rs_tail_100_20210103.log

Author

Owner

@GNY-001F2 commented on GitHub (Jan 3, 2021):

@BlackDex: So, I finally got another crash, and this time it is without icons being enabled.

@GNY-001F2 commented on GitHub (Jan 3, 2021): @BlackDex: So, I _finally_ got another crash, and this time it is without icons being enabled. [bitwarden_rs_tail_100_20210103.log](https://github.com/dani-garcia/bitwarden_rs/files/5761692/bitwarden_rs_tail_100_20210103.log)

kerem commented

Author

Owner

@BlackDex commented on GitHub (Jan 4, 2021):

This really looks like it has something to do with multiple threads not working correctly in your setup/environment.

@BlackDex commented on GitHub (Jan 4, 2021): This really looks like it has something to do with multiple threads not working correctly in your setup/environment.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Jan 4, 2021):

Well, i quickly tested your systemd file, and for me the problem lies there.
It doesn't start at all for me.
I got is solved by commenting out (or increasing) the LimitNPROC.
I suggest that you comment it out, or at least increase it to something like 4096 or 8192 and see what happens.

@BlackDex commented on GitHub (Jan 4, 2021): Well, i quickly tested your systemd file, and for me the problem lies there. It doesn't start at all for me. I got is solved by commenting out (or increasing) the `LimitNPROC`. I suggest that you comment it out, or at least increase it to something like `4096` or `8192` and see what happens.

kerem commented

https://github.com/dani-garcia/bitwarden_rs/wiki/Setup-as-a-systemd-service

Author

Owner

@GNY-001F2 commented on GitHub (Jan 4, 2021):

Well the file is copied exactly from the one in the wiki:

I only tweaked a few minor settings, and LimitNPROC is not one of them. I'm going to comment it out and see how it goes, but LimitNPROC is meant for a limit to the number of processes, not threads right? Is bitwarden_rs using multiple processes by any chance?

@GNY-001F2 commented on GitHub (Jan 4, 2021): Well the file is copied _exactly_ from the one in the wiki: https://github.com/dani-garcia/bitwarden_rs/wiki/Setup-as-a-systemd-service I only tweaked a few minor settings, and `LimitNPROC` is not one of them. I'm going to comment it out and see how it goes, but LimitNPROC is meant for a limit to the number of processes, not threads right? Is bitwarden_rs using multiple processes by any chance?

kerem commented

Author

Owner

@BlackDex commented on GitHub (Jan 4, 2021):

Well, the wiki is not really a faultless source of information.
It is updated and maintained by the community, which could indicate some settings work for most, but not all.

LimitNPROC should limit the amount of processes a specific user is allowed to run.
I have seen some reports that it sometimes counts all process with the same uid/username even when it runs within a docker or lxd or some other cgroups based containerization.

But still, even if all those items arn't the case here, what i saw in the error messages looked like process/thread starting issues. So my bet is on this setting.

Let us know if this solves your issue, because then we should update the wiki.

@BlackDex commented on GitHub (Jan 4, 2021): Well, the wiki is not really a faultless source of information. It is updated and maintained by the community, which could indicate some settings work for most, but not all. `LimitNPROC` should limit the amount of processes a specific user is allowed to run. I have seen some reports that it sometimes counts all process with the same uid/username even when it runs within a docker or lxd or some other `cgroups` based containerization. But still, even if all those items arn't the case here, what i saw in the error messages looked like process/thread starting issues. So my bet is on this setting. Let us know if this solves your issue, because then we should update the wiki.

kerem commented

Author

Owner

@mqus commented on GitHub (Jan 9, 2021):

Hey! I created the file in the wiki and indeed there seem to be issues when running this containerized, see the pinned comments in e.g. https://aur.archlinux.org/packages/bitwarden_rs-sqlite . I'll add it to the wiki page.

@mqus commented on GitHub (Jan 9, 2021): Hey! I created the file in the wiki and indeed there seem to be issues when running this containerized, see the pinned comments in e.g. https://aur.archlinux.org/packages/bitwarden_rs-sqlite . I'll add it to the wiki page.

kerem commented

Author

Owner

@mqus commented on GitHub (Jan 9, 2021):

Oh, @grawlinson already added it, it's in the Troubleshooting section of the systemd article

@mqus commented on GitHub (Jan 9, 2021): Oh, @grawlinson already added it, it's in the [Troubleshooting section of the systemd article](https://github.com/dani-garcia/bitwarden_rs/wiki/Setup-as-a-systemd-service#service-fails-to-start-in-a-container)

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Jan 11, 2021):

@mqus: In my case the service was, in fact, running as a native program.

I don't think I have any crashes to report at present, but I'm not going to clear this until I've given it an uptime of at least 30 days to be absolutely sure, because indeed my last crash was with icons disabled.

@GNY-001F2 commented on GitHub (Jan 11, 2021): @mqus: In my case the service was, in fact, running as a native program. I don't think I have any crashes to report at present, but I'm not going to clear this until I've given it an uptime of at least 30 days to be absolutely sure, because indeed my last crash was with icons disabled.

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Jan 31, 2021):

@BlackDex I have not faced any crashes over the past three weeks or so, and today while updating my builds I have taken the opportunity to re-enable icons. Since the crashes were way more frequent with icons enabled than without, hopefully I'll know for sure by mid-February the that the real cause was in fact the systemd service file.

If not, we'll be back to square one lol...

@GNY-001F2 commented on GitHub (Jan 31, 2021): @BlackDex I have not faced any crashes over the past three weeks or so, and today while updating my builds I have taken the opportunity to re-enable icons. Since the crashes were way more frequent with icons enabled than without, hopefully I'll know for sure by mid-February the that the real cause was in fact the systemd service file. If not, we'll be back to square one lol...

kerem commented

Author

Owner

@BlackDex commented on GitHub (Feb 2, 2021):

Just as a reference: https://bitwardenrs.discourse.group/t/error-code-101-on-a-systemd-triggered-service/394/7

@BlackDex commented on GitHub (Feb 2, 2021): Just as a reference: https://bitwardenrs.discourse.group/t/error-code-101-on-a-systemd-triggered-service/394/7

kerem commented

Author

Owner

@GNY-001F2 commented on GitHub (Feb 5, 2021):

Just as a reference: https://bitwardenrs.discourse.group/t/error-code-101-on-a-systemd-triggered-service/394/7

I'm curious as to why LIMITNPROC would be required to be removed on LXC but not necessarily on a native installation.

@GNY-001F2 commented on GitHub (Feb 5, 2021): > Just as a reference: https://bitwardenrs.discourse.group/t/error-code-101-on-a-systemd-triggered-service/394/7 I'm curious as to why LIMITNPROC would be required to be removed on LXC but not necessarily on a native installation.

kerem commented

Author

Owner

@BlackDex commented on GitHub (Feb 5, 2021):

Just as a reference: https://bitwardenrs.discourse.group/t/error-code-101-on-a-systemd-triggered-service/394/7

I'm curious as to why LIMITNPROC would be required to be removed on LXC but not necessarily on a native installation.

See the following issues where they are discussing this:
https://github.com/systemd/systemd/issues/6011
https://github.com/lxc/lxd/issues/3336

If even the developers are debating where the issue is at, then i'm not really sure where the fault lays.
Maybe if you configure the ulimit your self manually it could work.

@BlackDex commented on GitHub (Feb 5, 2021): > > Just as a reference: https://bitwardenrs.discourse.group/t/error-code-101-on-a-systemd-triggered-service/394/7 > > I'm curious as to why LIMITNPROC would be required to be removed on LXC but not necessarily on a native installation. See the following issues where they are discussing this: https://github.com/systemd/systemd/issues/6011 https://github.com/lxc/lxd/issues/3336 If even the developers are debating where the issue is at, then i'm not really sure where the fault lays. Maybe if you configure the ulimit your self manually it could work.

kerem commented