[GH-ISSUE #491] rabbitmq takes forever to start, fails, and still eats 100% CPU after started, if ulimit -n set to a high value #236

Closed
opened 2026-02-26 10:31:13 +03:00 by kerem · 14 comments
Owner

Originally created by @t-lo on GitHub (Sep 9, 2022).
Original GitHub issue: https://github.com/ONLYOFFICE/Docker-DocumentServer/issues/491

Do you want to request a feature or report a bug?

This is a bug report which includes a workaround (see below). Motivation for filing this issue is to share this workaround (which has cost me quite a bit of debugging time) with other affected users.

What is the current behavior?

When ulimit -n inside the document server container is set to a high value (depending on the on docker config) it takes multiple minutes to start, hanging at Starting RabbitMQ Messaging Server rabbitmq-server - which eventually fails - though the container continues to run. After that, a process (or thread?) erl_child_setup consumes 100% of a single CPU, and keeps running forever. The document server container is not usable at this point (health endpoint returns 502) because rabbitmq never started successfully.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

  1. check that ulimit -n is set to a high value inside of the onlyoffice container
    host $ docker run -ti --entrypoint /bin/bash onlyoffice/documentserver
    container $ ulimit -n
    1073741816
    
  2. Start the container
    host $ docker run onlyoffice/documentserver
    
  3. run htop on the host (which also shows container namespaced processes): shows start-stop-daemon [...] redis-server stuck for multiple minutes, consuming 100% CPU; then erl_child_setup doing the same.

What is the expected behavior?

  1. Container starts normally independent of ulimit -n setting inside of container
  2. If the start-up of any of the required components (rabbitmq, redis, documentserver, nginx, etc) fails, the container exits with an error.

Did this work in previous versions of DocumentServer?

Yes, but I'm unsure when it stopped working.

DocumentServer Docker tag:

  • dockerhub digest 8a1edcc13f9d
  • image ID 5a50e3a2d2ed

Host Operating System:

Fedora 36 w/ docker version 20.10.17, build 100c701

Workaround

Set ulimit for NOFILE to a lower value, either individually for the documentserver container or globally for all containers.

Individually: add (e.g.) --ulimit nofile=65536:65536 to the docker command line, or

   ...
   ulimits:
     nofile:
       soft: "65536"
       hard: "65536"
   ...

to your service configuration YAML for docker-compose.

Globally: Add --default-ulimit nofile=65536:65536 to the dockerd command line.

Originally created by @t-lo on GitHub (Sep 9, 2022). Original GitHub issue: https://github.com/ONLYOFFICE/Docker-DocumentServer/issues/491 **Do you want to request a *feature* or report a *bug*?** This is a bug report which includes a workaround (see below). Motivation for filing this issue is to share this workaround (which has cost me quite a bit of debugging time) with other affected users. **What is the current behavior?** When `ulimit -n` inside the document server container is set to a high value (depending on the on docker config) it takes multiple minutes to start, hanging at `Starting RabbitMQ Messaging Server rabbitmq-server` - which eventually fails - though the container continues to run. After that, a process (or thread?) `erl_child_setup` consumes 100% of a single CPU, and keeps running forever. The document server container is not usable at this point (health endpoint returns 502) because rabbitmq never started successfully. **If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.** 1. check that `ulimit -n` is set to a high value inside of the onlyoffice container ```shell host $ docker run -ti --entrypoint /bin/bash onlyoffice/documentserver container $ ulimit -n 1073741816 ``` 2. Start the container ```shell host $ docker run onlyoffice/documentserver ``` 3. run `htop` on the host (which also shows container namespaced processes): shows `start-stop-daemon [...] redis-server` stuck for multiple minutes, consuming 100% CPU; then `erl_child_setup` doing the same. **What is the expected behavior?** 1. Container starts normally independent of `ulimit -n` setting inside of container 2. If the start-up of any of the required components (rabbitmq, redis, documentserver, nginx, etc) fails, the container exits with an error. **Did this work in previous versions of DocumentServer?** Yes, but I'm unsure when it stopped working. **DocumentServer Docker tag:** * dockerhub digest 8a1edcc13f9d * image ID 5a50e3a2d2ed **Host Operating System:** Fedora 36 w/ docker version 20.10.17, build 100c701 **Workaround** Set `ulimit` for `NOFILE` to a lower value, either individually for the documentserver container or globally for all containers. Individually: add (e.g.) `--ulimit nofile=65536:65536` to the docker command line, or ```yaml ... ulimits: nofile: soft: "65536" hard: "65536" ... ``` to your service configuration YAML for docker-compose. Globally: Add `--default-ulimit nofile=65536:65536` to the dockerd command line.
kerem 2026-02-26 10:31:13 +03:00
Author
Owner

@ShockwaveNN commented on GitHub (Sep 9, 2022):

Hi, I understand thtat this issue is reported as workaround for a problem you've got

But shouldn't this issue also be reported to RabbittMQ developers? seems they got some problems with big ulimit limits?
If I understand this correctly

<!-- gh-comment-id:1242115514 --> @ShockwaveNN commented on GitHub (Sep 9, 2022): Hi, I understand thtat this issue is reported as workaround for a problem you've got But shouldn't this issue also be reported to RabbittMQ developers? seems they got some problems with big ulimit limits? If I understand this correctly
Author
Owner

@t-lo commented on GitHub (Sep 11, 2022):

Good point, but after some more digging I found this: https://github.com/docker-library/rabbitmq/issues/545
Seems to be fixed upstream.

<!-- gh-comment-id:1242963892 --> @t-lo commented on GitHub (Sep 11, 2022): Good point, but after some more digging I found this: https://github.com/docker-library/rabbitmq/issues/545 Seems to be fixed upstream.
Author
Owner

@ShockwaveNN commented on GitHub (Sep 11, 2022):

@t-lo Thanks for finding it

We use ubuntu as our base image, so I think it will take time until ubuntu will serve the version with the fix

<!-- gh-comment-id:1242964340 --> @ShockwaveNN commented on GitHub (Sep 11, 2022): @t-lo Thanks for finding it We use ubuntu as our base image, so I think it will take time until ubuntu will serve the version with the fix
Author
Owner

@t-lo commented on GitHub (Sep 11, 2022):

Might be worth considering setting ulimit -n 65536 explicitly in the container entry point (or the rabbitmq init script); this would work independent of a ubuntu upstream fix (and at the end of the day do the same thing the erlang code change in upstream rabbitmq does). This could even be taken from an env variable so docker users can override it if necessary.

<!-- gh-comment-id:1242965587 --> @t-lo commented on GitHub (Sep 11, 2022): Might be worth considering setting `ulimit -n 65536` explicitly in the container entry point (or the rabbitmq init script); this would work independent of a ubuntu upstream fix (and at the end of the day do the same thing the erlang code change in upstream rabbitmq does). This could even be taken from an env variable so docker users can override it if necessary.
Author
Owner

@ShockwaveNN commented on GitHub (Sep 11, 2022):

Thanks for this idea

I've create issue 58989 in our private issue tracker

Not sure if we will implement that, but at least we discuss it

<!-- gh-comment-id:1242976456 --> @ShockwaveNN commented on GitHub (Sep 11, 2022): Thanks for this idea I've create issue 58989 in our private issue tracker Not sure if we will implement that, but at least we discuss it
Author
Owner

@t-lo commented on GitHub (Sep 12, 2022):

I took a stab on this, see https://github.com/ONLYOFFICE/Docker-DocumentServer/pull/492.
It's more challenging than I thought since ubuntu 20.04 ignores /etc/security/limits.[conf|d/] in favour of systemd service file settings (LimitNOFILE=...) but the documentserver uses start-stop-daemon to run, ignoring systemd's limits in turn.
So /etc/default/rabbitmq-server seems like the best place to set this.

<!-- gh-comment-id:1243369485 --> @t-lo commented on GitHub (Sep 12, 2022): I took a stab on this, see https://github.com/ONLYOFFICE/Docker-DocumentServer/pull/492. It's more challenging than I thought since ubuntu 20.04 ignores `/etc/security/limits.[conf|d/]` in favour of systemd service file settings (`LimitNOFILE=...`) but the documentserver uses `start-stop-daemon` to run, ignoring systemd's limits in turn. So `/etc/default/rabbitmq-server` seems like the best place to set this.
Author
Owner

@ShockwaveNN commented on GitHub (Sep 12, 2022):

@t-lo Ok thanks, I'll notifiy our developers team

<!-- gh-comment-id:1243575546 --> @ShockwaveNN commented on GitHub (Sep 12, 2022): @t-lo Ok thanks, I'll notifiy our developers team
Author
Owner

@igwyd commented on GitHub (Dec 2, 2022):

Hello @t-lo, it's fixed at: https://github.com/ONLYOFFICE/Docker-DocumentServer/pull/530 and will be released in the next release.

<!-- gh-comment-id:1334866381 --> @igwyd commented on GitHub (Dec 2, 2022): Hello @t-lo, it's fixed at: https://github.com/ONLYOFFICE/Docker-DocumentServer/pull/530 and will be released in the next release.
Author
Owner

@t-lo commented on GitHub (Dec 2, 2022):

Thank you @igwyd ! What's the ETA of the next release?

(Also, I've updated PR #492 with a comment, feel free to close.)

<!-- gh-comment-id:1334886090 --> @t-lo commented on GitHub (Dec 2, 2022): Thank you @igwyd ! What's the ETA of the next release? (Also, I've updated PR #492 with a comment, feel free to close.)
Author
Owner

@igwyd commented on GitHub (Dec 2, 2022):

No release date yet.

<!-- gh-comment-id:1334895162 --> @igwyd commented on GitHub (Dec 2, 2022): No release date yet.
Author
Owner

@mkobel commented on GitHub (Jun 28, 2023):

As workaround I defined ulimits in the docker-compose file:

    ulimits:
      nofile: 65536
<!-- gh-comment-id:1611141777 --> @mkobel commented on GitHub (Jun 28, 2023): As workaround I defined ulimits in the docker-compose file: ```yml ulimits: nofile: 65536 ```
Author
Owner

@igwyd commented on GitHub (Jun 28, 2023):

Hello @t-lo, as far as I can see the problem is solved, can we close it?

<!-- gh-comment-id:1611423730 --> @igwyd commented on GitHub (Jun 28, 2023): Hello @t-lo, as far as I can see the problem is solved, can we close it?
Author
Owner

@Heath123 commented on GitHub (Aug 10, 2023):

I've realised that the OnlyOffice development server has been installed on my laptop for over a year and it must have been taking up a whole CPU core in the background the whole time! No wonder my battery life has been bad and my fan has been loud... After getting rid of it my idle CPU temperature dropped from 70°C to 40°C and my fan is far quieter where before it would run constantly

<!-- gh-comment-id:1673317417 --> @Heath123 commented on GitHub (Aug 10, 2023): I've realised that the OnlyOffice development server has been installed on my laptop for over a year and it must have been taking up a whole CPU core in the background the whole time! No wonder my battery life has been bad and my fan has been loud... After getting rid of it my idle CPU temperature dropped from 70°C to 40°C and my fan is far quieter where before it would run constantly
Author
Owner

@Rita-Bubnova commented on GitHub (Aug 11, 2023):

If it's resolved I'll close the issue. Feel free to comment or reopen it if you got further questions.

<!-- gh-comment-id:1674297034 --> @Rita-Bubnova commented on GitHub (Aug 11, 2023): If it's resolved I'll close the issue. Feel free to comment or reopen it if you got further questions.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/Docker-DocumentServer-ONLYOFFICE#236
No description provided.