[GH-ISSUE #230] "queue" directory accumulates thousands of .msg files over time #182

New issue

Closed

opened 2026-02-25 21:31:22 +03:00 by kerem · 1 comment

kerem commented

2026-02-25 21:31:22 +03:00

Owner

Originally created by @dgz on GitHub (Nov 28, 2020).
Original GitHub issue: https://github.com/ciur/papermerge/issues/230

Originally assigned to: @ciur on GitHub.

Description
The celery queue directory doesn't seem to empty itself.
I noticed a cpu would peg whenever the worker was started and while investigating the other bug I posted I came across the "queue" directory which contained what seemed like thousands of .msg files. Each was 538 bytes in size and in total amounted to about 1.1gb of disk space, it took several seconds for an 'ls' to complete.
Deleting the queue folder and restarting the worker brought the cpu back down to idle while it was running. However when running approximately 3 new .msg files are created every second so it will quickly grow again.

The dates on this original batch of files stretched back to when I started using Papermerge in the summer so I don't believe it has anything to do with recent commits.

-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:38 683446462_49072248-fd81-434e-bd87-d733d053d01a.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:38 683455488_b6cb611b-0298-43d7-b8eb-606b0cd9feaa.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:38 683476497_c47557f3-9259-4f0e-956b-9139bf2290a6.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:39 683506531_03e45a03-40d3-4b35-a86a-729b08569846.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:39 683519560_2a4d9f55-6e65-4cc6-a730-bec30508a6fb.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:39 683536566_81e32d1c-6230-4543-902f-dadf6d87592a.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:40 683566602_1a4879f5-f221-4c43-828f-d95082b65103.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:40 683583638_fd159ce8-1843-4f46-af52-1acf58718523.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:40 683596639_48af1c8e-ab52-4d11-b5ff-65be9279247c.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:41 683626671_5d51631a-e33c-4ed4-94a5-89d494686e16.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:41 683647712_238e5872-3c09-4e95-b4d1-d097da3875b5.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:41 683656708_0d59b6d4-303f-4c2a-a4de-2541a6c9180d.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg
-rw-rw-r--  1 papermerge papermerge   538 Nov 27 21:42 683685741_6f7a85f7-054b-47c6-85d0-70d522cc3a66.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg

The files all seem to have nearly the same contents with only the correlation_id and delivery_tag changing, two of them below:

{"body": "eyJ0YXNrX2lkIjogIjNjMjYzMWY3LWNiNTQtNDZkNC04OTllLTUxN2UwM2RjZmYzNCIsICJzdGF0dXMiOiAiU1VDQ0VTUyIsICJyZXN1bHQiOiBudWxsLCAidHJhY2ViYWNrIjogbnVsbCwgImNoaWxkcmVuIjogW119", "content-encoding": "utf-8", "content-type": "application/json", "headers": {}, "properties": {"correlation_id": "3c2631f7-cb54-46d4-899e-517e03dcff34", "delivery_mode": 1, "delivery_info": {"exchange": "", "routing_key": "6d6c36c3-5baf-34a4-b401-d9af345bf9f3"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "28ad325c-9970-4d18-9bdb-53c20d289b4c"}}

{"body": "eyJ0YXNrX2lkIjogIjJmMmQ3OTM4LTE1MjktNDE2Ni1iM2JjLTJjMDUzNjk2NDNhYyIsICJzdGF0dXMiOiAiU1VDQ0VTUyIsICJyZXN1bHQiOiBudWxsLCAidHJhY2ViYWNrIjogbnVsbCwgImNoaWxkcmVuIjogW119", "content-encoding": "utf-8", "content-type": "application/json", "headers": {}, "properties": {"correlation_id": "2f2d7938-1529-4166-b3bc-2c05369643ac", "delivery_mode": 1, "delivery_info": {"exchange": "", "routing_key": "6d6c36c3-5baf-34a4-b401-d9af345bf9f3"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "e83ebb0f-84cc-4007-9b69-9be9821c2401"}}

Info:

OS: Ubuntu 20.04 LTS
Database: SQLite
Papermerge Version: git master 69592d4

Probably should have noted in the other report I am running Papermerge per the manual way in the documentation using systemd user units running gunicorn and the worker.

Originally created by @dgz on GitHub (Nov 28, 2020). Original GitHub issue: https://github.com/ciur/papermerge/issues/230 Originally assigned to: @ciur on GitHub. **Description** The celery queue directory doesn't seem to empty itself. I noticed a cpu would peg whenever the worker was started and while investigating the other bug I posted I came across the "queue" directory which contained what seemed like thousands of .msg files. Each was 538 bytes in size and in total amounted to about 1.1gb of disk space, it took several seconds for an 'ls' to complete. Deleting the queue folder and restarting the worker brought the cpu back down to idle while it was running. However when running approximately 3 new .msg files are created every second so it will quickly grow again. The dates on this original batch of files stretched back to when I started using Papermerge in the summer so I don't believe it has anything to do with recent commits. ``` -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:38 683446462_49072248-fd81-434e-bd87-d733d053d01a.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:38 683455488_b6cb611b-0298-43d7-b8eb-606b0cd9feaa.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:38 683476497_c47557f3-9259-4f0e-956b-9139bf2290a6.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:39 683506531_03e45a03-40d3-4b35-a86a-729b08569846.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:39 683519560_2a4d9f55-6e65-4cc6-a730-bec30508a6fb.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:39 683536566_81e32d1c-6230-4543-902f-dadf6d87592a.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:40 683566602_1a4879f5-f221-4c43-828f-d95082b65103.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:40 683583638_fd159ce8-1843-4f46-af52-1acf58718523.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:40 683596639_48af1c8e-ab52-4d11-b5ff-65be9279247c.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:41 683626671_5d51631a-e33c-4ed4-94a5-89d494686e16.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:41 683647712_238e5872-3c09-4e95-b4d1-d097da3875b5.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:41 683656708_0d59b6d4-303f-4c2a-a4de-2541a6c9180d.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg -rw-rw-r-- 1 papermerge papermerge 538 Nov 27 21:42 683685741_6f7a85f7-054b-47c6-85d0-70d522cc3a66.6d6c36c3-5baf-34a4-b401-d9af345bf9f3.msg ``` The files all seem to have nearly the same contents with only the correlation_id and delivery_tag changing, two of them below: `{"body": "eyJ0YXNrX2lkIjogIjNjMjYzMWY3LWNiNTQtNDZkNC04OTllLTUxN2UwM2RjZmYzNCIsICJzdGF0dXMiOiAiU1VDQ0VTUyIsICJyZXN1bHQiOiBudWxsLCAidHJhY2ViYWNrIjogbnVsbCwgImNoaWxkcmVuIjogW119", "content-encoding": "utf-8", "content-type": "application/json", "headers": {}, "properties": {"correlation_id": "3c2631f7-cb54-46d4-899e-517e03dcff34", "delivery_mode": 1, "delivery_info": {"exchange": "", "routing_key": "6d6c36c3-5baf-34a4-b401-d9af345bf9f3"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "28ad325c-9970-4d18-9bdb-53c20d289b4c"}}` `{"body": "eyJ0YXNrX2lkIjogIjJmMmQ3OTM4LTE1MjktNDE2Ni1iM2JjLTJjMDUzNjk2NDNhYyIsICJzdGF0dXMiOiAiU1VDQ0VTUyIsICJyZXN1bHQiOiBudWxsLCAidHJhY2ViYWNrIjogbnVsbCwgImNoaWxkcmVuIjogW119", "content-encoding": "utf-8", "content-type": "application/json", "headers": {}, "properties": {"correlation_id": "2f2d7938-1529-4166-b3bc-2c05369643ac", "delivery_mode": 1, "delivery_info": {"exchange": "", "routing_key": "6d6c36c3-5baf-34a4-b401-d9af345bf9f3"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "e83ebb0f-84cc-4007-9b69-9be9821c2401"}}` **Info:** - OS: Ubuntu 20.04 LTS - Database: SQLite - Papermerge Version: git master 69592d4 Probably should have noted in the other report I am running Papermerge per the manual way in the documentation using systemd user units running gunicorn and the worker.

kerem

2026-02-25 21:31:22 +03:00

closed this issue
added the
bug
label

kerem commented

2026-02-25 21:31:23 +03:00

Author

Owner

@dgz commented on GitHub (Nov 28, 2020):

Nevermind! This is related to #198 and I see the workarounds are listed there already. Closing this one, sorry!

@dgz commented on GitHub (Nov 28, 2020): Nevermind! This is related to #198 and I see the workarounds are listed there already. Closing this one, sorry!