[PR #921] [CLOSED] Call LREM with count=1 on the active queue item #2937

New issue

Closed

opened 2026-03-15 21:11:30 +03:00 by kerem · 0 comments

kerem commented

2026-03-15 21:11:30 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hibiken/asynq/pull/921
Author: @pior
Created: 10/14/2024
Status: ❌ Closed

Base: master ← Head: lrem-1

📝 Commits (1)

e0ab52e Call LREM with count=-1 on the active queue item

📊 Changes

1 file changed (+7 additions, -7 deletions)

View changed files

📝 internal/rdb/rdb.go (+7 -7)

📄 Description

When a task is done, marked as completed, archived or requeued, it is removed from the active list in Redis.
This operation is done by a LREM operation with count = 0 (remove all elements matching the value).

For 100 concurrently active tasks or so, LREM is not an issue.
For 1000 concurrent active tasks or more, the O(N+M) complexity of LREM seems to become a significant waste of Redis cpu time.

It represents almost 50% of the the Redis CPU time in some of our queues.

Note: we plan to test this change in a realistic environment, which is why the PR is in Draft.

Issue: https://github.com/hibiken/asynq/issues/389

Solution

Change the LREM count parameter from 0 to -1 (deletes up to 1 element, starting from the tail, doc).

Tasks pushed with RPOPLPUSH should generally be at the tail when they are done/completed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hibiken/asynq/pull/921 **Author:** [@pior](https://github.com/pior) **Created:** 10/14/2024 **Status:** ❌ Closed **Base:** `master` ← **Head:** `lrem-1` --- ### 📝 Commits (1) - [`e0ab52e`](https://github.com/hibiken/asynq/commit/e0ab52e38a3a1109bfa40702bcd97884d95e79cc) Call LREM with count=-1 on the active queue item ### 📊 Changes **1 file changed** (+7 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `internal/rdb/rdb.go` (+7 -7) </details> ### 📄 Description When a task is done, marked as completed, archived or requeued, it is removed from the `active` list in Redis. This operation is done by a `LREM` operation with `count = 0` (remove **all** elements matching the value). For 100 concurrently active tasks or so, `LREM` is not an issue. For 1000 concurrent active tasks or more, the `O(N+M)` complexity of LREM seems to become a significant waste of Redis cpu time. It represents almost 50% of the the Redis CPU time in some of our queues. Note: we plan to test this change in a realistic environment, which is why the PR is in Draft. Issue: https://github.com/hibiken/asynq/issues/389 ### Solution Change the LREM count parameter from 0 to -1 (deletes up to 1 element, starting from the tail, [doc](https://redis.io/docs/latest/commands/lrem/)). Tasks pushed with RPOPLPUSH should generally be at the tail when they are done/completed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-15 21:11:30 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2937

No description provided.

Rows
Columns