[PR #946] [MERGED] Improve performance of enqueueing tasks #2957

New issue

Closed

opened 2026-03-15 21:12:35 +03:00 by kerem · 0 comments

kerem commented

2026-03-15 21:12:35 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hibiken/asynq/pull/946
Author: @pior
Created: 10/28/2024
Status: ✅ Merged
Merged: 10/30/2024
Merged by: @kamikazechaser

Base: master ← Head: optimize-enqueue-sadd

📝 Commits (3)

1ad70c3 Improve performance of enqueueing tasks
2e99b71 Use sync.Map to simplify the conditional SADD
24da63e Cleanup queuePublished in RemoveQueue

📊 Changes

3 files changed (+89 additions, -15 deletions)

View changed files

📝 internal/rdb/inspect.go (+2 -1)
📝 internal/rdb/rdb.go (+34 -14)
📝 internal/rdb/rdb_test.go (+53 -0)

📄 Description

Context

Before enqueueing every task, the name of the queue is published to a Redis Set key (base.AllQueues).
This is only used by the inspector for inspection/monitoring purposes (not for the processing of tasks).

Two consequences:

Enqueuing one task requires two round-trip to Redis
The same Redis set key is updated constantly, causing significant Redis load

The issue #549 has more details about the impact on performance.
We've confirmed the figures in production.

Fixes #549
Rework of #550

Changes

Update the RDB client to only write to base.AllQueues once per worker

Notes:

This light implementation does not protect against race-conditions, which is not an issue.
Writing to base.AllQueues will still be retried until it worked.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hibiken/asynq/pull/946 **Author:** [@pior](https://github.com/pior) **Created:** 10/28/2024 **Status:** ✅ Merged **Merged:** 10/30/2024 **Merged by:** [@kamikazechaser](https://github.com/kamikazechaser) **Base:** `master` ← **Head:** `optimize-enqueue-sadd` --- ### 📝 Commits (3) - [`1ad70c3`](https://github.com/hibiken/asynq/commit/1ad70c359b2b212aaf01f3e4a56d0978d25ae391) Improve performance of enqueueing tasks - [`2e99b71`](https://github.com/hibiken/asynq/commit/2e99b71a49786ab153281dc16e79c4e4b6643b06) Use sync.Map to simplify the conditional SADD - [`24da63e`](https://github.com/hibiken/asynq/commit/24da63e6d2eb7e20c3acf2368569ab2752baca1b) Cleanup queuePublished in RemoveQueue ### 📊 Changes **3 files changed** (+89 additions, -15 deletions) <details> <summary>View changed files</summary> 📝 `internal/rdb/inspect.go` (+2 -1) 📝 `internal/rdb/rdb.go` (+34 -14) 📝 `internal/rdb/rdb_test.go` (+53 -0) </details> ### 📄 Description ## Context Before enqueueing **every** task, the name of the queue is published to a Redis Set key (`base.AllQueues`). This is only used by the inspector for inspection/monitoring purposes (not for the processing of tasks). Two consequences: - Enqueuing one task requires two round-trip to Redis - The same Redis set key is updated constantly, causing significant Redis load The issue #549 has more details about the impact on performance. We've confirmed the figures in production. Fixes #549 Rework of #550 ## Changes - Update the RDB client to only write to `base.AllQueues` once per worker Notes: - This light implementation does not protect against race-conditions, which is not an issue. - Writing to `base.AllQueues` will still be retried until it worked. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-15 21:12:35 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2957

No description provided.

Rows
Columns