[PR #550] [CLOSED] Improve performance of enqueueing tasks #1781

New issue

Closed

opened 2026-03-07 22:13:12 +03:00 by kerem · 0 comments

kerem commented

2026-03-07 22:13:12 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hibiken/asynq/pull/550
Author: @yousifh
Created: 9/25/2022
Status: ❌ Closed

Base: master ← Head: enqueue-sadd-cache-optimize

📝 Commits (1)

11acdba Improve performance of enqueueing tasks

📊 Changes

2 files changed (+123 additions, -17 deletions)

View changed files

📝 internal/rdb/rdb.go (+75 -17)
📝 internal/rdb/rdb_test.go (+48 -0)

📄 Description

Shows a potential solution to #549

Create a map of queue names -> expiration dates in the RDB client. Every time we enqueue a task to queue X, cache queue X in the map and expire it after 10 secs. A mutex is used to protect the access, but there isn't a concern of lock contention as the clients will spend most of their time issuing network calls to enqueue tasks, and acquiring locks to check expiration is very fast.

The issue highlights the performance improvements but will re-post them here as well

	Before	After
Producer CPU	195 mcores	173 mcores
Redis CPU	290 ms	222 ms
Redis cmds/sec	8300	6600

All deployments were done in a normal k8s environment and metrics were measured through Datadog

Caveat:
This approach keeps the in-memory map of queues forever. It doesn't delete the entry when it expires. This is much simpler to implement, but if there is an app that enqueues tasks to millions of randomly named queues and the app never gets restarted, there is a potential memory leak. However that seems to be an unlikely use case of this library.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hibiken/asynq/pull/550 **Author:** [@yousifh](https://github.com/yousifh) **Created:** 9/25/2022 **Status:** ❌ Closed **Base:** `master` ← **Head:** `enqueue-sadd-cache-optimize` --- ### 📝 Commits (1) - [`11acdba`](https://github.com/hibiken/asynq/commit/11acdba32d6f6b62e482efc3c7663fc078029d91) Improve performance of enqueueing tasks ### 📊 Changes **2 files changed** (+123 additions, -17 deletions) <details> <summary>View changed files</summary> 📝 `internal/rdb/rdb.go` (+75 -17) 📝 `internal/rdb/rdb_test.go` (+48 -0) </details> ### 📄 Description Shows a potential solution to #549 Create a map of queue names -> expiration dates in the RDB client. Every time we enqueue a task to queue X, cache queue X in the map and expire it after 10 secs. A mutex is used to protect the access, but there isn't a concern of lock contention as the clients will spend most of their time issuing network calls to enqueue tasks, and acquiring locks to check expiration is very fast. The issue highlights the performance improvements but will re-post them here as well | | Before | After | | --- | ---- | --- | | Producer CPU | 195 mcores | 173 mcores | | Redis CPU | 290 ms | 222 ms | | Redis cmds/sec | 8300 | 6600 | All deployments were done in a normal k8s environment and metrics were measured through Datadog Caveat: This approach keeps the in-memory map of queues forever. It doesn't delete the entry when it expires. This is much simpler to implement, but if there is an app that enqueues tasks to millions of randomly named queues and the app never gets restarted, there is a potential memory leak. However that seems to be an unlikely use case of this library. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-07 22:13:12 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#1781

No description provided.

Rows
Columns