[GH-ISSUE #119] [FEATURE REQUEST] Support unique jobs

kerem commented

2026-03-07 22:04:39 +03:00

Owner

Originally created by @cayter on GitHub (Mar 15, 2020).
Original GitHub issue: https://github.com/hibiken/asynq/issues/119

Originally assigned to: @hibiken on GitHub.

Is your feature request related to a problem? Please describe.
Any chance we would be supporting this? More details can be found here. Thanks.

Originally created by @cayter on GitHub (Mar 15, 2020). Original GitHub issue: https://github.com/hibiken/asynq/issues/119 Originally assigned to: @hibiken on GitHub. **Is your feature request related to a problem? Please describe.** Any chance we would be supporting this? More details can be found [here](https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs). Thanks.

kerem

2026-03-07 22:04:39 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-07 22:04:41 +03:00

Author

Owner

@hibiken commented on GitHub (Mar 15, 2020):

Thanks for opening this issue!

I've seen Sidekiq and other libraries supporting uniqueness of tasks(jobs) in Redis. It sounds like a useful feature to include, and I'm onboard with adding this feature to this package.

I haven't personally used this feature in the past. Would you mind adding more context as to why/when you would need this feature?

I've taken a look at Sidekiq's uniqueness feature and it seems like they support uniqueness based on (type, args, queue) of a task, which seems reasonable. The semantics around uniqueness TTL (unique_for: <duration>) is a bit confusing to me, especially when you consider task retries.
For example, in Sidekiq, if you set unique_for: 10.minutes in MyWorker class, a job of that class is guaranteed to be unique in the next 10 mins or until that first job is processed successfully. But the doc states that

If your job retries for a while, 10 minutes can pass, thus allowing another copy of the same job to be pushed to Redis. Design your jobs so that uniqueness is considered best effort, not a 100% guarantee

I'm not quite sure if this is the behavior we want from the uniqueness feature.

I think I need to think this through and come up with something more intuitive and simpler to understand (better for both implementors & users of the package).

@hibiken commented on GitHub (Mar 15, 2020): Thanks for opening this issue! I've seen Sidekiq and other libraries supporting uniqueness of tasks(jobs) in Redis. It sounds like a useful feature to include, and I'm onboard with adding this feature to this package. I haven't personally used this feature in the past. Would you mind adding more context as to why/when you would need this feature? I've taken a look at Sidekiq's uniqueness feature and it seems like they support uniqueness based on `(type, args, queue)` of a task, which seems reasonable. The semantics around uniqueness TTL (`unique_for: <duration>`) is a bit confusing to me, especially when you consider task retries. For example, in Sidekiq, if you set `unique_for: 10.minutes` in `MyWorker` class, a job of that class is guaranteed to be unique in the next 10 mins or until that first job is processed successfully. But the doc states that > If your job retries for a while, 10 minutes can pass, thus allowing another copy of the same job to be pushed to Redis. Design your jobs so that uniqueness is considered best effort, not a 100% guarantee I'm not quite sure if this is the behavior we want from the uniqueness feature. I think I need to think this through and come up with something more intuitive and simpler to understand (better for both implementors & users of the package).

kerem commented

2026-03-07 22:04:42 +03:00

Author

Owner

@cayter commented on GitHub (Mar 16, 2020):

Here's 1 use case example: https://blog.francium.tech/avoiding-duplicate-jobs-in-sidekiq-dcbb1aca1e20.

In general, when we have a set of servers running behind a load balancer, it's very likely our servers would enqueue multiple jobs that do the same thing. As such, having the unique job feature would ensure that we don't enqueue the same job multiple times and waste redundant computing resource(especially when it's writing into the SQL DB).

In my current job where payment gateways would sometimes hit our webhook endpoint with the exact same payload multiple times(can be within a short period of time), this leads to some issues on updating the same data in the SQL database which can be redundant/incorrect.

@cayter commented on GitHub (Mar 16, 2020): Here's 1 use case example: https://blog.francium.tech/avoiding-duplicate-jobs-in-sidekiq-dcbb1aca1e20. In general, when we have a set of servers running behind a load balancer, it's very likely our servers would enqueue multiple jobs that do the same thing. As such, having the unique job feature would ensure that we don't enqueue the same job multiple times and waste redundant computing resource(especially when it's writing into the SQL DB). In my current job where payment gateways would sometimes hit our webhook endpoint with the exact same payload multiple times(can be within a short period of time), this leads to some issues on updating the same data in the SQL database which can be redundant/incorrect.

kerem commented

2026-03-07 22:04:42 +03:00

Author

Owner

@hibiken commented on GitHub (Mar 16, 2020):

I see. Thank you for the link and the explanation!

After reading the use cases and Sidekiq's wiki, it sounds like it's reasonable to have best-effort uniqueness (i.e., If unique TTL expires then it's okay to enqueue a duplicate task) .

Here's my initial proposal for the API of the feature (Please provide feedback!).

Provide a function to create uniqueness option.

// Unique returns an option to specify uniqueness lock for the given task.
//
// Uniqueness of a task is based on (type, payload, queue).
func Unique(ttl time.Duration) Option

Example of enqueueing a task:

// task will be enqueued if duplicates don't exist.
// if enqueued, duplicate tasks won't be enqueued for the next 10 minutes.
err := client.Enqueue(t, asynq.Unique(10*time.Minute))

Example of scheduling a task (I have two proposals):

Option 1: Treat TTL the same as Enqueue, so TTL behaves the same.

// task is unique for the next 70 minutes.
err := client.EnqueueIn(time.Hour, t, asynq.Unique(70 * time.Minute)

Option 2: TTL behaves differently when using scheduling (TTL is set to delay + duration_provided_by_unique)

// task is unique for the next 70 minutes (delay + unique TTL).
err := client.EnqueueIn(time.Hour, t, asynq.Unique(10 * time.Minute)

I'm not sure if we want to treat uniqueness TTL differently when scheduling like Sidekiq does. Let me know your thoughts.

Return ErrDuplicateTask when using Unique option and duplicate exists.

err := client.Enqueue(t, asynq.Unique(10 * time.Minute))
if errors.Is(err, asynq.ErrDuplicateTask) {
   // logic to handle duplicate, if any.
}

Retry should be treated the same as Sidekiq does:

A task that is pending retry will still hold the unique lock and prevent further task) from being enqueued until the retry succeeds or the timeout passes. Manually removing the task from the retry queue will not remove the lock.

I don't think we need this Unlock Policy. Let me know if you disagree!

For bypassing uniqueness, I suggest that we also provide an option to do so:

// IgnoreUnique returns an option to ignore task uniqueness when enqueuing. 
func IgnoreUnique() Option

Example for ignoring uniqueness:

// Really want to push this task, bypass uniqueness constraint!
err := client.Enqueue(t, asynq.IgnoreUnique())

Let me know your thoughts and feedback!

@hibiken commented on GitHub (Mar 16, 2020): I see. Thank you for the link and the explanation! After reading the use cases and Sidekiq's wiki, it sounds like it's reasonable to have **best-effort uniqueness** (i.e., If unique TTL expires then it's okay to enqueue a duplicate task) . ### Here's my initial proposal for the API of the feature (Please provide feedback!). Provide a function to create uniqueness option. ```go // Unique returns an option to specify uniqueness lock for the given task. // // Uniqueness of a task is based on (type, payload, queue). func Unique(ttl time.Duration) Option ``` Example of enqueueing a task: ```go // task will be enqueued if duplicates don't exist. // if enqueued, duplicate tasks won't be enqueued for the next 10 minutes. err := client.Enqueue(t, asynq.Unique(10*time.Minute)) ``` Example of scheduling a task (I have two proposals): *Option 1*: Treat TTL the same as `Enqueue`, so TTL behaves the same. ```go // task is unique for the next 70 minutes. err := client.EnqueueIn(time.Hour, t, asynq.Unique(70 * time.Minute) ``` *Option 2*: TTL behaves differently when using scheduling (TTL is set to `delay + duration_provided_by_unique`) ```go // task is unique for the next 70 minutes (delay + unique TTL). err := client.EnqueueIn(time.Hour, t, asynq.Unique(10 * time.Minute) ``` I'm not sure if we want to treat uniqueness TTL differently when scheduling[ like Sidekiq does](https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs#scheduled-jobs). Let me know your thoughts. Return `ErrDuplicateTask` when using `Unique` option and duplicate exists. ```go err := client.Enqueue(t, asynq.Unique(10 * time.Minute)) if errors.Is(err, asynq.ErrDuplicateTask) { // logic to handle duplicate, if any. } ``` Retry should be treated the same as Sidekiq does: > A task that is pending retry will still hold the unique lock and prevent further task) from being enqueued until the retry succeeds or the timeout passes. Manually removing the task from the retry queue will not remove the lock. I don't think we need this [Unlock Policy](https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs#unlock-policy). Let me know if you disagree! For bypassing uniqueness, I suggest that we also provide an option to do so: ```go // IgnoreUnique returns an option to ignore task uniqueness when enqueuing. func IgnoreUnique() Option ``` Example for ignoring uniqueness: ```go // Really want to push this task, bypass uniqueness constraint! err := client.Enqueue(t, asynq.IgnoreUnique()) ``` Let me know your thoughts and feedback!

kerem commented

2026-03-07 22:04:42 +03:00

Author

Owner

@cayter commented on GitHub (Mar 17, 2020):

For example of scheduling a task...

I am leaning towards option 2 as this library is gonna be made friendly to those who had used Sidekiq before.

For bypassing uniqueness...

The suggestion is good.

For ignoring uniqueness...

The suggestion is good.

I don't think we need this Unlock Policy. Let me know if you disagree!

I just roughly looked into the possible options here. I can imagine how much efforts we need to put into to support all of them. So, I wouldn't be pushing for it now as I think the default policy which is unlocking right after a job is successfully processed should be sufficient for most use cases.

Question
We're planning to support this uniqueness across multiple workers yeah? Thanks!

@cayter commented on GitHub (Mar 17, 2020): > For example of scheduling a task... I am leaning towards option 2 as this library is gonna be made friendly to those who had used Sidekiq before. > For bypassing uniqueness... The suggestion is good. > For ignoring uniqueness... The suggestion is good. > I don't think we need this Unlock Policy. Let me know if you disagree! I just roughly looked into the possible options [here](https://github.com/mhenrixon/sidekiq-unique-jobs#locks). I can imagine how much efforts we need to put into to support all of them. So, I wouldn't be pushing for it now as I think the default policy which is unlocking right after a job is successfully processed should be sufficient for most use cases. **Question** We're planning to support this uniqueness across multiple workers yeah? Thanks!

kerem commented

2026-03-07 22:04:42 +03:00

Author

Owner

@hibiken commented on GitHub (Mar 17, 2020):

We're planning to support this uniqueness across multiple workers yeah?

If your worker processes are connecting to the same Redis instance, then answer is yes. My thought is to create a lock key in Redis with the given TTL and check that before inserting the task into Redis.

I'm currently planning the feature right now. I should be able to get started on this feature in the next few days 👍

Additional Note: While I was planning this feature, I realized that we do not need IgnoreUnique option since simply omitting the Unique option when enqueuing should give you the same effect.

@hibiken commented on GitHub (Mar 17, 2020): > We're planning to support this uniqueness across multiple workers yeah? If your worker processes are connecting to the same Redis instance, then answer is yes. My thought is to create a lock key in Redis with the given TTL and check that before inserting the task into Redis. I'm currently planning the feature right now. I should be able to get started on this feature in the next few days 👍 **Additional Note**: While I was planning this feature, I realized that we do not need `IgnoreUnique` option since simply omitting the `Unique` option when enqueuing should give you the same effect.

kerem referenced this issue

2026-03-07 22:10:49 +03:00

[GH-ISSUE #1046] [BUG] Scheduler Tasks Frequently Skipped When Large Number of Handlers Registered #1512

kerem referenced this issue

2026-03-15 20:46:41 +03:00

[GH-ISSUE #1046] [BUG] Scheduler Tasks Frequently Skipped When Large Number of Handlers Registered #2522

Rows
Columns

[GH-ISSUE #119] [FEATURE REQUEST] Support unique jobs #1046

Here's my initial proposal for the API of the feature (Please provide feedback!).