starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-26 07:25:56 +03:00

[GH-ISSUE #326] [FEATURE REQUEST] Introduce Task Locals #1156

New issue

Closed

opened 2026-03-07 22:06:36 +03:00 by kerem · 3 comments

kerem commented

2026-03-07 22:06:36 +03:00

Owner

Originally created by @hibiken on GitHub (Sep 14, 2021).
Original GitHub issue: https://github.com/hibiken/asynq/issues/326

Originally assigned to: @hibiken on GitHub.

Is your feature request related to a problem? Please describe.
In some cases, we have middleware functions that want to add more context or decorate task so that downstream handler functions can retrieve that data. Similar issues: #285 #313

Describe the solution you'd like
To allow users to store local data within a handling of task, we can provide map[string]interface{} as "task locals".
Task locals is not persisted so any modification to the map doesn't persist across retries or locals set before enqueue is not visible to the handler code. This local should be used within the same Handler.ProcessTask scope.

Introduce the following method on Task

func (t *Task) Locals() map[string]interface{}

Describe alternatives you've considered
#313 : However, I'm worried that this API will confuse some users. Additionally, the reason net/http package embedded the context within the Request struct is to avoid breaking changes in their API.

Additional context
None for now.

Originally created by @hibiken on GitHub (Sep 14, 2021). Original GitHub issue: https://github.com/hibiken/asynq/issues/326 Originally assigned to: @hibiken on GitHub. **Is your feature request related to a problem? Please describe.** In some cases, we have middleware functions that want to add more context or decorate task so that downstream handler functions can retrieve that data. Similar issues: #285 #313 **Describe the solution you'd like** To allow users to store local data within a handling of task, we can provide `map[string]interface{}` as "task locals". Task locals is not persisted so any modification to the map doesn't persist across retries or locals set before enqueue is not visible to the handler code. This local should be used within the same `Handler.ProcessTask` scope. Introduce the following method on `Task` ```go func (t *Task) Locals() map[string]interface{} ``` **Describe alternatives you've considered** #313 : However, I'm worried that this API will confuse some users. Additionally, the reason `net/http` package embedded the context within the `Request` struct is to avoid breaking changes in their API. **Additional context** None for now.

kerem

2026-03-07 22:06:36 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-07 22:06:37 +03:00

Author

Owner

@crossworth commented on GitHub (Sep 14, 2021):

Hello, I have thought more about #313 and think that the context should be the first parameter as well, I dont think we should "hide" it inside a struct, go docs/context.Context clearly says:

Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. The Context should be the first parameter, typically named ctx

The problem that #313 tries to solve is a way to change the context inside a task handler, the only reason is to allow context propagation (and all the patterns that is has already, like scoped values, tracing).

If we think that the task handler is the last step on the "task processing", this problem would not exist, but in some cases there is a need for an additional step, so a context propagation from the task handler is needed.

I think this PR solves the problem of returning a value from the task to the middleware, but I'm not sure if this solves the context propagation problem.

So I don't know a easy way to allow context propagation.

One pattern that we could make work (I think) is using a custom context, context is already an interface, we could create an TaskContext that implement the context interface and have more functions to it. Users that need the concrete type/functions could do a interface upgrade.

This should be fully backwards compatible, I just don't know if it something that we want to do.

I will close #313 since I dont think is a good solution.

I have give more thought about this PR. I think this a good solution, ofcourse I would like a way to share the context with the middleware back from a handler, but I guess there is ways we can achieve if we really need (on the consumer side of the library, like a middleware that use a custom context, using Locals or a global shared state). I would like to know how would the context be shared/handled when using pipelines/workflows.

If the workflow implementation allow some support of context sharing or job batching it would be cool to allow to share the same context or provide a workflow context with the task context.

@crossworth commented on GitHub (Sep 14, 2021): Hello, I have thought more about #313 and think that the context should be the first parameter as well, I dont think we should "hide" it inside a struct, go docs/context.Context clearly says: > Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. The Context should be the first parameter, typically named ctx The problem that #313 tries to solve is a way to change the context inside a task handler, the only reason is to allow context propagation (and all the patterns that is has already, like scoped values, tracing). If we think that the task handler is the last step on the "task processing", this problem would not exist, but in some cases there is a need for an additional step, so a context propagation from the task handler is needed. I think this PR solves the problem of returning a value from the task to the middleware, but I'm not sure if this solves the context propagation problem. So I don't know a easy way to allow context propagation. One pattern that we could make work (I think) is using a custom context, context is already an interface, we could create an `TaskContext` that implement the context interface and have more functions to it. Users that need the concrete type/functions could do a interface upgrade. This should be fully backwards compatible, I just don't know if it something that we want to do. I will close #313 since I dont think is a good solution. ------------ I have give more thought about this PR. I think this a good solution, ofcourse I would like a way to share the context with the middleware back from a handler, but I guess there is ways we can achieve if we really need (on the consumer side of the library, like a middleware that use a custom context, using Locals or a global shared state). I would like to know how would the context be shared/handled when using pipelines/workflows. If the workflow implementation allow some support of context sharing or job batching it would be cool to allow to share the same context or provide a workflow context with the task context.

kerem commented

2026-03-07 22:06:38 +03:00

Author

Owner

@dhh93 commented on GitHub (Sep 15, 2021):

@hibiken +1 to that implementation.

I've tried implementing message passing/data sharing in middleware functions -- either by mutating the task payload directly or by wrapping handlers in a custom struct and passing data thru a field. The problem with the latter approach is that it is best meant for sharing global data as the data would be available to all tasks processed by that handler. The new method would best address cases in which you'd want to share one-off data with a specific task.

My only (minor) feedback if it could be named Metadata instead of Locals for more clarity 😅 .

@dhh93 commented on GitHub (Sep 15, 2021): @hibiken +1 to that implementation. I've tried implementing message passing/data sharing in middleware functions -- either by mutating the task payload directly or by wrapping handlers in a custom struct and passing data thru a field. The problem with the latter approach is that it is best meant for sharing global data as the data would be available to all tasks processed by that handler. The new method would best address cases in which you'd want to share one-off data with a specific task. My only (minor) feedback if it could be named `Metadata` instead of `Locals` for more clarity 😅 .

kerem commented

2026-03-07 22:06:38 +03:00

Author

Owner

@hibiken commented on GitHub (Jan 16, 2022):

Closing this issue for now since I don't think this is a common need.

@hibiken commented on GitHub (Jan 16, 2022): Closing this issue for now since I don't think this is a common need.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#1156

No description provided.

Rows
Columns