[GH-ISSUE #393] [FEATURE REQUEST] Add message headers

kerem commented

2026-03-07 22:06:57 +03:00

Owner

Originally created by @AlexeyBelezeko on GitHub (Feb 1, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/393

Originally assigned to: @hibiken on GitHub.

I'm struggling a bit with passing trace id with the task. Cause there is no way to get it inside middleware without unmarshaling the payload. This adds unnecessary dependencies to middleware code.

I suggest adding map[string]string with headers alongside with payload which lets users pass meta info alongside with payload.

P.S.: Big thank you for your work. Sorry if i missed something in the docs and this actually possible.

Originally created by @AlexeyBelezeko on GitHub (Feb 1, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/393 Originally assigned to: @hibiken on GitHub. I'm struggling a bit with passing trace id with the task. Cause there is no way to get it inside middleware without unmarshaling the payload. This adds unnecessary dependencies to middleware code. I suggest adding `map[string]string` with headers alongside with payload which lets users pass meta info alongside with payload. P.S.: Big thank you for your work. Sorry if i missed something in the docs and this actually possible.

kerem

2026-03-07 22:06:57 +03:00

closed this issue
added the
enhancement

designing
labels

kerem commented

2026-03-07 22:06:58 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 1, 2022):

@AlexeyBelezeko Thank you for opening a feature request!

Is it possible to use context values?

@hibiken commented on GitHub (Feb 1, 2022): @AlexeyBelezeko Thank you for opening a feature request! Is it possible to use [context values](https://pkg.go.dev/context#WithValue)?

kerem commented

2026-03-07 22:06:58 +03:00

Author

Owner

@AlexeyBelezeko commented on GitHub (Feb 2, 2022):

Unfortunately not. I inject trace_id and other meta info into context to use it inside Handlers but it didn't save me from exposing the inner structure of concrete handlers to middleware which I'm wanna avoid.

@AlexeyBelezeko commented on GitHub (Feb 2, 2022): Unfortunately not. I inject trace_id and other meta info into context to use it inside Handlers but it didn't save me from exposing the inner structure of concrete handlers to middleware which I'm wanna avoid.

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 3, 2022):

Hmm I see. Maybe similar to this feature request #326?

Could you give me a more concrete example of what you're trying to achieve? Code snippet would be very helpful here :)

@hibiken commented on GitHub (Feb 3, 2022): Hmm I see. Maybe similar to this feature request #326? Could you give me a more concrete example of what you're trying to achieve? Code snippet would be very helpful here :)

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@AlexeyBelezeko commented on GitHub (Feb 4, 2022):

In our project we run asynq server and client in two different processes. And to pass traceID i was forced to write middleware like this:

type TaskType1 struct {
	TraceID   string
	OtherData []byte
	URL       string
}

type TaskType2 struct {
	TraceID     string
	Name        string
	FullAddress string
	URL         string
}

func TraceInjectorMiddleware(h asynq.Handler) asynq.Handler {
	return asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error {
		var traceID string
		switch t.Type() {
		case "type1":
			task := TaskType1{}
			_ = json.Unmarshal(t.Payload(), &task)
			traceID = task.TraceID
		case "type2":
			task := TaskType2{}
			_ = json.Unmarshal(t.Payload(), &task)
			traceID = task.TraceID
		}
		return h.ProcessTask(context.WithValue(ctx, "traceID", traceID), t)
	})
}

I suggest adding headers:

// Task represents a unit of work to be performed.
type Task struct {
	// typename indicates the type of task to be performed.
	typename string

	// payload holds data needed to perform the task.
	payload []byte

	headers map[string]string

	// opts holds options for the task.
	opts []Option

	// w is the ResultWriter for the task.
	w *ResultWriter
}

func (t *Task) Headers() map[string]string { return t.headers }

And this change will help to simplify middleware like this:

func TraceInjectorMiddleware(h asynq.Handler) asynq.Handler {
	return asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error {
		traceID, ok := t.Headers()["traceID"]
		if ok {
			ctx = context.WithValue(ctx, "traceID", traceID)
		}
		return h.ProcessTask(ctx, t)
	})
}

@AlexeyBelezeko commented on GitHub (Feb 4, 2022): In our project we run asynq server and client in two different processes. And to pass traceID i was forced to write middleware like this: ``` type TaskType1 struct { TraceID string OtherData []byte URL string } type TaskType2 struct { TraceID string Name string FullAddress string URL string } func TraceInjectorMiddleware(h asynq.Handler) asynq.Handler { return asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error { var traceID string switch t.Type() { case "type1": task := TaskType1{} _ = json.Unmarshal(t.Payload(), &task) traceID = task.TraceID case "type2": task := TaskType2{} _ = json.Unmarshal(t.Payload(), &task) traceID = task.TraceID } return h.ProcessTask(context.WithValue(ctx, "traceID", traceID), t) }) } ``` I suggest adding headers: ``` // Task represents a unit of work to be performed. type Task struct { // typename indicates the type of task to be performed. typename string // payload holds data needed to perform the task. payload []byte headers map[string]string // opts holds options for the task. opts []Option // w is the ResultWriter for the task. w *ResultWriter } func (t *Task) Headers() map[string]string { return t.headers } ``` And this change will help to simplify middleware like this: ``` func TraceInjectorMiddleware(h asynq.Handler) asynq.Handler { return asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error { traceID, ok := t.Headers()["traceID"] if ok { ctx = context.WithValue(ctx, "traceID", traceID) } return h.ProcessTask(ctx, t) }) } ```

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@crossworth commented on GitHub (Feb 4, 2022):

Would a partial unmarshal work here?
Since you dont need the others fields, just unmarshal on a struct with the fields that you need and are common to the tasks.

type TaskType1 struct {
	TraceID   string
	OtherData []byte
	URL       string
}

type TaskType2 struct {
	TraceID     string
	Name        string
	FullAddress string
	URL         string
}

type TaskWithTraceID struct {
	TraceID string
}

func TraceInjectorMiddleware(h asynq.Handler) asynq.Handler {
	return asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error {
		var traceID string
		task := TaskWithTraceID{}
		_ = json.Unmarshal(t.Payload(), &task)
		traceID = task.TraceID
		return h.ProcessTask(context.WithValue(ctx, "traceID", traceID), t)
	})
}

Or you could access the payload using another library for json handling like: https://github.com/tidwall/gjson

@crossworth commented on GitHub (Feb 4, 2022): Would a partial unmarshal work here? Since you dont need the others fields, just unmarshal on a struct with the fields that you need and are common to the tasks. ```go type TaskType1 struct { TraceID string OtherData []byte URL string } type TaskType2 struct { TraceID string Name string FullAddress string URL string } type TaskWithTraceID struct { TraceID string } func TraceInjectorMiddleware(h asynq.Handler) asynq.Handler { return asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error { var traceID string task := TaskWithTraceID{} _ = json.Unmarshal(t.Payload(), &task) traceID = task.TraceID return h.ProcessTask(context.WithValue(ctx, "traceID", traceID), t) }) } ``` Or you could access the payload using another library for json handling like: https://github.com/tidwall/gjson

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@AlexeyBelezeko commented on GitHub (Feb 4, 2022):

@crossworth thank you for your suggestion. But I also want to exclude this info from the payload to not affect uniqueness of the task.
For example, we have a few places in code which can add tasks to the same queue. Those tasks will have different trace ID and different payloads. To avoid duplications of work I have to use the TaskID. This will work perfectly with your solution. But I believe that meta info in headers more explicit and more simple solution.

@AlexeyBelezeko commented on GitHub (Feb 4, 2022): @crossworth thank you for your suggestion. But I also want to exclude this info from the payload to not affect uniqueness of the task. For example, we have a few places in code which can add tasks to the same queue. Those tasks will have different trace ID and different payloads. To avoid duplications of work I have to use the TaskID. This will work perfectly with your solution. But I believe that meta info in headers more explicit and more simple solution.

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 9, 2022):

Sorry for the delay in response here!

Let me think about this for a bit more since once a feature is added, it's hard to remove if we decide otherwise later.

@hibiken commented on GitHub (Feb 9, 2022): Sorry for the delay in response here! Let me think about this for a bit more since once a feature is added, it's hard to remove if we decide otherwise later.

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@zhangzitao commented on GitHub (Oct 8, 2022):

@hibiken Any news? I long for this feature. Thanks for your work

@zhangzitao commented on GitHub (Oct 8, 2022): @hibiken Any news? I long for this feature. Thanks for your work

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@rapita commented on GitHub (May 10, 2023):

@hibiken Any news? If you accept this feature, I can continue to implement it in the PR https://github.com/hibiken/asynq/pull/606

@rapita commented on GitHub (May 10, 2023): @hibiken Any news? If you accept this feature, I can continue to implement it in the PR https://github.com/hibiken/asynq/pull/606

kerem commented

2026-03-07 22:07:00 +03:00

Author

Owner

@dennypenta commented on GitHub (May 10, 2023):

pls delegate the project owning to active contributors

@dennypenta commented on GitHub (May 10, 2023): pls delegate the project owning to active contributors

kerem commented

2026-03-07 22:07:00 +03:00

Author

Owner

@agorman commented on GitHub (Apr 26, 2024):

+1. I could really use this as well.

@agorman commented on GitHub (Apr 26, 2024): +1. I could really use this as well.

kerem commented

2026-03-07 22:07:00 +03:00

Author

Owner

@dennypenta commented on GitHub (May 29, 2024):

share a link pls if anyone forked it

@dennypenta commented on GitHub (May 29, 2024): share a link pls if anyone forked it

Rows
Columns

[GH-ISSUE #393] [FEATURE REQUEST] Add message headers #1187