[GH-ISSUE #109] [Proposal] Change Payload to bytes & Add Wrapper around Key-Value pair payload

kerem commented

2026-03-07 22:04:39 +03:00

Owner

Originally created by @hibiken on GitHub (Mar 11, 2020).
Original GitHub issue: https://github.com/hibiken/asynq/issues/109

Originally assigned to: @hibiken on GitHub.

Is your feature request related to a problem? Please describe.
I'm always frustrated when I need to call GetX method for all the values in Payload to get all the data (also need to make sure that I don't make a typo in payload key string).
It'd be nice if asynq supported en/decoding protobuf or gob message directly so that I can load that data to an object with one method call.

Describe the solution you'd like
initial proposal:

type EmailTaskPayload struct {
    UserID int
}

// client side.
func main() {
     c := asynq.NewClient(asynq.RedisClientOpt{Addr: ":6379"})

    t := asynq.NewTask("email:welcome", EmailTaskPayload{UserID: 42})
    err := c.Enqueue(t)
    if err != nil {
        log.Fatalf("could not enqueue task: %v", err)
    }
}

// background workers side
func main() {
    bg := asynq.NewBackground(r, &asynq.Config{
        Concurrency: 10,
    })

    mux := asynq.NewMux()
    mux.HandleFunc("email:welcome", emailHandler)

    bg.Run(mux)
}

func emailHandler(ctx context.Context, t *asynq.Task) error {
    var p EmailTaskPayload
    err := t.ReadPayload(&p)
    if err != nil {
        return err
    }
    fmt.Printf("Send Email to User %d\n", p.UserID)
    return nil
}

Describe alternatives you've considered
None for now.

Additional context
None for now.

Originally created by @hibiken on GitHub (Mar 11, 2020). Original GitHub issue: https://github.com/hibiken/asynq/issues/109 Originally assigned to: @hibiken on GitHub. **Is your feature request related to a problem? Please describe.** I'm always frustrated when I need to call `GetX` method for all the values in Payload to get all the data (also need to make sure that I don't make a typo in payload key string). It'd be nice if `asynq` supported en/decoding protobuf or gob message directly so that I can load that data to an object with one method call. **Describe the solution you'd like** initial proposal: ```go type EmailTaskPayload struct { UserID int } // client side. func main() { c := asynq.NewClient(asynq.RedisClientOpt{Addr: ":6379"}) t := asynq.NewTask("email:welcome", EmailTaskPayload{UserID: 42}) err := c.Enqueue(t) if err != nil { log.Fatalf("could not enqueue task: %v", err) } } // background workers side func main() { bg := asynq.NewBackground(r, &asynq.Config{ Concurrency: 10, }) mux := asynq.NewMux() mux.HandleFunc("email:welcome", emailHandler) bg.Run(mux) } func emailHandler(ctx context.Context, t *asynq.Task) error { var p EmailTaskPayload err := t.ReadPayload(&p) if err != nil { return err } fmt.Printf("Send Email to User %d\n", p.UserID) return nil } ``` **Describe alternatives you've considered** None for now. **Additional context** None for now.

kerem

2026-03-07 22:04:39 +03:00

closed this issue
added the
enhancement

idea
labels

kerem commented

2026-03-07 22:04:41 +03:00

Author

Owner

@hibiken commented on GitHub (Mar 13, 2020):

Closing this for now.

@hibiken commented on GitHub (Mar 13, 2020): Closing this for now.

kerem commented

2026-03-07 22:04:41 +03:00

Author

Owner

@andygrunwald commented on GitHub (Feb 22, 2021):

Hey @hibiken,
coming back to this, because I am running the same problem here.
Would love to store []byte in the message itself.

What was your reasoning for closing this?

@andygrunwald commented on GitHub (Feb 22, 2021): Hey @hibiken, coming back to this, because I am running the same problem here. Would love to store []byte in the message itself. What was your reasoning for closing this?

kerem commented

2026-03-07 22:04:42 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 22, 2021):

Hey @andygrunwald thanks for your comment!
I'm actually considering changing the signature of NewTask to

func NewTask(typename string, payload []byte) *Task

and add a nice wrapper around the payload to make it easier to do key-value pairs like the current API supports.

something like:

t := NewTask("mytask", asynq.KV(map[string]interface{}{"mykey": 123}))

I'll open this issue again so that it's more visible to people.

In the meantime, is it possible to workaround this by storing the bytes as string in the payload?

@hibiken commented on GitHub (Feb 22, 2021): Hey @andygrunwald thanks for your comment! I'm actually considering changing the signature of `NewTask` to ```go func NewTask(typename string, payload []byte) *Task ``` and add a nice wrapper around the payload to make it easier to do key-value pairs like the current API supports. something like: ```go t := NewTask("mytask", asynq.KV(map[string]interface{}{"mykey": 123})) ``` I'll open this issue again so that it's more visible to people. In the meantime, is it possible to workaround this by storing the bytes as string in the payload?

kerem commented

2026-03-07 22:04:42 +03:00

Author

Owner

@andygrunwald commented on GitHub (Feb 22, 2021):

Thanks @hibiken.
I thought about something similar.

In the meantime, is it possible to workaround this by storing the bytes as string in the payload?

Didn't check this yet. Need to do another test.
However, I am also running into this issue:

I think the []byte payload is able to solve both.

@andygrunwald commented on GitHub (Feb 22, 2021): Thanks @hibiken. I thought about something similar. > In the meantime, is it possible to workaround this by storing the bytes as string in the payload? Didn't check this yet. Need to do another test. However, I am also running into this issue: <img width="977" alt="Screen Shot 2021-02-22 at 19 38 05" src="https://user-images.githubusercontent.com/320064/108753934-9d77cf00-7545-11eb-8851-890a043e9d29.png"> I think the []byte payload is able to solve both.

kerem commented

2026-03-07 22:04:42 +03:00

Author

Owner

@andygrunwald commented on GitHub (Feb 23, 2021):

@andygrunwald commented on GitHub (Feb 23, 2021): Related to https://github.com/hibiken/asynq/issues/245

kerem commented

2026-03-07 22:04:43 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 23, 2021):

@andygrunwald Would you mind describing the issue you're seeing?
I want to understand your use case so that I can take that into account when I make API change to support storing payload as []byte.

@hibiken commented on GitHub (Feb 23, 2021): @andygrunwald Would you mind describing the issue you're seeing? I want to understand your use case so that I can take that into account when I make API change to support storing payload as `[]byte`.

kerem commented

2026-03-07 22:04:43 +03:00

Author

Owner

@disc commented on GitHub (Mar 11, 2021):

Hello @hibiken

I have the same issue when consumer tried to execute task with this payload:

WARN: Could not remove task id=f83cca71-620b-45ff-a309-c6917dd2f248 type="app.event" from "asynq:{app.event}:active" err: NOT FOUND; Will retry syncing

Payload:

{"eventType": "Signup", "countryCode": "au"}

The full message is:

{"Type":"app.event","Payload":{"countryCode":"au","eventType":"Signup"},"ID":"5663097f-6cb7-4464-b1d2-b6a8c5a59bb7","Queue":"app.event","Retry":25,"Retried":0,"ErrorMsg":"","Timeout":1800,"Deadline":0,"UniqueKey":""}

If I change key from eventType to abc in the payload everything is ok and no errors occur.

{"abc": "Signup", "countryCode": "au"}

Is there any way to fix it?

@disc commented on GitHub (Mar 11, 2021): Hello @hibiken I have the same issue when consumer tried to execute task with this payload: ``` WARN: Could not remove task id=f83cca71-620b-45ff-a309-c6917dd2f248 type="app.event" from "asynq:{app.event}:active" err: NOT FOUND; Will retry syncing ``` Payload: ``` {"eventType": "Signup", "countryCode": "au"} ``` The full message is: ``` {"Type":"app.event","Payload":{"countryCode":"au","eventType":"Signup"},"ID":"5663097f-6cb7-4464-b1d2-b6a8c5a59bb7","Queue":"app.event","Retry":25,"Retried":0,"ErrorMsg":"","Timeout":1800,"Deadline":0,"UniqueKey":""} ``` If I change key from **eventType** to **abc** in the payload everything is ok and no errors occur. ``` {"abc": "Signup", "countryCode": "au"} ``` Is there any way to fix it?

kerem commented

2026-03-07 22:04:43 +03:00

Author

Owner

@disc commented on GitHub (Mar 11, 2021):

^^ UPD:

I found the difference, the keys of object sorted in different way, looks like expected payload sorted alphabetically.

Payload in redis queue asynq:{app.event}:active

"Payload":{"eventType":"Signup","countryCode":"au"}"

Payload in consumer

"Payload":{"countryCode":"au","eventType":"Signup"}"

The problem related to php producer, json_encode method does not work as json.Marshal in golang

@disc commented on GitHub (Mar 11, 2021): ^^ **UPD**: I found the difference, the keys of object sorted in different way, looks like expected payload sorted alphabetically. Payload in redis queue `asynq:{app.event}:active` ``` "Payload":{"eventType":"Signup","countryCode":"au"}" ``` Payload in consumer ``` "Payload":{"countryCode":"au","eventType":"Signup"}" ``` The problem related to php producer, [json_encode](https://www.php.net/json_encode) method does not work as json.Marshal in golang

kerem commented

2026-03-07 22:04:44 +03:00

Author

Owner

@hibiken commented on GitHub (Mar 11, 2021):

@disc Thanks for updating this issue.
Just to clarify, you're enqueueing a task using PHP client? Not using asynq.Client?

FYI: This issue should go away in the next release. I'm currently working on redis key re-design so that we only need to move IDs around and don't need to use JSON data to remove entry from redis sets.

@hibiken commented on GitHub (Mar 11, 2021): @disc Thanks for updating this issue. Just to clarify, you're enqueueing a task using PHP client? Not using `asynq.Client`? FYI: This issue should go away in the next release. I'm currently working on redis key re-design so that we only need to move IDs around and don't need to use JSON data to remove entry from redis sets.

kerem commented

2026-03-07 22:04:44 +03:00

Author

Owner

@disc commented on GitHub (Mar 11, 2021):

@disc Thanks for updating this issue.
Just to clarify, you're enqueueing a task using PHP client? Not using asynq.Client?

Yes, I wrote PHP client for asynq, it just produces a message to the redis queue as the asynq worker expects

@disc commented on GitHub (Mar 11, 2021): > @disc Thanks for updating this issue. > Just to clarify, you're enqueueing a task using PHP client? Not using `asynq.Client`? Yes, I wrote PHP client for asynq, it just produces a message to the redis queue as the asynq worker expects

kerem commented

2026-03-07 22:04:44 +03:00

Author

Owner

@hibiken commented on GitHub (Mar 11, 2021):

That's great! I should probably document the message schema in Wiki so that people can write clients in other languages :)

Related: #105

@hibiken commented on GitHub (Mar 11, 2021): That's great! I should probably document the message schema in Wiki so that people can write clients in other languages :) Related: #105

kerem referenced this issue

2026-03-07 22:10:49 +03:00

[GH-ISSUE #1048] Release: v0.26.0 #1511

kerem referenced this issue

2026-03-07 22:14:19 +03:00

[PR #1084] [MERGED] Prepare release (docs): v0.26.0 #2021

kerem referenced this issue

2026-03-15 20:46:41 +03:00

[GH-ISSUE #1048] Release: v0.26.0 #2523

kerem referenced this issue

2026-03-15 21:16:31 +03:00

[PR #1084] [MERGED] Prepare release (docs): v0.26.0 #3031