[GH-ISSUE #77] Other serialization format

kerem commented

2026-03-07 22:04:15 +03:00

Owner

Originally created by @hibiken on GitHub (Feb 9, 2020).
Original GitHub issue: https://github.com/hibiken/asynq/issues/77

Originally assigned to: @hibiken on GitHub.

Experiment with other serialization format and benchmark performance and memory usage:

Other serialization format:

msgpack
protobuf

Originally created by @hibiken on GitHub (Feb 9, 2020). Original GitHub issue: https://github.com/hibiken/asynq/issues/77 Originally assigned to: @hibiken on GitHub. Experiment with other serialization format and benchmark performance and memory usage: Other serialization format: - msgpack - protobuf

kerem

2026-03-07 22:04:15 +03:00

closed this issue
added the
idea
label

kerem commented

2026-03-07 22:04:18 +03:00

Author

Owner

@cielu commented on GitHub (Feb 11, 2020):

Hello, hibiken , how to use this lib in go-gin ?

@cielu commented on GitHub (Feb 11, 2020): Hello, hibiken , how to use this lib in go-gin ?

kerem commented

2026-03-07 22:04:18 +03:00

Author

Owner

@cielu commented on GitHub (Feb 11, 2020):

Why I can't kill the process ?

@cielu commented on GitHub (Feb 11, 2020): Why I can't kill the process ?

kerem commented

2026-03-07 22:04:19 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 11, 2020):

@cielu Which OS are you using?
If you are on Windows, maybe something related to https://golang.org/pkg/os/signal/#hdr-Windows.

Please take a look at the Signal Wiki page on how to gracefully shutdown the background workers.

@hibiken commented on GitHub (Feb 11, 2020): @cielu Which OS are you using? If you are on Windows, maybe something related to https://golang.org/pkg/os/signal/#hdr-Windows. Please take a look at [the Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) on how to gracefully shutdown the background workers.

kerem commented

2026-03-07 22:04:19 +03:00

Author

Owner

@cielu commented on GitHub (Feb 11, 2020):

@hibiken macOS, But I start the worker in go-gin . Do you have any good idea ? thanks ..

I'm new at go

@cielu commented on GitHub (Feb 11, 2020): @hibiken macOS, But I start the worker in go-gin . Do you have any good idea ? thanks .. I'm new at go

kerem commented

2026-03-07 22:04:19 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 11, 2020):

So you are using github.com/gin-gonic/gin for your web framework, correct?
And are you starting the workers (bg.Run(handler)) from your web server's main?

@hibiken commented on GitHub (Feb 11, 2020): So you are using [github.com/gin-gonic/gin](https://github.com/gin-gonic/gin) for your web framework, correct? And are you starting the workers (`bg.Run(handler)`) from your web server's main?

kerem commented

2026-03-07 22:04:20 +03:00

Author

Owner

@cielu commented on GitHub (Feb 11, 2020):

@hibiken Yes , that's right . Here is my main func code

func main() {
	// Set Mode
	gin.SetMode(setting.AppCfg.AppMode)
	// Init router
	router := routes.InitRouter()
	// run
	router.Run(":" + setting.AppCfg.AppPost)
	// init asynq worker
	asynqWorker.InitAsynqWorker()
	// init asynq client
	asynqClient.InitAsynqClient()
}

@cielu commented on GitHub (Feb 11, 2020): @hibiken Yes , that's right . Here is my main func code ``` func main() { // Set Mode gin.SetMode(setting.AppCfg.AppMode) // Init router router := routes.InitRouter() // run router.Run(":" + setting.AppCfg.AppPost) // init asynq worker asynqWorker.InitAsynqWorker() // init asynq client asynqClient.InitAsynqClient() } ```

kerem commented

2026-03-07 22:04:20 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 11, 2020):

I see.
The intended use case of asynq is that you use Client to schedule tasks to be processed in the background outside of a user request, and have another process(main.go) to process those tasks in the background.
So you will create another main.go apart from your web server.

So for example, you will have something like this in your web server's main file:

package main

import "github.com/gin-gonic/gin"

func main() {
	r := gin.Default()
        client := asynq.NewClient(/* redis conn opt */)
	r.POST("/signup", func(c *gin.Context) {
                t := asynq.NewTask("send_email", map[string]interface{}{"user_id": 42})
		client.Schedule(t, time.Now())
                // render html or json
	})
	r.Run() 
}

and create a separate main.go file for workers:

func main() {
    r := &asynq.RedisClientOpt{
        Addr: "localhost:6379",
    }

    bg := asynq.NewBackground(r, &asynq.Config{
        Concurrency: 10,
    })

    bg.Run(handler)
}

@hibiken commented on GitHub (Feb 11, 2020): I see. The intended use case of `asynq` is that you use `Client` to schedule tasks to be processed in the background outside of a user request, and have another process(main.go) to process those tasks in the background. So you will create another `main.go` apart from your web server. So for example, you will have something like this in your web server's main file: ```go package main import "github.com/gin-gonic/gin" func main() { r := gin.Default() client := asynq.NewClient(/* redis conn opt */) r.POST("/signup", func(c *gin.Context) { t := asynq.NewTask("send_email", map[string]interface{}{"user_id": 42}) client.Schedule(t, time.Now()) // render html or json }) r.Run() } ``` and create a separate `main.go` file for workers: ```go func main() { r := &asynq.RedisClientOpt{ Addr: "localhost:6379", } bg := asynq.NewBackground(r, &asynq.Config{ Concurrency: 10, }) bg.Run(handler) } ```

kerem commented

2026-03-07 22:04:21 +03:00

Author

Owner

@cielu commented on GitHub (Feb 11, 2020):

Yeah . enhh, I got you . But , can i start the worker in one project ? I don't want separate it .That is a lot of trouble

thanks 😊

@cielu commented on GitHub (Feb 11, 2020): Yeah . enhh, I got you . But , can i start the worker in one project ? I don't want separate it .That is a lot of trouble thanks 😊

kerem commented

2026-03-07 22:04:21 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 11, 2020):

I see.

Background.Run method will block and wait for a termination signal (OS signal TERM or INTERRUPT).

I think if you want to start the workers from your web server's main then try this:

func main() {
	// Set Mode
	gin.SetMode(setting.AppCfg.AppMode)
	// Init router
	router := routes.InitRouter()

        bgDone := make(chan, struct{})

       go func() {
           bg := asynq.NewBackground(r, &asynq.Config{/* options */})
           bg.Run(handler)
           bgDone <- struct{}{}
       }()
	// run
	router.Run(":" + setting.AppCfg.AppPost)
        <-bgDone
}

If that doesn't work, then another option is to fork this repo and export Background.Start and Background.Stop methods (currently they are unexported) and call those to start and stop the background workers, those methods are non-blocking.

@hibiken commented on GitHub (Feb 11, 2020): I see. `Background.Run` method will block and wait for a termination signal (OS signal TERM or INTERRUPT). I think if you want to start the workers from your web server's main then try this: ```go func main() { // Set Mode gin.SetMode(setting.AppCfg.AppMode) // Init router router := routes.InitRouter() bgDone := make(chan, struct{}) go func() { bg := asynq.NewBackground(r, &asynq.Config{/* options */}) bg.Run(handler) bgDone <- struct{}{} }() // run router.Run(":" + setting.AppCfg.AppPost) <-bgDone } ``` If that doesn't work, then another option is to fork this repo and export `Background.Start` and `Background.Stop` methods (currently they are unexported) and call those to start and stop the background workers, those methods are non-blocking.

kerem commented

2026-03-07 22:04:21 +03:00

Author

Owner

@cielu commented on GitHub (Feb 11, 2020):

Ohh. Thank you very much !

@cielu commented on GitHub (Feb 11, 2020): Ohh. Thank you very much !

kerem commented

2026-03-07 22:04:22 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 22, 2020):