[GH-ISSUE #244] [FEATURE REQUEST] Support Workflow

kerem commented

2026-03-15 19:13:10 +03:00

Owner

Originally created by @hibiken on GitHub (Feb 15, 2021).
Original GitHub issue: https://github.com/hibiken/asynq/issues/244

Originally assigned to: @hibiken on GitHub.

Is your feature request related to a problem? Please describe.
We'd like to add support for creating a workflow composed of multiple tasks.
A workflow should be a directed-acyclic graph and can be grouped into multiple phases (or stages).
A good example is Github actions, where a user can compose a workflow from multiple tasks (aka jobs).

Describe the solution you'd like
A workflow should be treated as a unit. If a failure happens during a workflow execution, the entire workflow should be retried.
Web UI should support inspecting and visualizing a workflow.

Additional context
Similar feature request #218

Originally created by @hibiken on GitHub (Feb 15, 2021). Original GitHub issue: https://github.com/hibiken/asynq/issues/244 Originally assigned to: @hibiken on GitHub. **Is your feature request related to a problem? Please describe.** We'd like to add support for creating a workflow composed of multiple tasks. A workflow should be a directed-acyclic graph and can be grouped into multiple phases (or stages). A good example is Github actions, where a user can compose a workflow from multiple tasks (aka jobs). **Describe the solution you'd like** A workflow should be treated as a unit. If a failure happens during a workflow execution, the entire workflow should be retried. Web UI should support inspecting and visualizing a workflow. **Additional context** Similar feature request #218

kerem added the

idea

designing

labels

2026-03-15 19:13:10 +03:00

kerem commented

2026-03-15 19:13:21 +03:00

Author

Owner

@beyoung commented on GitHub (Jul 16, 2021):

I think airflow is a good example.

@beyoung commented on GitHub (Jul 16, 2021): I think [airflow](https://airflow.apache.org/) is a good example.

kerem commented

2026-03-15 19:13:27 +03:00

Author

Owner

@aon108 commented on GitHub (Nov 9, 2021):

temporal.io is the best example

@aon108 commented on GitHub (Nov 9, 2021): temporal.io is the best example

kerem commented

2026-03-15 19:13:32 +03:00

Author

Owner

@aon108 commented on GitHub (Nov 11, 2021):

Just contributing some thoughts to this workflow idea, as I'm rolling my own implementation of this at the moment.
The simplest implementation I had was just specifying the next task that had to be scheduled inside the payload struct.
So when the worker was finished doing task X, it knew to schedule task Y. The treating workflow as a unit comes with a lot of overhead to get right - as can be seen in the temporal.io infrastructure. I like asynq as it is a versatile, to be used in many ways without bloating and complexity.

@aon108 commented on GitHub (Nov 11, 2021): Just contributing some thoughts to this workflow idea, as I'm rolling my own implementation of this at the moment. The simplest implementation I had was just specifying the next task that had to be scheduled inside the payload struct. So when the worker was finished doing task X, it knew to schedule task Y. The `treating workflow as a unit` comes with a lot of overhead to get right - as can be seen in the temporal.io infrastructure. I like asynq as it is a versatile, to be used in many ways without bloating and complexity.

kerem commented

2026-03-15 19:13:37 +03:00

Author

Owner

@hibiken commented on GitHub (Nov 12, 2021):

@aon108 thanks for your input. I currently have a rough idea on how I want to go about implementing this. I agree with you that we want to keep the simplicity of the library and my hope is that this workflow feature is a layer on top of what we currently have so that it doesn't change the core of the library :)

@hibiken commented on GitHub (Nov 12, 2021): @aon108 thanks for your input. I currently have a rough idea on how I want to go about implementing this. I agree with you that we want to keep the simplicity of the library and my hope is that this workflow feature is a layer on top of what we currently have so that it doesn't change the core of the library :)

kerem commented

2026-03-15 19:13:42 +03:00

Author

Owner

@aon108 commented on GitHub (May 19, 2022):

Temporal does dehydration of a failed workflow . That means whatever part of a workflow fails , it can pickup where it left off.

Async does not have this feature , but let me know if I am incorrect. But I think all async needs to do is let you know it failed and for your own data layer to be idempotent so that repeats of parts of the workflow are fine anyway .

Yes agreed with the above. My approach to this has been to persist a state machine in PG and then have each async task check the state and execute the next corresponding task in workflow. if the entire queue fails the workflow is consider failed. first thing the asynq server checks on startup is if the queue is empty and there are workflows that are not complete then schedules the correct tasks to continue processing. Asynq has the great feature of unique tasks, so that even if multiple servers check the state on startup only 1 unique task per workflow is started.

@aon108 commented on GitHub (May 19, 2022): > Temporal does dehydration of a failed workflow . That means whatever part of a workflow fails , it can pickup where it left off. > > Async does not have this feature , but let me know if I am incorrect. But I think all async needs to do is let you know it failed and for your own data layer to be idempotent so that repeats of parts of the workflow are fine anyway . Yes agreed with the above. My approach to this has been to persist a state machine in PG and then have each async task check the state and execute the next corresponding task in workflow. if the entire queue fails the workflow is consider failed. first thing the asynq server checks on startup is if the queue is empty and there are workflows that are not complete then schedules the correct tasks to continue processing. Asynq has the great feature of unique tasks, so that even if multiple servers check the state on startup only 1 unique task per workflow is started.

kerem commented

2026-03-15 19:13:47 +03:00

Author

Owner

@aon108 commented on GitHub (May 19, 2022):

Ok that makes sense , and nice and simple - I like simple.
In a workflow the retry then does not happen and you do that on purpose to avoid the side effects that could occur ?

If i understand you correctly you mean that the entire worfklow does not get rerun from start on a failure and recovery.

If so then yes to avoid side effects. In my particular use case some of the workflow steps are quite computational intense and take time. so rather than waste resources by doing the work again even if it is idempotent I chose to skip and just continue from the latest step.

@aon108 commented on GitHub (May 19, 2022): > Ok that makes sense , and nice and simple - I like simple. > In a workflow the retry then does not happen and you do that on purpose to avoid the side effects that could occur ? If i understand you correctly you mean that the entire worfklow does not get rerun from start on a failure and recovery. If so then yes to avoid side effects. In my particular use case some of the workflow steps are quite computational intense and take time. so rather than waste resources by doing the work again even if it is idempotent I chose to skip and just continue from the latest step.

kerem commented

2026-03-15 19:13:52 +03:00

Author

Owner

@s8sg commented on GitHub (Apr 21, 2023):

@hibiken I tried to build something similar with https://github.com/s8sg/goflow. This library keeps the execution state in redis . It also support complex branching and looping strategy. Lot of the components/ideas from this project can be reused

@s8sg commented on GitHub (Apr 21, 2023): @hibiken I tried to build something similar with https://github.com/s8sg/goflow. This library keeps the execution state in redis . It also support complex branching and looping strategy. Lot of the components/ideas from this project can be reused

kerem commented

2026-03-15 19:13:57 +03:00

Author

Owner

@yuyang0 commented on GitHub (Apr 8, 2024):

@hibiken I developed a dag workflow library based on asynq, It's simple and easy to use. https://github.com/yuyang0/dagflow

@yuyang0 commented on GitHub (Apr 8, 2024): @hibiken I developed a dag workflow library based on asynq, It's simple and easy to use. https://github.com/yuyang0/dagflow

Rows
Columns

[GH-ISSUE #244] [FEATURE REQUEST] Support Workflow #2115