[GH-ISSUE #179] Additional Pipelines #176

New issue

Closed

opened 2026-02-27 15:55:29 +03:00 by kerem · 2 comments

kerem commented

2026-02-27 15:55:29 +03:00

Owner

Originally created by @TomKirby on GitHub (Aug 14, 2018).
Original GitHub issue: https://github.com/RD17/ambar/issues/179

Hi,

Looking at the docker compose file I can see that the following code around creating a new pipeline container seems to be structured in a way which will allow multipul pipelines to be created and distrubutue the load?

pipeline0:
    depends_on: 
      serviceapi: 
        condition: service_healthy                            
    image: ambar/ambar-pipeline:latest
    restart: always
    networks:
      - internal_network  
    environment:                
      - id=0
      - api_url=http://serviceapi:8081
- rabbit_host=amqp://rabbit

we are currently experiencing problems with large backlog within RabbitMQ of around 50,000 and it appears that the pipline is our bottleneck, could i create additional instances of the pipeline by duplicating the following code 2-3 times and chaging the name of the instance from pipeline0: to pipeline1:

and also the id under the enviroment section by +1?

Regards,

Tom.

Originally created by @TomKirby on GitHub (Aug 14, 2018). Original GitHub issue: https://github.com/RD17/ambar/issues/179 Hi, Looking at the docker compose file I can see that the following code around creating a new pipeline container seems to be structured in a way which will allow multipul pipelines to be created and distrubutue the load? ``` pipeline0: depends_on: serviceapi: condition: service_healthy image: ambar/ambar-pipeline:latest restart: always networks: - internal_network environment: - id=0 - api_url=http://serviceapi:8081 - rabbit_host=amqp://rabbit ``` we are currently experiencing problems with large backlog within RabbitMQ of around 50,000 and it appears that the pipline is our bottleneck, could i create additional instances of the pipeline by duplicating the following code 2-3 times and chaging the name of the instance from ``pipeline0:`` to ``pipeline1:`` and also the id under the enviroment section by +1? Regards, Tom.

kerem closed this issue

2026-02-27 15:55:29 +03:00

kerem commented

2026-02-27 15:55:30 +03:00

Author

Owner

@sochix commented on GitHub (Aug 14, 2018):

Hi Tom,
yep you can!
Butkeep in mind, that lot of pipelines can overload webapi or es.

@sochix commented on GitHub (Aug 14, 2018): Hi Tom, yep you can! Butkeep in mind, that lot of pipelines can overload webapi or es.

kerem commented

2026-02-27 15:55:30 +03:00

Author

Owner

@TomKirby commented on GitHub (Aug 16, 2018):

This worked, the Issue can be closed 👍

@TomKirby commented on GitHub (Aug 16, 2018): This worked, the Issue can be closed :+1: