starred/ambar
1
0
Fork 0
mirror of https://github.com/RD17/ambar.git synced 2026-04-25 15:35:49 +03:00
Code Issues 2 Projects Releases 3 Packages Wiki Activity

[GH-ISSUE #284] How to Build Docker Containers #270

New issue
Closed
opened 2026-02-27 15:55:54 +03:00 by kerem · 11 comments
kerem commented 2026-02-27 15:55:54 +03:00
Owner
Copy link

Originally created by @ark- on GitHub (Jun 22, 2020).
Original GitHub issue: https://github.com/RD17/ambar/issues/284

Hi. I had to work through multiple issues and the source code to get it working in my instance so I would like to pay it forward.

Build

  1. git clone https://github.com/RD17/ambar.git
  2. cd ~/ambar/FrontEnd
  3. If compiling this on a different machine than what you're deploying on then before performing compile export PORT={YOUR_WEB_FRONTEND_PORT) and export FRONTEND_HOST={YOUR_HOST_IP}. I used 80 and 192.168.1.1.
  4. node npm install
  5. npm run compile
  6. docker build . -t frontend
  7. cd ~/ambar/WebApi
  8. docker build . -t webapi
  9. cd ~/ambar/ServiceApi
  10. docker build . -t serviceapi
  11. cd ~/ambar/Redis
  12. docker build . -t redis
  13. cd ~/ambar/Rabbit
  14. docker build . -t rabbit
  15. cd ~/ambar/Pipeline
  16. docker build . -t pipeline0
  17. cd ~/ambar/LocalCrawler
  18. docker build . -t localcrawler
  19. cd ~/ambar/MongoDB
  20. docker build . -t mongodb
  21. cd ~/ambar/ElasticSearch
  22. docker build . -t elasticsearch
  23. Set elasticsearch docker host settings, these are referenced here
sysctl -w vm.max_map_count=262144
sysctl -w net.ipv4.ip_local_port_range="15000 61000"
sysctl -w net.ipv4.tcp_fin_timeout=30
sysctl -w net.core.somaxconn=1024
sysctl -w net.core.netdev_max_backlog=2000
sysctl -w net.ipv4.tcp_max_syn_backlog=2048
sysctl -w vm.overcommit_memory=1

Docker Compose

My final docker compose:

version: "2.1"
networks:
  internal_network:
services:      
  db:
    restart: always
    networks:
      - internal_network
    build: ./MongoDB
    environment:
      - cacheSizeGB=2
    volumes:
      - ~/ambar/mapped_vols/db:/data/db
    expose:
      - "27017"   
  es:
    restart: always
    networks:
      - internal_network
    build: ./ElasticSearch
    expose:
      - "9200" 
    environment:
      - cluster.name=ambar-es
      - ES_JAVA_OPTS=-Xms2g -Xmx2g
      - max_map_count=262144
    ulimits:
      memlock:
        soft: -1
        hard: -1  
      nofile:
        soft: 65536
        hard: 65536
    cap_add:
      - IPC_LOCK
    volumes:
      - ~/ambar/mapped_vols/es:/usr/share/elasticsearch/data
  rabbit:
    restart: always
    networks:
      - internal_network
    build: ./Rabbit
    hostname: rabbit
    expose:
      - "15672"
      - "5672"   
    volumes:
      - ~/ambar/mapped_vols/rabbit:/var/lib/rabbitmq
  redis:
    restart: always
    sysctls:
      - net.core.somaxconn=1024
    networks:
      - internal_network
    build: ./Redis
    expose:
      - "6379" 
  serviceapi:
    depends_on:
      redis:
        condition: service_healthy
      rabbit:
        condition: service_healthy
      es:
        condition: service_healthy
      db:
        condition: service_healthy
    restart: always
    networks:
      - internal_network
    build: ./ServiceApi
    expose:
      - "8081"  
    ports:
      - "8081:8081"
    environment:            
      - mongoDbUrl=mongodb://db:27017/ambar_data
      - elasticSearchUrl=http://es:9200
      - redisHost=redis
      - redisPort=6379
      - rabbitHost=amqp://rabbit
      - langAnalyzer=ambar_en
  webapi:
    depends_on:
      serviceapi:
        condition: service_healthy
    restart: always
    networks:
      - internal_network
    build: ./WebApi
    expose:
      - "8080"
    ports:
      - "8080:8080"
    environment:            
      - uiLang=en
      - mongoDbUrl=mongodb://db:27017/ambar_data
      - elasticSearchUrl=http://es:9200
      - redisHost=redis
      - redisPort=6379
      - serviceApiUrl=http://serviceapi:8081
      - rabbitHost=amqp://rabbit  
  frontend:
    depends_on:
      webapi:
        condition: service_healthy
    build: ./FrontEnd
    restart: always
    networks:
      - internal_network
    ports:
      - "82:80"
    expose:
      - "82"
    environment:
      - api=http://192.168.1.1:8080  
  pipeline0:
    depends_on: 
      serviceapi: 
        condition: service_healthy                            
    build: ./Pipeline
    restart: always
    networks:
      - internal_network  
    environment:                
      - id=0
      - api_url=http://serviceapi:8081
      - rabbit_host=amqp://rabbit
  my-docs:
    depends_on: 
      serviceapi: 
        condition: service_healthy 
    build: ./LocalCrawler
    restart: always
    networks:
      - internal_network
    expose:
      - "8082"
    environment:      
      - name=my-docs
      - apiUrl=http://serviceapi:8081
    volumes:
      - ~/Documents:/usr/data

The differences between my docker-compose.yml and those listed https://ambar.cloud/docs/installation-docker/ are:

  • I performed the modifications suggested on this page.
  • I switched image attributes to build as I have built these locally
  • I added - max_map_count=262144 to elasticsearch environment
  • I changed the external frontend port to 82 as per step 3 above
  • I added - apiUrl=http://serviceapi:8081 to the crawler as it was polling the wrong port without this

Finally, to get it going:

cd ~/ambarr && docker-compose up -d
Originally created by @ark- on GitHub (Jun 22, 2020). Original GitHub issue: https://github.com/RD17/ambar/issues/284 Hi. I had to work through multiple issues and the source code to get it working in my instance so I would like to pay it forward. ## Build 1. `git clone https://github.com/RD17/ambar.git` 2. `cd ~/ambar/FrontEnd` 3. If compiling this on a different machine than what you're deploying on then before performing compile `export PORT={YOUR_WEB_FRONTEND_PORT)` and `export FRONTEND_HOST={YOUR_HOST_IP}`. I used `80` and `192.168.1.1`. 3. `node npm install` 4. `npm run compile` 5. `docker build . -t frontend` 6. `cd ~/ambar/WebApi` 7. `docker build . -t webapi` 8. `cd ~/ambar/ServiceApi` 9. `docker build . -t serviceapi` 10. `cd ~/ambar/Redis` 11. `docker build . -t redis` 12. `cd ~/ambar/Rabbit` 13. `docker build . -t rabbit` 14. `cd ~/ambar/Pipeline` 15. `docker build . -t pipeline0` 16. `cd ~/ambar/LocalCrawler` 17. `docker build . -t localcrawler` 18. `cd ~/ambar/MongoDB` 19. `docker build . -t mongodb` 20. `cd ~/ambar/ElasticSearch` 21. `docker build . -t elasticsearch` 22. Set elasticsearch docker host settings, these are referenced [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144) ``` sysctl -w vm.max_map_count=262144 sysctl -w net.ipv4.ip_local_port_range="15000 61000" sysctl -w net.ipv4.tcp_fin_timeout=30 sysctl -w net.core.somaxconn=1024 sysctl -w net.core.netdev_max_backlog=2000 sysctl -w net.ipv4.tcp_max_syn_backlog=2048 sysctl -w vm.overcommit_memory=1 ``` ## Docker Compose My final docker compose: ``` yml version: "2.1" networks: internal_network: services: db: restart: always networks: - internal_network build: ./MongoDB environment: - cacheSizeGB=2 volumes: - ~/ambar/mapped_vols/db:/data/db expose: - "27017" es: restart: always networks: - internal_network build: ./ElasticSearch expose: - "9200" environment: - cluster.name=ambar-es - ES_JAVA_OPTS=-Xms2g -Xmx2g - max_map_count=262144 ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 cap_add: - IPC_LOCK volumes: - ~/ambar/mapped_vols/es:/usr/share/elasticsearch/data rabbit: restart: always networks: - internal_network build: ./Rabbit hostname: rabbit expose: - "15672" - "5672" volumes: - ~/ambar/mapped_vols/rabbit:/var/lib/rabbitmq redis: restart: always sysctls: - net.core.somaxconn=1024 networks: - internal_network build: ./Redis expose: - "6379" serviceapi: depends_on: redis: condition: service_healthy rabbit: condition: service_healthy es: condition: service_healthy db: condition: service_healthy restart: always networks: - internal_network build: ./ServiceApi expose: - "8081" ports: - "8081:8081" environment: - mongoDbUrl=mongodb://db:27017/ambar_data - elasticSearchUrl=http://es:9200 - redisHost=redis - redisPort=6379 - rabbitHost=amqp://rabbit - langAnalyzer=ambar_en webapi: depends_on: serviceapi: condition: service_healthy restart: always networks: - internal_network build: ./WebApi expose: - "8080" ports: - "8080:8080" environment: - uiLang=en - mongoDbUrl=mongodb://db:27017/ambar_data - elasticSearchUrl=http://es:9200 - redisHost=redis - redisPort=6379 - serviceApiUrl=http://serviceapi:8081 - rabbitHost=amqp://rabbit frontend: depends_on: webapi: condition: service_healthy build: ./FrontEnd restart: always networks: - internal_network ports: - "82:80" expose: - "82" environment: - api=http://192.168.1.1:8080 pipeline0: depends_on: serviceapi: condition: service_healthy build: ./Pipeline restart: always networks: - internal_network environment: - id=0 - api_url=http://serviceapi:8081 - rabbit_host=amqp://rabbit my-docs: depends_on: serviceapi: condition: service_healthy build: ./LocalCrawler restart: always networks: - internal_network expose: - "8082" environment: - name=my-docs - apiUrl=http://serviceapi:8081 volumes: - ~/Documents:/usr/data ``` The differences between my `docker-compose.yml` and those listed [https://ambar.cloud/docs/installation-docker/](https://ambar.cloud/docs/installation-docker/) are: - I performed the modifications suggested on [this page](https://ambar.cloud/docs/installation-docker/). - I switched `image` attributes to `build` as I have built these locally - I added `- max_map_count=262144` to elasticsearch environment - I changed the external `frontend` port to 82 as per step 3 above - I added `- apiUrl=http://serviceapi:8081` to the crawler as it was polling the wrong port without this Finally, to get it going: ``` bash cd ~/ambarr && docker-compose up -d ```
kerem 2026-02-27 15:55:54 +03:00
  • closed this issue
  • added the
    wontfix
         label
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@Sicaine commented on GitHub (Jul 4, 2020):

I hacked together a multi stage dockerfile for the Frontend. With it, you don't need to install anything on your host to build it:

I think it works, but i hacked it together and wanted to share it before i forget about it.

Btw. it doesn't build with nodejs 12 and there are plenty of warnings and deprecations for the used libs. So be careful if you put out the Frontend without protection.

FROM ubuntu:20.04 as build

RUN apt update && apt install -y curl sudo

RUN curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -

RUN apt update && apt install --no-install-recommends --no-install-suggests -y nodejs python2.7 build-essential

WORKDIR /dist

COPY . .
 
RUN npm install && npm run compile



FROM nginx:latest

RUN apt-get update && apt-get install --no-install-recommends --no-install-suggests -y curl

# Set a timezone
ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

COPY default /etc/nginx/conf.d/default.conf
COPY --from=build /dist/dist /usr/share/nginx/html

CMD echo $api > /usr/share/nginx/html/apiUrl.txt && nginx -g "daemon off;"

HEALTHCHECK --interval=5s --timeout=30s --retries=50 \
  CMD curl -f localhost:80 || exit 1
<!-- gh-comment-id:653785669 --> @Sicaine commented on GitHub (Jul 4, 2020): I hacked together a multi stage dockerfile for the Frontend. With it, you don't need to install anything on your host to build it: I think it works, but i hacked it together and wanted to share it before i forget about it. Btw. it doesn't build with nodejs 12 and there are plenty of warnings and deprecations for the used libs. So be careful if you put out the Frontend without protection. ``` FROM ubuntu:20.04 as build RUN apt update && apt install -y curl sudo RUN curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - RUN apt update && apt install --no-install-recommends --no-install-suggests -y nodejs python2.7 build-essential WORKDIR /dist COPY . . RUN npm install && npm run compile FROM nginx:latest RUN apt-get update && apt-get install --no-install-recommends --no-install-suggests -y curl # Set a timezone ENV TZ=UTC RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone COPY default /etc/nginx/conf.d/default.conf COPY --from=build /dist/dist /usr/share/nginx/html CMD echo $api > /usr/share/nginx/html/apiUrl.txt && nginx -g "daemon off;" HEALTHCHECK --interval=5s --timeout=30s --retries=50 \ CMD curl -f localhost:80 || exit 1 ```
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@kitsune242 commented on GitHub (Jul 16, 2020):

THX
i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ?

<!-- gh-comment-id:659627867 --> @kitsune242 commented on GitHub (Jul 16, 2020): THX i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ?
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@ark- commented on GitHub (Jul 17, 2020):

I had this same issue. Take a look at step 3

  1. If compiling this on a different machine than what you're deploying on then before performing compile export PORT={YOUR_WEB_FRONTEND_PORT) and export FRONTEND_HOST={YOUR_HOST_IP}. I used 80 and 192.168.1.1.

If you compiling in a docker on the same machine (as I was) you'll also need to do this. Try setting these variables and see if this helps.

<!-- gh-comment-id:660062649 --> @ark- commented on GitHub (Jul 17, 2020): I had this same issue. Take a look at step 3 > 3. If compiling this on a different machine than what you're deploying on then before performing compile `export PORT={YOUR_WEB_FRONTEND_PORT)` and `export FRONTEND_HOST={YOUR_HOST_IP}`. I used 80 and 192.168.1.1. If you compiling in a docker on the same machine (as I was) you'll also need to do this. Try setting these variables and see if this helps.
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@kitsune242 commented on GitHub (Jul 17, 2020):

Unfortunately, it's still the same.
its look it connect to 172.17.0.2:3000 and cant connect to backend. I use this Export command i rebuild docker but still have this problem.
I may use them incorrectly or something

<!-- gh-comment-id:660099691 --> @kitsune242 commented on GitHub (Jul 17, 2020): Unfortunately, it's still the same. its look it connect to 172.17.0.2:3000 and cant connect to backend. I use this Export command i rebuild docker but still have this problem. I may use them incorrectly or something
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@mlucasdasilva commented on GitHub (Jul 28, 2020):

Instead of:

npm install && npm run compile

Try:

npm install && npm run deploy:prd

It worked for me.
It generated the files in ./FrontEnd/dist without fixed IP and Port.

<!-- gh-comment-id:665272272 --> @mlucasdasilva commented on GitHub (Jul 28, 2020): Instead of: > npm install && npm run compile Try: > npm install && npm run deploy:prd It worked for me. It generated the files in `./FrontEnd/dist` without fixed IP and Port.
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@deyraka commented on GitHub (Jul 30, 2020):

THX
i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ?

I've following the instruction above. Unfortunately, I face the same issue.
there are no problem while docker-compose up. But, when I open {ipAddress}:82 it's only show white blank page with "Loading..." title
What did I miss?

<!-- gh-comment-id:666280391 --> @deyraka commented on GitHub (Jul 30, 2020): > THX > i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ? I've following the instruction above. Unfortunately, I face the same issue. there are no problem while docker-compose up. But, when I open {ipAddress}:82 it's only show white blank page with "Loading..." title What did I miss?
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@ark- commented on GitHub (Jul 30, 2020):

Unfortunately I cannot test as I have decided to stop using Ambar.

This is because 9 containers for document search is too many. Also ElasticSearch image seemed to always be using the CPU. I think Ambar was overkill for my purposes (home document scan search).

<!-- gh-comment-id:666369277 --> @ark- commented on GitHub (Jul 30, 2020): Unfortunately I cannot test as I have decided to stop using Ambar. This is because 9 containers for document search is too many. Also ElasticSearch image seemed to always be using the CPU. I think Ambar was overkill for my purposes (home document scan search).
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@iddqd-dev commented on GitHub (Jul 31, 2020):

Unfortunately I cannot test as I have decided to stop using Ambar.

This is because 9 containers for document search is too many. Also ElasticSearch image seemed to always be using the CPU. I think Ambar was overkill for my purposes (home document scan search).

Of course, windows indexing does a great job of searching through less than a million documents.

<!-- gh-comment-id:667065892 --> @iddqd-dev commented on GitHub (Jul 31, 2020): > Unfortunately I cannot test as I have decided to stop using Ambar. > > This is because 9 containers for document search is too many. Also ElasticSearch image seemed to always be using the CPU. I think Ambar was overkill for my purposes (home document scan search). Of course, windows indexing does a great job of searching through less than a million documents.
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@deyraka commented on GitHub (Aug 3, 2020):

THX
i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ?

I've following the instruction above. Unfortunately, I face the same issue.
there are no problem while docker-compose up. But, when I open {ipAddress}:82 it's only show white blank page with "Loading..." title
What did I miss?

Hi, I think I found solution for my problem. Based on my trial in the last two days, I think the problem is the network config in docker-compose.yml.
When I tried the scenario to expose all service port to the host, frontend page appeared as it should. But, not when all service tried to connect each other via docker network.
It's good, finally the frontend page could be accessed. But, I'm sure that not a good choice to expose all the service port to the host which means our server would be more vulnerable from attacks.
Communication via docker network still being the best practice.
And then, I tried to modify docker-compose.yml by specifying network driver : bridge.

Voila, everything works good now.
Hopefully, this will help everyone who face the same issue

<!-- gh-comment-id:668015863 --> @deyraka commented on GitHub (Aug 3, 2020): > > THX > > i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ? > > I've following the instruction above. Unfortunately, I face the same issue. > there are no problem while docker-compose up. But, when I open {ipAddress}:82 it's only show white blank page with "Loading..." title > What did I miss? Hi, I think I found solution for my problem. Based on my trial in the last two days, I think the problem is the network config in docker-compose.yml. When I tried the scenario to expose all service port to the host, frontend page appeared as it should. But, not when all service tried to connect each other via docker network. It's good, finally the frontend page could be accessed. But, I'm sure that not a good choice to expose all the service port to the host which means our server would be more vulnerable from attacks. Communication via docker network still being the best practice. And then, I tried to modify docker-compose.yml by specifying network driver : bridge. Voila, everything works good now. Hopefully, this will help everyone who face the same issue
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@ropheefd commented on GitHub (Aug 10, 2020):

THX
i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ?

I've following the instruction above. Unfortunately, I face the same issue.
there are no problem while docker-compose up. But, when I open {ipAddress}:82 it's only show white blank page with "Loading..." title
What did I miss?

Hi, I think I found solution for my problem. Based on my trial in the last two days, I think the problem is the network config in docker-compose.yml.
When I tried the scenario to expose all service port to the host, frontend page appeared as it should. But, not when all service tried to connect each other via docker network.
It's good, finally the frontend page could be accessed. But, I'm sure that not a good choice to expose all the service port to the host which means our server would be more vulnerable from attacks.
Communication via docker network still being the best practice.
And then, I tried to modify docker-compose.yml by specifying network driver : bridge.

Voila, everything works good now.
Hopefully, this will help everyone who face the same issue

I tried modifying docker-compose.yml with network driver: bridge, but I'm getting an error. Can you show your final docker-compose as I'm having the same issue as you are?

<!-- gh-comment-id:671374673 --> @ropheefd commented on GitHub (Aug 10, 2020): > > > THX > > > i build all but when docker start i have infinite loading screen and white board on localhost:82 in logs i dont see any errors. Anyone have similar problem ? > > > > > > I've following the instruction above. Unfortunately, I face the same issue. > > there are no problem while docker-compose up. But, when I open {ipAddress}:82 it's only show white blank page with "Loading..." title > > What did I miss? > > Hi, I think I found solution for my problem. Based on my trial in the last two days, I think the problem is the network config in docker-compose.yml. > When I tried the scenario to expose all service port to the host, frontend page appeared as it should. But, not when all service tried to connect each other via docker network. > It's good, finally the frontend page could be accessed. But, I'm sure that not a good choice to expose all the service port to the host which means our server would be more vulnerable from attacks. > Communication via docker network still being the best practice. > And then, I tried to modify docker-compose.yml by specifying network driver : bridge. > > Voila, everything works good now. > Hopefully, this will help everyone who face the same issue I tried modifying docker-compose.yml with network driver: bridge, but I'm getting an error. Can you show your final docker-compose as I'm having the same issue as you are?
kerem commented 2026-02-27 15:55:54 +03:00
Author
Owner
Copy link

@stale[bot] commented on GitHub (Aug 27, 2020):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

<!-- gh-comment-id:681239545 --> @stale[bot] commented on GitHub (Aug 27, 2020): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v2.1.18
v2.1.8
v2.0.0rc
Labels
Clear labels   
$$ Paid Support

   
bug

   
bug

   
enhancement

   
help wanted

   
invalid

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
wontfix
No labels
$$ Paid Support
bug
bug
enhancement
help wanted
invalid
pull-request
question
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/ambar#270
No description provided.