mirror of
https://github.com/RD17/ambar.git
synced 2026-04-25 15:35:49 +03:00
[GH-ISSUE #162] Ambar working but crawling shows issue : connect ECONNREFUSED 172.18.0.6:8080 #160
Labels
No labels
$$ Paid Support
bug
bug
enhancement
help wanted
invalid
pull-request
question
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/ambar#160
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @vteyssier on GitHub (Jun 7, 2018).
Original GitHub issue: https://github.com/RD17/ambar/issues/162
Hi ambar team,
Great piece of software, congrats.
I made an "almost" working solution in no time thanks to your howto. I'm now able to fully assess ambar and ready to recommend.
There is still one little conf issue I cannot fix by myself : the crawling does not work for me. The docs and pdf I have in my data directory are not crawled and not taken in account into the index.
May I ask you help ?
Here is my docker-compose.yml :
version: "2.1"
networks:
internal_network:
services:
db:
restart: always
networks:
- internal_network
image: ambar/ambar-mongodb:latest
environment:
- cacheSizeGB=2
volumes:
- /home/vincent/ambar_data/db
expose:
- "27017"
es:
restart: always
networks:
- internal_network
image: ambar/ambar-es:latest
expose:
- "9200"
environment:
- cluster.name=ambar-es
- ES_JAVA_OPTS=-Xms1500M -Xmx1500M
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- /home/vincent/ambar_data/es
rabbit:
restart: always
networks:
- internal_network
image: ambar/ambar-rabbit:latest
hostname: rabbit
expose:
- "15672"
- "5672"
volumes:
- /home/vincent/ambar_data/rabbitmq
redis:
restart: always
sysctls:
- net.core.somaxconn=1024
networks:
- internal_network
image: ambar/ambar-redis:latest
expose:
- "6379"
serviceapi:
depends_on:
redis:
condition: service_healthy
rabbit:
condition: service_healthy
es:
condition: service_healthy
db:
condition: service_healthy
restart: always
networks:
- internal_network
image: ambar/ambar-serviceapi:latest
expose:
- "8081"
environment:
- mongoDbUrl=mongodb://db:27017/ambar_data
- elasticSearchUrl=http://es:9200
- redisHost=redis
- redisPort=6379
- rabbitHost=amqp://rabbit
- langAnalyzer=ambar_en
webapi:
depends_on:
serviceapi:
condition: service_healthy
restart: always
networks:
- internal_network
image: ambar/ambar-webapi:latest
expose:
- "8080"
ports:
- "8080:8080"
environment:
- analyticsToken=cda4b0bb11a1f32aed7564b08c455992
- uiLang=en
- mongoDbUrl=mongodb://db:27017/ambar_data
- elasticSearchUrl=http://es:9200
- redisHost=redis
- redisPort=6379
- serviceApiUrl=http://serviceapi:8081
- rabbitHost=amqp://rabbit
frontend:
depends_on:
webapi:
condition: service_healthy
image: ambar/ambar-frontend:latest
restart: always
networks:
- internal_network
ports:
- "80:80"
expose:
- "80"
environment:
- api=http://192.168.80.128:8080
pipeline0:
depends_on:
serviceapi:
condition: service_healthy
image: ambar/ambar-pipeline:latest
restart: always
networks:
- internal_network
environment:
- id=0
- api_url=http://serviceapi:8081
- rabbit_host=amqp://rabbit
mycrawler:
depends_on:
serviceapi:
condition: service_healthy
image: ambar/ambar-local-crawler
restart: always
networks:
- internal_network
expose:
- "8082"
environment:
- name=mycrawler
volumes:
- /home/vincent/source_data
And here is my crawler log :
vincent@ubuntu:~$ sudo docker logs 07c42384e4a8
[sudo] password for vincent:
[info] API runs on master thread
[info] Creating fork for the file-watcher process
Started API on :::8082
{ RequestError: Error: connect ECONNREFUSED 172.18.0.6:8080
at new RequestError (/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/node_modules/request/request.js:186:22)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at Request.onRequestError (/node_modules/request/request.js:878:8)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
name: 'RequestError',
message: 'Error: connect ECONNREFUSED 172.18.0.6:8080',
cause:
{ Error: connect ECONNREFUSED 172.18.0.6:8080
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '172.18.0.6',
port: 8080 },
error:
{ Error: connect ECONNREFUSED 172.18.0.6:8080
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '172.18.0.6',
port: 8080 },
options:
{ uri: 'http://serviceapi:8080/api/logs',
method: 'POST',
body: '{"source_id":"mycrawler","type":"info","message":"Creating fork for the file-watcher process","created_datetime":"2018-06-06 16:37:27.915"}',
headers: { 'Content-Type': 'application/json' },
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false },
response: undefined }
(node:5) UnhandledPromiseRejectionWarning: RequestError: Error: connect ECONNREFUSED 172.18.0.6:8080
at new RequestError (/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/node_modules/request/request.js:186:22)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at Request.onRequestError (/node_modules/request/request.js:878:8)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
(node:5) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:5) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
{ RequestError: Error: connect ECONNREFUSED 172.18.0.6:8080
at new RequestError (/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/node_modules/request/request.js:186:22)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at Request.onRequestError (/node_modules/request/request.js:878:8)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
name: 'RequestError',
message: 'Error: connect ECONNREFUSED 172.18.0.6:8080',
cause:
{ Error: connect ECONNREFUSED 172.18.0.6:8080
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '172.18.0.6',
port: 8080 },
error:
{ Error: connect ECONNREFUSED 172.18.0.6:8080
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '172.18.0.6',
port: 8080 },
options:
{ uri: 'http://serviceapi:8080/api/logs',
method: 'POST',
body: '{"source_id":"mycrawler","type":"info","message":"API runs on master thread","created_datetime":"2018-06-06 16:37:27.732"}',
headers: { 'Content-Type': 'application/json' },
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false },
response: undefined }
(node:5) UnhandledPromiseRejectionWarning: RequestError: Error: connect ECONNREFUSED 172.18.0.6:8080
at new RequestError (/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/node_modules/request/request.js:186:22)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at Request.onRequestError (/node_modules/request/request.js:878:8)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
(node:5) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
[info] File-watcher runs on worker thread
{ RequestError: Error: connect ECONNREFUSED 172.18.0.6:8080
at new RequestError (/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/node_modules/request/request.js:186:22)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at Request.onRequestError (/node_modules/request/request.js:878:8)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
name: 'RequestError',
message: 'Error: connect ECONNREFUSED 172.18.0.6:8080',
cause:
{ Error: connect ECONNREFUSED 172.18.0.6:8080
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '172.18.0.6',
port: 8080 },
error:
{ Error: connect ECONNREFUSED 172.18.0.6:8080
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '172.18.0.6',
port: 8080 },
options:
{ uri: 'http://serviceapi:8080/api/logs',
method: 'POST',
body: '{"source_id":"mycrawler","type":"info","message":"File-watcher runs on worker thread","created_datetime":"2018-06-06 16:37:38.042"}',
headers: { 'Content-Type': 'application/json' },
callback: [Function: RP$callback],
transform: undefined,
simple: true,
resolveWithFullResponse: false,
transform2xxOnly: false },
response: undefined }
(node:23) UnhandledPromiseRejectionWarning: RequestError: Error: connect ECONNREFUSED 172.18.0.6:8080
at new RequestError (/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/node_modules/request/request.js:186:22)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at Request.onRequestError (/node_modules/request/request.js:878:8)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
(node:23) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:23) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Thanks for your help or suggestions !
Vincent
@vteyssier commented on GitHub (Jun 7, 2018):
Just a few details more.
My Ambar setup is working fine and I can search any document I submit with the portal.
But, my {dataPath} still remains empty. I was expecting to find here Ambar data (es, db, rabbit ...)
Thx
Vince
@sochix commented on GitHub (Jun 8, 2018):
Hello, I noticed two problems.
First one, your forget to correctly mount the data folder for crawler. In your example
But, it should be something like
Second one, for some reason crawler can't connect to webapi. Check your networks settings. Try to ping the webapi from inside your host:
curl 172.18.0.6:8080/api@vteyssier commented on GitHub (Jun 8, 2018):
Dear Ilya,
Thanks for your help.
After fixing the volume path, it works perfectly.
May I ask you the meaning, in Ambar point of view, of the two paths separated with a colon ?
Once again, thanks
Vince
@sochix commented on GitHub (Jun 8, 2018):
It's recommended way for mounting docker volumes
@chrisl8 commented on GitHub (Jun 8, 2018):
@vteyssier Docker is running a virtual machine. The left side of the colon is the path on your hard drive and the right side is the path inside of the Docker container that is mapped to your folder.