mirror of
https://github.com/RD17/ambar.git
synced 2026-04-25 15:35:49 +03:00
[GH-ISSUE #28] Ambar behind a HTTP proxy #29
Labels
No labels
$$ Paid Support
bug
bug
enhancement
help wanted
invalid
pull-request
question
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/ambar#29
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @knewbery on GitHub (May 3, 2017).
Original GitHub issue: https://github.com/RD17/ambar/issues/28
Hi,
I seem to be having problems with the ambar_webapi docker not using the system HTTP proxy correctly.
I have installed ambar self-hosted community edition onto Centos 7, which is behind a HTTP proxy. I have setup systemd for docker to define the HTTP_PROXY and HTTPS_PROXY correctly. ie I can download/book amabar ok.
I can then access the web front end ok, (changed to port 8005) but everything else is standard.. however I can't login, signup or anything - get an 'opps something went wrong message'.
Inspecting the docker log for ambar_webapi seems to show attempts to access a remote host (52.64.9.77) (and amazonaws.com host - mandrillapp.com??) without using the HTTPS proxy
`
[root@kgs-sts-fusion ambar]# ./ambar.py start
/\ _ \ /'_/
\/\ _\ /\ _ /\ `\\ \ \L\ /\ \ \ \L\ \ \ \L\ \ \ \L\ \
\ \ __ \ \ _\ \ \ _ <'\ \ __ \ \ , /
\ \ /\ \ \ _/\ \ \ \L\ \ \ /\ \ \ \ \
\ _\ _\ _\ _\ _/ \ _\ _\ _\ _
////// //// /////_// /
Docker version 17.04.0-ce, build 4845c56
docker-compose version 1.13.0, build 1719ceb
vm.max_map_count = 262144
net.ipv4.ip_local_port_range = 15000 61000
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2000
net.ipv4.tcp_max_syn_backlog = 2048
Creating network "ambar_internal_network" with the default driver
Creating ambar_db_1 ...
Creating ambar_rabbit_1 ...
Creating ambar_proxy_1 ...
Creating ambar_es_1 ...
Creating ambar_db_1
Creating ambar_webapi-cache_1 ...
Creating ambar_rabbit_1
Creating ambar_es_1
Creating ambar_webapi-cache_1
Creating ambar_es_1 ... done
Creating ambar_webapi_1 ...
Creating ambar_webapi_1 ... done
Creating ambar_frontend_1 ...
Creating ambar_frontend_1 ... done
Waiting for Ambar to start...
Ambar is running on http://147.66.12.53:8005
[root@kgs-sts-fusion ambar]# cat
docker inspect --format='{{.LogPath}}' ambar_webapi_1{"log":"2017/05/03 05:44:08 Waiting for host: \n","stream":"stderr","time":"2017-05-03T05:44:08.385748978Z"}
{"log":"2017/05/03 05:44:08 Waiting for host: es:9200\n","stream":"stderr","time":"2017-05-03T05:44:08.385884331Z"}
{"log":"2017/05/03 05:44:08 Connected to unix:///var/run/docker.sock\n","stream":"stderr","time":"2017-05-03T05:44:08.388017144Z"}
{"log":"2017/05/03 05:44:22 Received 200 from http://es:9200\n","stream":"stderr","time":"2017-05-03T05:44:22.302769292Z"}
{"log":"Crawler schedule service initialized\n","stream":"stdout","time":"2017-05-03T05:44:24.380922736Z"}
{"log":"Pipeline initialized\n","stream":"stdout","time":"2017-05-03T05:44:24.71064609Z"}
{"log":"Started on :::8080\n","stream":"stdout","time":"2017-05-03T05:44:24.720793191Z"}
{"log":"{ [Error: connect ECONNREFUSED 52.64.27.232:443]\n","stream":"stderr","time":"2017-05-03T06:53:42.270438821Z"}
{"log":" code: 'ECONNREFUSED',\n","stream":"stderr","time":"2017-05-03T06:53:42.270489177Z"}
{"log":" errno: 'ECONNREFUSED',\n","stream":"stderr","time":"2017-05-03T06:53:42.270497139Z"}
{"log":" syscall: 'connect',\n","stream":"stderr","time":"2017-05-03T06:53:42.270503494Z"}
{"log":" address: '52.64.27.232',\n","stream":"stderr","time":"2017-05-03T06:53:42.27050999Z"}
{"log":" port: 443 }\n","stream":"stderr","time":"2017-05-03T06:53:42.270516275Z"}
{"log":"{ [Error: connect ECONNREFUSED 52.64.9.77:443]\n","stream":"stderr","time":"2017-05-03T06:53:43.182362118Z"}
{"log":" code: 'ECONNREFUSED',\n","stream":"stderr","time":"2017-05-03T06:53:43.18240549Z"}
{"log":" errno: 'ECONNREFUSED',\n","stream":"stderr","time":"2017-05-03T06:53:43.182413382Z"}
{"log":" syscall: 'connect',\n","stream":"stderr","time":"2017-05-03T06:53:43.182444112Z"}
{"log":" address: '52.64.9.77',\n","stream":"stderr","time":"2017-05-03T06:53:43.182451306Z"}
{"log":" port: 443 }\n","stream":"stderr","time":"2017-05-03T06:53:43.182457382Z"}
`
it seems that when I try to recover my password I type my email, and hit 'recover password' causes a new entry in the ambar_webapi docker log which looks like our http proxy (see above).
I have not yet been able to login at all to the Ambar web front end.
any ideas?
Regards
Kym
@sochix commented on GitHub (May 4, 2017):
Hello! Seems like your proxy blocks access to mandrill. We use it to send confirmation link. As a temp workaround you can set auth to
noneinconfig.json.@knewbery commented on GitHub (May 8, 2017):
Hi,
I already have auth set to 'none', and our proxy doesn't block access to mandrill.com (ie I can wget mandrill.com) , I think it is more a problem of an instance where the proxy is not used?
Kym
@sochix commented on GitHub (May 10, 2017):
@knewbery try to update to the latest version 0.9.3. We fixed some network problems. Use
sudo ./ambar.py update@knewbery commented on GitHub (May 11, 2017):
Hi, ok - tried that: (my proxy is 147.66.7.6:8080)
(I am running as root, and docker is using http proxy ok)
./ambar.py update
Docker version 17.04.0-ce, build 4845c56
docker-compose version 1.13.0, build 1719ceb
Stopping Ambar...
"docker rm" requires at least 1 argument(s).
See 'docker rm --help'.
Usage: docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
Crawlers containers removed
"docker rm" requires at least 1 argument(s).
See 'docker rm --help'.
Usage: docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
Pipeline containers removed
Removing network ambar_internal_network
WARNING: Network ambar_internal_network not found.
Ambar is stopped
--2017-05-11 10:04:06-- https://static.ambar.cloud/docker-compose.template.yml
Connecting to 147.66.7.6:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 3077 (3.0K) [application/octet-stream]
Saving to: ‘/home/ambar/docker-compose.template.yml’
100%[===============================================================================================================================================================>] 3,077 --.-K/s in 0s
2017-05-11 10:04:08 (472 MB/s) - ‘/home/ambar/docker-compose.template.yml’ saved [3077/3077]
Error response from daemon: Get https://registry-1.docker.io/v2/: read tcp 147.66.12.53:38839->147.66.7.6:443: read: connection reset by peer
Traceback (most recent call last):
File "./ambar.py", line 275, in
update(configuration)
File "./ambar.py", line 211, in update
pullImages(configuration)
File "./ambar.py", line 100, in pullImages
runShellCommandStrict("docker pull {0}/ambar-crawler:latest".format(dockerRepo))
File "./ambar.py", line 47, in runShellCommandStrict
subprocess.check_call(command, shell = True)
File "/usr/lib64/python3.4/subprocess.py", line 558, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker pull ambar/ambar-crawler:latest' returned non-zero exit status 1
--> tried wget to check the URL
wget https://registry-1.docker.io/v2/
--2017-05-11 10:04:56-- https://registry-1.docker.io/v2/
Connecting to 147.66.7.6:8080... connected.
Proxy request sent, awaiting response... 401 Unauthorized
Authorization failed.
response in browser is = {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
probably some authorisation problem??
K
@sochix commented on GitHub (May 11, 2017):
Seems like some problems on docker side - docker servers not responding. Just try again a bit later.
@knewbery commented on GitHub (May 12, 2017):
Hi,
I think I have found the problem - in a subtly related to the proxy specification for docker - I had to use a specific port (8080) not 80 on our proxy that then allowed docker to access the docker.io repositories. All solved now - except for login problems
so now that I get to the Ambar login page - is there a default login for Ambar CE ? because I can't get the 'signup' feature working
Cheers
Kym
@knewbery commented on GitHub (May 12, 2017):
p.s. I can signup on a self hosted system. but upon login I am told 'user is not in an active state'. (and I have auth = 'none' in config.json
@sochix commented on GitHub (May 12, 2017):
change auth to 'basic' in config.json and do .
sudo ./ambar.py restart@knewbery commented on GitHub (May 14, 2017):
Have made the change to Ambar CE (self hosted on Centos 7) config.json, setting auth = 'basic', restarted ambar.py (as you stated). Also tried restarting docker - but as expected didn't solve the problem. I have attempted signup with my email address, but each time I come back to login I am presented with 'user not in an active state' ... trying to recover my password also fails ('Opps.. something went wrong. please reload the page') and since only 1 user account is allowed for Ambar CE, I can't login with any other account.. or sign up.
Is there a default login to manage accounts and make my signup account active?
this query may be off-topic from the original problem now... maybe make a new issue?
@sochix commented on GitHub (May 16, 2017):
@knewbery , seems like the best option for you is to reset all data in Ambar and start from the beginning. For doing it type
sudo ./ambar.py reset@knewbery commented on GitHub (May 17, 2017):
Done that a few times.. even re-installed.. There is something wrong in the 'sign up' process, I suppose - do I need the host to have a functional SMTP sendmail to send out a password to a valid email address on signup?
@sochix commented on GitHub (May 19, 2017):
@knewbery nope, you shouldn't have an SMTP sendmail. You should have access to mandrill servers from your machine. Did you recieve "Set Password" mailing?
@knewbery commented on GitHub (May 22, 2017):
that's probably the problem ? - on my company network, smtp access to the mandrill servers is blocked - ie everything in/out of our network except http/https (which have to go via http/http proxy)
So I don't get the 'set password' mailing.
Unless you can access mandrill via http/https (proxy) then Ambar on a company network behind a firewall with only http/https access is going to be a problem.
@isido993 commented on GitHub (May 22, 2017):
@knewbery
You can set
api.authin yourconfig.jsontononeto disable authentication (sudo ./ambar.py reset,sudo ./ambar.py updateandsudo ./ambar.py restartrequired), that'll update your Ambar to the latest version and allow you to use Ambar without signing up.@knewbery commented on GitHub (May 23, 2017):
sigh I still seem to be going around in circles.
I have stopped ambar, reset and then updated it to the latest (as of 2017/05/23)
[root@kgs-sts-fusion ambar]# ./ambar.py start
/\ _ \ /'_/
\/\ _\ /\ _ /\ `\\ \ \L\ /\ \ \ \L\ \ \ \L\ \ \ \L\ \
\ \ __ \ \ _\ \ \ _ <'\ \ __ \ \ , /
\ \ /\ \ \ _/\ \ \ \L\ \ \ /\ \ \ \ \
\ _\ _\ _\ _\ _/ \ _\ _\ _\ _
////// //// /////_// /
Docker version 17.04.0-ce, build 4845c56
docker-compose version 1.13.0, build 1719ceb
vm.max_map_count = 262144
net.ipv4.ip_local_port_range = 15000 61000
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2000
net.ipv4.tcp_max_syn_backlog = 2048
Creating network "ambar_internal_network" with the default driver
Creating ambar_es_1 ...
Creating ambar_webapi-cache_1 ...
Creating ambar_db_1 ...
Creating ambar_proxy_1 ...
Creating ambar_es_1
Creating ambar_db_1
Creating ambar_rabbit_1 ...
Creating ambar_webapi-cache_1
Creating ambar_proxy_1
Creating ambar_db_1 ... done
Creating ambar_webapi_1 ...
Creating ambar_webapi_1 ... done
Creating ambar_frontend_1 ...
Creating ambar_frontend_1 ... done
Waiting for Ambar to start...
Ambar is running on http://147.66.12.53:80
[root@kgs-sts-fusion ambar]# nano config.json
[root@kgs-sts-fusion ambar]# ./ambar.py --version
0.0.2
So - signing up fails. and I can't see anyway of using Ambar without having an active email/password combination? Even though config.json api:auth = none (I restarted ambar after changing config.json) I am still presented with a login screen asking for an email/password.
So what do you mean 'without signing up' ? is there a default email/password?
@isido993 commented on GitHub (May 23, 2017):
@knewbery
Please do:
sudo docker logs ambar_webapi_1and post the results hereThnx.
@knewbery commented on GitHub (May 23, 2017):
Looks like part of the web api not using the proxy or trying to access mandrill:
[root@kgs-sts-fusion ambar]# docker logs ambar_webapi_1
2017/05/23 23:32:38 Waiting for host:
2017/05/23 23:32:38 Waiting for host: es:9200
2017/05/23 23:32:38 Connected to unix:///var/run/docker.sock
2017/05/23 23:32:50 Received 200 from http://es:9200
Crawler schedule service initialized
Pipeline initialized
Started on :::8080
{ Error: connect ECONNREFUSED 52.64.27.232:443
at Object.exports._errnoException (util.js:1018:11)
at exports._exceptionWithHostPort (util.js:1041:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '52.64.27.232',
port: 443 }
{ Error: connect ECONNREFUSED 52.64.9.77:443
at Object.exports._errnoException (util.js:1018:11)
at exports._exceptionWithHostPort (util.js:1041:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '52.64.9.77',
port: 443 }
@sochix commented on GitHub (May 28, 2017):
@knewbery can you post your current
config.jsonplease. We want to fix your problem for sure. Maybe if it suitable for you we can have a live-chat session via Skype or smthng.@knewbery commented on GitHub (May 29, 2017):
147.66.12.53 is the IP of the host running ambar on our internal network. We have a http proxy on another host.
here is current config.json
{
"ocr": {
"pdfMaxPageCount": 5000,
"pdfSymbolsPerPageThreshold": 100
},
"db": {
"cacheSizeGb": 2
},
"dockerComposeTemplate": "https://static.ambar.cloud/docker-compose.template.yml",
"dataPath": "/opt/ambar",
"es": {
"containerSize": "2g",
"heapSize": "1g"
},
"api": {
"auth": "none",
"defaultLangAnalyzer": "ambar_en",
"crawlerCount": 1,
"external": {
"host": "147.66.12.53",
"port": "80",
"protocol": "http"
},
"pipelineCount": 1,
"mode": "ce",
"analyticsToken": "cda4b0bb11a1f32aed7564b08c455992",
"showFilePreview": "false",
"cacheSize": "1g"
},
"dropbox": {
"redirectUri": "",
"clientId": ""
},
"fe": {
"external": {
"host": "147.66.12.53",
"port": "80",
"protocol": "http"
}
},
"dockerRepo": "ambar"
}
@sochix commented on GitHub (Jun 2, 2017):
Hi @knewbery thanks for your patience, it was a bug. We just ignore
authsetting inconfig.json. We fixed it in 0.10.0 release. Please runsudo ./ambar.py reset. && sudo ./ambar.py update@knewbery commented on GitHub (Jun 5, 2017):
Hi,
I am afraid I still have a problem. I used 'docker logs ambar_webapi_1" after I got the usual login screen (and only option was to 'sign up' - I am still presented with the 'login' screen after a reset & update)
Even so, access to mandrill.com doesn't seem to be using a proxy. I cannot sign up, and do not receive a password nor can I recover a password/etc.. Don't forget my host is behind a firewall and the only access to external services is via proxy http/https
I have reset and updated ambar as requested. My config.json is un-modified compared to what I posted previously..
I am only using one config.json file - I tried the --useLocalConfig but didn't seem to make any difference
docker logs ambar_webapi_1
2017/06/05 01:54:56 Waiting for host:
2017/06/05 01:54:56 Waiting for host: es:9200
2017/06/05 01:54:56 Connected to unix:///var/run/docker.sock
2017/06/05 01:55:08 Received 200 from http://es:9200
Crawler schedule service initialized
Pipeline initialized
Started on :::8080
{ Error: connect ECONNREFUSED 52.64.9.77:443
at Object.exports._errnoException (util.js:1018:11)
at exports._exceptionWithHostPort (util.js:1041:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '52.64.9.77',
port: 443 }
{ Error: connect ECONNREFUSED 52.64.9.77:443
at Object.exports._errnoException (util.js:1018:11)
at exports._exceptionWithHostPort (util.js:1041:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '52.64.9.77',
port: 443 }
#./ambar.py start
/\ _ \ /'_/
\/\ _\ /\ _ /\ `\\ \ \L\ /\ \ \ \L\ \ \ \L\ \ \ \L\ \
\ \ __ \ \ _\ \ \ _ <'\ \ __ \ \ , /
\ \ /\ \ \ _/\ \ \ \L\ \ \ /\ \ \ \ \
\ _\ _\ _\ _\ _/ \ _\ _\ _\ _
////// //// /////_// /
Docker version 17.04.0-ce, build 4845c56
docker-compose version 1.13.0, build 1719ceb
vm.max_map_count = 262144
net.ipv4.ip_local_port_range = 15000 61000
net.ipv4.tcp_fin_timeout = 30
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2000
net.ipv4.tcp_max_syn_backlog = 2048
ambar_es_1 is up-to-date
ambar_webapi-cache_1 is up-to-date
ambar_rabbit_1 is up-to-date
ambar_proxy_1 is up-to-date
ambar_db_1 is up-to-date
ambar_webapi_1 is up-to-date
ambar_frontend_1 is up-to-date
Waiting for Ambar to start...
Ambar is running on http://147.66.12.53:80
@sochix commented on GitHub (Jun 18, 2017):
Hi! We fixed the issue in the latest release, please update Ambar and try again.
@knewbery commented on GitHub (Jun 19, 2017):
OK! seems to be working now. Will try it out.
@sochix commented on GitHub (Jun 19, 2017):
Ok, I will close issue