mirror of
https://github.com/floccusaddon/floccus.git
synced 2026-04-25 14:16:12 +03:00
[GH-ISSUE #947] Issues with BlueCoat proxy #616
Labels
No labels
browser-specific
bug
correctness issues
enhancement
feature: Google Drive
feature: Linkwarden
feature: git
feature: nextcloud-bookmarks
feature: tabs
feature: webdav
help wanted
native-app
priority: high
priority: low
priority: medium
pull-request
question
question
stale
upstream
waiting for more information
wontfix
🙁 Not following issue template
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/floccus#616
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @trupf on GitHub (Aug 26, 2021).
Original GitHub issue: https://github.com/floccusaddon/floccus/issues/947
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
After a successfull connection setup the sync should also work
Screenshots
Desktop
(please complete the following information)
Server
(please complete the following information)
Note: Please write down the actual version numbers instead of writing 'latest'.
Debug log
2021-08-26T06:40:01.836Z Starting sync process for account ******
2021-08-26T06:40:01.854Z Using "merge default" strategy (no cache available)
2021-08-26T06:40:02.543Z Fetching bookmarks
2021-08-26T06:40:03.161Z Syncing failed with E018: Konnte nicht am Server authentifiziert werden.
Additional context
The sync is not working from my company laptop+network which requires a proxy with authentication for web access. It works though form my private laptop/network where I don't need a proxy. So I guess it has something to do with proxy settings. Firefox itself does work with the proxy. I have the same problem with Chrome - doesn't work from company laptop/network.
@MicMun commented on GitHub (Sep 12, 2021):
I have the same problem, but I don't believe that this can be fixed by Floccus.
Our proxy (Blue Coat, now Broadcom ...) deletes the Authentication header and that is the cause of the error.
I guess, your proxy do the same.
@trupf commented on GitHub (Sep 13, 2021):
I don't know what the proxy does and if it deletes the header (and also don't know how to debug). But I can login to Nextcloud via the web page with Chrome or Firefox and the nextcloud sync agent does also work. The sign in for account setup in floccus does also work but the actual sync doesn't.
@mnalis commented on GitHub (Sep 13, 2021):
@trupf can you see what nextcloud logs on the server (eg.
/var/lib/nextcloud/data/nextcloud.logor wherever it is on your server) say for that same timestamps when login fails?@trupf commented on GitHub (Sep 17, 2021):
I can see the nextcloud logs, but nothing is logged there. But I can also see sslh logs from the server and see that a connection attempt was made, Nginx is configured to not log every access attempt so I think it is really the problem as described by MicMun, that the nextcloud auth header is missing due to the proxy, although I don't understand than how it can work with the nextcloud sync client. Is there a way to authenticate a client without that header?
@mnalis commented on GitHub (Sep 17, 2021):
@trupf I'm not that familiar with the floccus code, but it looks like
src/lib/adapters/NextcloudBookmarks.tsuses HTTP Basic auth. It would be great if you could:Alternatively as a workaround, perhaps you can make exception for your nextcloud instance so it does not go via proxy (see
No proxy forin firefox config docs above)? That would only work if it is manual/automatic proxy, not in case it is transparent proxy, though.@trupf commented on GitHub (Sep 18, 2021):
error_debug.log
@mnalis commented on GitHub (Sep 18, 2021):
@trupf OK, I see that is also using
Bluecoatproxy as @MicMun.If BlueCoat removes the HTTP Basic auth headers as alleged (and your nginx
error_debug.logindeed does not showAuthorizationheader), it would explain whyGET /owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=300at line397fails with401 Unauthorizedat line626.But what is also interesting is that
GET /owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hashat line30fails withHTTP/1.1 400 Bad requestat line256before that, which is not401 Unauthorized.Can you try to (after logging in to your nextcloud instance) open in your normal browser tabs:
https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hashhttps://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=300(where
your_nextcloud.example.comis replaced with your instance FQDN)and see if they show valid JSON, or return 400/401 error messages? Also share what a
nginx.loglooks like when you try to open those URLs from browser.Also, could you try if (while using that proxy) you can connect to simple site using HTTP basic auth, like this one:
UserandPass), and after that redirect to page saying if it was successful or not.@trupf commented on GitHub (Sep 19, 2021):
I can not see any issues with both statements in the browser, they complete with no error and return valid json data.
The basic auth does also work. I have even another instance on my web-server where I'm actually using basic out with nginx and that works too.
error_1.log
error_2.log
@mnalis commented on GitHub (Sep 19, 2021):
Few observations, this seems to be the difference in headers sent to server when fetching
/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hashWhile any difference might cause suspect proxy to react differently, the most glaring difference is that headers sent by Floccus lack any Cookies, which seem to be already be used for authentication at this stage (your nginx debug logs do not mention
http header: "Authorization: basic"anywhere).So I would guess the problem occurs somewhere before
error_debug.logthat you've sent before, @trupfCould you try:
That should capture the authentication attempt.
@trupf commented on GitHub (Sep 20, 2021):
Above you wrote "after logging in to your nextcloud instance" so it is clear authentication is already done. Anyway as long as I don't explicitly log off or delete cookies I don't need new log in to nextcloud, even if I restarted my computer or started a new browser session. So authentication is probably done with cookies and only if no valid cookie found with a new login. I can follow your procedure above but you will find multiple connections than, as it will take longer an I cannot filter for just floccus. Just to make you aware I don't fill in the password field in floccus. Instead I'm using the button right at the end of the user name field to sign in with the browser (which is working, but the sync afterwards doesn't). This way I would expect floccus to create a cookie for authentication and use it (same way as the nextcloud sync client does). This way floccus does work on my home computer (without company proxy). I also tried to use a separate app password for floccus (as I have 2 factor auth enabled) but this doesn't work either.
@marcelklehr commented on GitHub (Sep 20, 2021):
Fyi, using the nextcloud login flow produces an app password which is sent via the Basic Auth header just like a normal password.
To test the authentication I recommend using curl as follows:
@trupf commented on GitHub (Sep 20, 2021):
here is the requested debug log. This time i have not deleted previous (as I think unrelated) requests.
error_debug3.log
I can not use curl on my company laptop (and if I would need a way to use the proxy). But with my private and without the proxy the request ist working.
@mnalis commented on GitHub (Sep 21, 2021):
Yes I did, for the 2 URLs you manually entered in web browser. What I'm trying to find out here is how floccus internal method (which doesn't work for you) differs from the way normal browser tabs works. If they worked in exactly the same way, they would both work (or both fail). The interesting part is that one method (web login in normal browsing tab, which produces cookie) works, while the other (floccus addon auth) fails.
Request generated by floccus in my case (with separate app password) looks like:
So in my case Floccus uses HTTP basic auth on each request, instead of "form login submit sets cookie" like web-browser login on nextcloud does. Since you say you were able to login on https://authenticationtest.com/HTTPAuth/ which also use same HTTP basic authentication method, floccus HTTP basic auth should work too.
If it doesn't, and BlueCoat strips authentication header before it reaches your nextcloud instance, but does not strip it for https://authenticationtest.com/HTTPAuth/, there must be a reason, and that reason is determined by request that is sent to proxy server. If requests were exactly the same, the result must be exactly the same too. But they never are completely the same, so BlueCoat might strip authentication header (if it really does that) depending on whether:
Accept: */*orDNT: 1?) - which might be helped by new floccus version, if we find out exactly what it doesn't like (havingcurlwould be very helpful here)nextcloudin the name), in which case you can workaround that by changing (or adding alias) FQDN for your site namecurlwould also help here)publicin path name, or/-1/or something - not much help here without kludging both floccus and nextcloud code if that is the case!). Not very likely to be the issue, though.Or in all cases, you can try contacting your proxy administrator, and see if the restrictions can be changed/lifted, of course.
Knowing what exactly is the problem would likely help with that too (if it is an option at all, of course).
Ah, I'm not sure how that way works; but is possibly harder to debug if it works differently. Could you try removing that floccus account on your company laptop, and setup a new account with separate app password for floccus, and send me that 6-step logs for that? Mixed requests should not be a huge problem, as all that pass through your company proxy seem to have
X-BlueCoat-Viaheader.Also, when you say you can't use
curl, is it because you can't download/run an executable from https://curl.se/windows/ ? Or simply because you must use company proxy to access the web? Because curl has--proxy https://your.proxy.server.example.com:portoption which should work with your company proxy.If you can download and run
curl, it would greatly simplify the debug process.@mnalis commented on GitHub (Sep 21, 2021):
@trupf if you can install
curl, the output of the following would be useful:It would hopefully show if any of the extra headers have effect on proxy, and are there certificate differences that might be problematic. Depending on the results, I can setup (if needed) few URLs that would test if it is path values that are problematic instead.
@MicMun commented on GitHub (Sep 21, 2021):
https://authenticationtest.com/HTTPAuth/ is blocked at my company, but it is interesting that folder and bookmark is acting different.
curl --proxy https://your.proxy.server.example.com:port -u "username:apppassword" -vvv https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
HTTP/1.1 400 Bad request
{"status":"error","data":"Could not find folder"}
curl --proxy https://your.proxy.server.example.com:port -u "username:apppassword" -vvv https://your_nextcloud.example.com/owncloud/index.php/apps/bookmarks/public/rest/v2/bookmark?page=0&limit=300
HTTP/1.1 401 Unauthorized
{"status":"error","data":"Unauthorized"}
The request is identical: Authorization, User-Agent and "Accept: */*" is send to nextcloud. The DNT header makes no difference in output (I tried with and without DNT).
The main differences in the response I can see:
1.: Cache-Control: no-cache, no-store, must-revalidate
2.: Cache-Control: no-cache, no-store, must-revalidate, proxy-revalidate
Only in 2.:
Authentication problem. Ignoring this.
WWW-Authenticate: Basic realm="Nextcloud", charset="UTF-8"
Set-Cookie: BCSI-CS-4935385577b06649=1; Path=/
Proxy-support: Session-based-authentication
The whole output has too much critical infos, so I cannot provide this to you. But perhaps the diff helps.
The first request .../folder/-1/hash returns the same message in browser before login, after login to nextcloud the request returns valid data without error.
I'm really surprised, that the requests differs in the result. I thought, the proxy does always the same thing and the problem must also be always the same.
@trupf commented on GitHub (Sep 21, 2021):
to answer some of your questions:
@trupf commented on GitHub (Sep 21, 2021):
What I also can confirm now is, that the Authentication: Basic header is received from my home computer (without BlueCoat proxy) but not from company. I can not tell if it is sent at all. But for another site on the same server requiring Basic auth this header is sent via BlueCoat and received. I found also, that in this case a first attempt is made without the header, server returns error 401, following second request includes the header. I have not seen this second request in the floccus debug logs.
Could it be BlueCoat is only filtering unrequested authentication credentials?
@mnalis commented on GitHub (Sep 21, 2021):
thanks @MicMun, can you also show what HTTP response does the proxy send when trying to access https://authenticationtest.com/HTTPAuth/ when it is blocked? Also, does the proxy in your case work with any site using HTTP basic auth?
I've checked with my non-proxied nextcloud, and that behavior (sending
400 Bad requestfor 1st URL if no auth header is sent) seems to be the way nextcloud bookmarks behaves, so it seems not to be related to our proxy problem. But I do agree it would likely be more correct if it sent401 Unauthorizedinstead, as the 2nd URL does.Could you try those connections with overridden
Acceptheaders, like this:-H "Accept:"to thecurlline (that would delete defaultAcceptheader)-H "Accept: application/json"to thecurlline (that would replace defaultAcceptheader with different one).I understand that the data can be sensitive. I would however like to get all headers that are sent (lines begining with
>, like 4 lines in example below) and at least the HTTP response (first line beginning with<, like< HTTP/1.1 200 OKin example below). Please do redact the two lines containing sensitive data (text afterAuthorization: Basicand hostname afterHost:), but do leave the lines themselves in the output, like the following:(lines above starting with
*mentioning TLS/SSL are also not important at this time, I've left them so you can more easily find the output I want).Thanks, that is consistent behavior with my test results when not using the proxy at all.
In this case it is nextcloud (bookmarks) which behaves differently, but proxy might well be doing it's own similar thing.
@mnalis commented on GitHub (Sep 21, 2021):
@trupf
I don't think
curlaccepts that syntaxuser:pass@proxy; you need to use separate--proxy-userif your proxy requires authentication. eg.That is where
curlwill help, it would show what exactly it is sending (and then, if needed, we can compare that against your nginx log to see what exactly is being received, to determine what exactly the proxy has changed/deleted).Here are few revised
curlcalls to try. Please do execute them exactly in the order shown (some are repeated):Of course, replace
your.proxy.server.example.com:portandyourproxyuser:yourproxypasswordwith your proxy information, as well asnextcloudusername:nextcloudapppasswordwith you nextclound username / app password.That is quite possible indeed! In fact, it could even kind-of make sense that it does that to try to prevent a certain type of CSRF attacks.
If you could use curl in previous 3 command, try this (note: only try it if you could get curl to work from previous 3 commands, and only try it once and in exact order as shown - some are duplicates and some without username/pass for a reason). Also save full output locally even if you send here just redacted information as I suggested to @MicMun, as I might have few followup questions on them:
@trupf commented on GitHub (Sep 21, 2021):
Finally I had some success with the statements as in curl_req2.txt....
curl_req2.txt
error_debug10.log
bookmarks.db#681