[GH-ISSUE #816] Regression in the latest version 1.38.1 #134

Closed
opened 2026-03-03 12:08:36 +03:00 by kerem · 7 comments
Owner

Originally created by @drehelis on GitHub (Jun 6, 2022).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/816

Version 1.38.0 works as expected, while the latest 1.38.1 we've started to receive:

17:49:45  + python gcs_utils.py --host fake-gcs-censored-pr-291-4.jenkins.svc --action download --bucket xxx --dst ./
17:49:45  DEBUG:__main__:Downloading from xxx...
17:49:45  DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
17:49:45  DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): fake-gcs-censored-pr-291-4.jenkins.svc:4443
17:49:45  DEBUG:google.api_core.retry:Retrying due to HTTPConnectionPool(host='fake-gcs-censored-pr-291-4.jenkins.svc', port=4443): Max retries exceeded with url: /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f18d8549b50>: Failed to establish a new connection: [Errno -2] Name does not resolve')), sleeping 0.4s ...
17:49:46  DEBUG:urllib3.connectionpool:Starting new HTTP connection (2): fake-gcs-censored-pr-291-4.jenkins.svc:4443
17:49:46  DEBUG:google.api_core.retry:Retrying due to HTTPConnectionPool(host='fake-gcs-censored-pr-291-4.jenkins.svc', port=4443): Max retries exceeded with url: /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f18d856e070>: Failed to establish a new connection: [Errno -2] Name does not resolve')), sleeping 1.4s ...
17:49:47  DEBUG:urllib3.connectionpool:Starting new HTTP connection (3): fake-gcs-censored-pr-291-4.jenkins.svc:4443
17:49:47  DEBUG:google.api_core.retry:Retrying due to HTTPConnectionPool(host='fake-gcs-censored-pr-291-4.jenkins.svc', port=4443): Max retries exceeded with url: /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f18d85493d0>: Failed to establish a new connection: [Errno -2] Name does not resolve')), sleeping 4.6s ...
17:49:53  DEBUG:urllib3.connectionpool:Starting new HTTP connection (4): fake-gcs-censored-pr-291-4.jenkins.svc:4443
17:49:53  DEBUG:urllib3.connectionpool:http://fake-gcs-censored-pr-291-4.jenkins.svc:4443 "GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1" 200 None
17:49:53  DEBUG:urllib3.connectionpool:http://fake-gcs-censored-pr-291-4.jenkins.svc:4443 "GET /download/storage/v1/b/xxx/o/env-cerebro?alt=media&generation=1654526936027844 HTTP/1.1" 404 10
17:49:53  Traceback (most recent call last):
17:49:53    File "/usr/local/lib/python3.9/site-packages/google/cloud/storage/client.py", line 1113, in download_blob_to_file
17:49:53      blob_or_uri._do_download(
17:49:53    File "/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 1003, in _do_download
17:49:53      response = download.consume(transport, timeout=timeout)
17:49:53    File "/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/download.py", line 174, in consume
17:49:53      return _request_helpers.wait_and_retry(
17:49:53    File "/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/_request_helpers.py", line 147, in wait_and_retry
17:49:53      response = func()
17:49:53    File "/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/download.py", line 167, in retriable_request
17:49:53      self._process_response(result)
17:49:53    File "/usr/local/lib/python3.9/site-packages/google/resumable_media/_download.py", line 184, in _process_response
17:49:53      _helpers.require_status_code(
17:49:53    File "/usr/local/lib/python3.9/site-packages/google/resumable_media/_helpers.py", line 99, in require_status_code
17:49:53      raise common.InvalidResponse(
17:49:53  google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

gcs_utils.py is a tiny wrapper to upload/download/list objects using google-cloud-storage

fake-gcs is started with the following args:

args: ["-scheme", "http", "-data", "/var/tmp", "-public-host", "fake-gcs-censored-pr-291-4.jenkins.svc:4443"]
Originally created by @drehelis on GitHub (Jun 6, 2022). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/816 Version `1.38.0` works as expected, while the latest `1.38.1` we've started to receive: ``` 17:49:45 + python gcs_utils.py --host fake-gcs-censored-pr-291-4.jenkins.svc --action download --bucket xxx --dst ./ 17:49:45 DEBUG:__main__:Downloading from xxx... 17:49:45 DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) 17:49:45 DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): fake-gcs-censored-pr-291-4.jenkins.svc:4443 17:49:45 DEBUG:google.api_core.retry:Retrying due to HTTPConnectionPool(host='fake-gcs-censored-pr-291-4.jenkins.svc', port=4443): Max retries exceeded with url: /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f18d8549b50>: Failed to establish a new connection: [Errno -2] Name does not resolve')), sleeping 0.4s ... 17:49:46 DEBUG:urllib3.connectionpool:Starting new HTTP connection (2): fake-gcs-censored-pr-291-4.jenkins.svc:4443 17:49:46 DEBUG:google.api_core.retry:Retrying due to HTTPConnectionPool(host='fake-gcs-censored-pr-291-4.jenkins.svc', port=4443): Max retries exceeded with url: /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f18d856e070>: Failed to establish a new connection: [Errno -2] Name does not resolve')), sleeping 1.4s ... 17:49:47 DEBUG:urllib3.connectionpool:Starting new HTTP connection (3): fake-gcs-censored-pr-291-4.jenkins.svc:4443 17:49:47 DEBUG:google.api_core.retry:Retrying due to HTTPConnectionPool(host='fake-gcs-censored-pr-291-4.jenkins.svc', port=4443): Max retries exceeded with url: /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f18d85493d0>: Failed to establish a new connection: [Errno -2] Name does not resolve')), sleeping 4.6s ... 17:49:53 DEBUG:urllib3.connectionpool:Starting new HTTP connection (4): fake-gcs-censored-pr-291-4.jenkins.svc:4443 17:49:53 DEBUG:urllib3.connectionpool:http://fake-gcs-censored-pr-291-4.jenkins.svc:4443 "GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1" 200 None 17:49:53 DEBUG:urllib3.connectionpool:http://fake-gcs-censored-pr-291-4.jenkins.svc:4443 "GET /download/storage/v1/b/xxx/o/env-cerebro?alt=media&generation=1654526936027844 HTTP/1.1" 404 10 17:49:53 Traceback (most recent call last): 17:49:53 File "/usr/local/lib/python3.9/site-packages/google/cloud/storage/client.py", line 1113, in download_blob_to_file 17:49:53 blob_or_uri._do_download( 17:49:53 File "/usr/local/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 1003, in _do_download 17:49:53 response = download.consume(transport, timeout=timeout) 17:49:53 File "/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/download.py", line 174, in consume 17:49:53 return _request_helpers.wait_and_retry( 17:49:53 File "/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/_request_helpers.py", line 147, in wait_and_retry 17:49:53 response = func() 17:49:53 File "/usr/local/lib/python3.9/site-packages/google/resumable_media/requests/download.py", line 167, in retriable_request 17:49:53 self._process_response(result) 17:49:53 File "/usr/local/lib/python3.9/site-packages/google/resumable_media/_download.py", line 184, in _process_response 17:49:53 _helpers.require_status_code( 17:49:53 File "/usr/local/lib/python3.9/site-packages/google/resumable_media/_helpers.py", line 99, in require_status_code 17:49:53 raise common.InvalidResponse( 17:49:53 google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>) ``` `gcs_utils.py` is a tiny wrapper to upload/download/list objects using `google-cloud-storage` fake-gcs is started with the following args: ``` args: ["-scheme", "http", "-data", "/var/tmp", "-public-host", "fake-gcs-censored-pr-291-4.jenkins.svc:4443"] ```
kerem 2026-03-03 12:08:36 +03:00
  • closed this issue
  • added the
    bug
    label
Author
Owner

@fsouza commented on GitHub (Jun 6, 2022):

hmm probably something with the range changes.

I'll give investigate later.

<!-- gh-comment-id:1147701461 --> @fsouza commented on GitHub (Jun 6, 2022): hmm probably something with the range changes. I'll give investigate later.
Author
Owner

@fsouza commented on GitHub (Jun 6, 2022):

@drehelis would you be able to share the logs from the server? Just so we had the actual requests more readily available.

<!-- gh-comment-id:1147702408 --> @fsouza commented on GitHub (Jun 6, 2022): @drehelis would you be able to share the logs from the server? Just so we had the actual requests more readily available.
Author
Owner

@drehelis commented on GitHub (Jun 6, 2022):

@fsouza, hope this helps:

time="2022-06-06T17:44:33Z" level=info msg="couldn't load any objects or buckets from \"/var/tmp\", starting empty"
time="2022-06-06T17:44:33Z" level=info msg="server started at http://[::]:4443"
...
...
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 467"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 462"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 465"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 462"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 464"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 469"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 472"
time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 5635"
...
...
time="2022-06-06T17:48:45Z" level=info msg="10.0.0.249 - - [06/Jun/2022:17:48:45 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 5635"
time="2022-06-06T17:48:45Z" level=info msg="10.0.0.249 - - [06/Jun/2022:17:48:45 +0000] \"GET /download/storage/v1/b/xxx/o/env-cerebro?alt=media&generation=1654537675491586 HTTP/1.1\" 404 10"
<!-- gh-comment-id:1147721171 --> @drehelis commented on GitHub (Jun 6, 2022): @fsouza, hope this helps: ``` time="2022-06-06T17:44:33Z" level=info msg="couldn't load any objects or buckets from \"/var/tmp\", starting empty" time="2022-06-06T17:44:33Z" level=info msg="server started at http://[::]:4443" ... ... time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 467" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 462" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 465" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 462" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 464" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 469" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 461" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 472" time="2022-06-06T17:47:55Z" level=info msg="127.0.0.1 - - [06/Jun/2022:17:47:55 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 5635" ... ... time="2022-06-06T17:48:45Z" level=info msg="10.0.0.249 - - [06/Jun/2022:17:48:45 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 5635" time="2022-06-06T17:48:45Z" level=info msg="10.0.0.249 - - [06/Jun/2022:17:48:45 +0000] \"GET /download/storage/v1/b/xxx/o/env-cerebro?alt=media&generation=1654537675491586 HTTP/1.1\" 404 10" ```
Author
Owner

@fsouza commented on GitHub (Jun 6, 2022):

Oh I see, I think this is a side effect of the generation fix for the PHP client.

@drehelis can you check if this PR fixes the issue for you? https://github.com/fsouza/fake-gcs-server/pull/818

<!-- gh-comment-id:1147968557 --> @fsouza commented on GitHub (Jun 6, 2022): Oh I see, I think this is a side effect of the generation fix for the PHP client. @drehelis can you check if this PR fixes the issue for you? https://github.com/fsouza/fake-gcs-server/pull/818
Author
Owner

@fsouza commented on GitHub (Jun 6, 2022):

While at it, can you try #819 too? That would be less disruptive.

<!-- gh-comment-id:1147977641 --> @fsouza commented on GitHub (Jun 6, 2022): While at it, can you try #819 too? That would be less disruptive.
Author
Owner

@drehelis commented on GitHub (Jun 7, 2022):

Tested #819 only, seems to work.

fake-gcs-server git:(3fdc10f) docker build -t local/fake-gcs-server .
...
...
fake-gcs-server git:(3fdc10f) docker run --rm --name fake-gcs-server -p 4443:4443 local/fake-gcs-server -scheme http -public-host storage.gcs.127.0.0.1.nip.io:4443
time="2022-06-07T12:43:22Z" level=info msg="couldn't load any objects or buckets from \"/data\", starting empty"
time="2022-06-07T12:43:22Z" level=info msg="server started at http://[::]:4443"
time="2022-06-07T12:43:29Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:29 +0000] \"POST /storage/v1/b?project=dummy&prettyPrint=false HTTP/1.1\" 200 135"
time="2022-06-07T12:43:29Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:29 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 456"
time="2022-06-07T12:43:29Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:29 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 526"
time="2022-06-07T12:43:33Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:33 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 526"
time="2022-06-07T12:43:33Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:33 +0000] \"GET /download/storage/v1/b/xxx/o/env-cerebro?alt=media&generation=1654605809095405 HTTP/1.1\" 200 4"
<!-- gh-comment-id:1148618855 --> @drehelis commented on GitHub (Jun 7, 2022): Tested #819 only, seems to work. ``` fake-gcs-server git:(3fdc10f) docker build -t local/fake-gcs-server . ... ... fake-gcs-server git:(3fdc10f) docker run --rm --name fake-gcs-server -p 4443:4443 local/fake-gcs-server -scheme http -public-host storage.gcs.127.0.0.1.nip.io:4443 time="2022-06-07T12:43:22Z" level=info msg="couldn't load any objects or buckets from \"/data\", starting empty" time="2022-06-07T12:43:22Z" level=info msg="server started at http://[::]:4443" time="2022-06-07T12:43:29Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:29 +0000] \"POST /storage/v1/b?project=dummy&prettyPrint=false HTTP/1.1\" 200 135" time="2022-06-07T12:43:29Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:29 +0000] \"POST /upload/storage/v1/b/xxx/o?uploadType=multipart HTTP/1.1\" 200 456" time="2022-06-07T12:43:29Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:29 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 526" time="2022-06-07T12:43:33Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:33 +0000] \"GET /storage/v1/b/xxx/o?projection=noAcl&prettyPrint=false HTTP/1.1\" 200 526" time="2022-06-07T12:43:33Z" level=info msg="172.17.0.1 - - [07/Jun/2022:12:43:33 +0000] \"GET /download/storage/v1/b/xxx/o/env-cerebro?alt=media&generation=1654605809095405 HTTP/1.1\" 200 4" ```
Author
Owner

@fsouza commented on GitHub (Jun 7, 2022):

Fixed via #819, which is included in 1.38.2.

Thanks for reporting and apologies for the disruption!

<!-- gh-comment-id:1148700353 --> @fsouza commented on GitHub (Jun 7, 2022): Fixed via #819, which is included in 1.38.2. Thanks for reporting and apologies for the disruption!
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/fake-gcs-server#134
No description provided.