starred/fake-gcs-server

Fork 0

mirror of https://github.com/fsouza/fake-gcs-server.git synced 2026-04-25 21:55:56 +03:00

[GH-ISSUE #1281] Resumable upload fails due to the "Location" header in the server's response #184

New issue

Closed

opened 2026-03-03 12:08:59 +03:00 by kerem · 6 comments

kerem commented

2026-03-03 12:08:59 +03:00

Owner

Originally created by @manoss96 on GitHub (Aug 7, 2023).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1281

First of all, thanks for creating this project, it's pretty cool!

As I was trying to perform a resumable upload via the GCS Python API, I noticed that even though I could create and initiate a resumable upload session without any issues, the program would stall while attempting to transfer the first chunk of bytes. Having dug into the code a little bit, I figured out it was due to the server's response to the resumable upload session initiation request, more specifically, due to the response's Location header. What happens is essentially this:

A POST request is made to https://127.0.0.1:4443/upload/storage/v1/b/bucket/o?uploadType=resumable&name={FILE_NAME}
The server answers by sending over a response whose Location header includes the initial request's URL, only this time it includes an upload_id. However, 127.0.0.1 within the original URL has been turned into 0.0.0.0. Thus, the header contains something like this: https://0.0.0.0:4443/upload/storage/v1/b/bucket/o?uploadType=resumable&name={FILE_NAME}&upload_id={UPLOAD_ID}.
Internally, the Python API uses this header in order to gain access to the upoad_id and completely replaces the original URL. Therefore, when it attempts to upload the first chunk of bytes, the request is sent to https://0.0.0.0:4443 instead of https://127.0.0.1:4443, causing it to stall.

I've tested this by manually replacing 0.0.0.0 with 127.0.0.1 and it works just fine.

Originally created by @manoss96 on GitHub (Aug 7, 2023). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1281 First of all, thanks for creating this project, it's pretty cool! As I was trying to perform a resumable upload via the GCS Python API, I noticed that even though I could create and initiate a resumable upload session without any issues, the program would stall while attempting to transfer the first chunk of bytes. Having dug into the code a little bit, I figured out it was due to the server's response to the resumable upload session initiation request, more specifically, due to the response's ``Location`` header. What happens is essentially this: 1. A POST request is made to ``https://127.0.0.1:4443/upload/storage/v1/b/bucket/o?uploadType=resumable&name={FILE_NAME}`` 2. The server answers by sending over a response whose ``Location`` header includes the initial request's URL, only this time it includes an ``upload_id``. **However**, ``127.0.0.1`` within the original URL has been turned into ``0.0.0.0``. Thus, the header contains something like this: ``https://0.0.0.0:4443/upload/storage/v1/b/bucket/o?uploadType=resumable&name={FILE_NAME}&upload_id={UPLOAD_ID}``. 3. Internally, the Python API uses this header in order to gain access to the ``upoad_id`` and completely replaces the original URL. Therefore, when it attempts to upload the first chunk of bytes, the request is sent to ``https://0.0.0.0:4443`` instead of ``https://127.0.0.1:4443``, causing it to stall. I've tested this by manually replacing ``0.0.0.0`` with ``127.0.0.1`` and it works just fine.

kerem closed this issue

2026-03-03 12:08:59 +03:00

kerem commented

2026-03-03 12:09:00 +03:00

Author

Owner

@fsouza commented on GitHub (Aug 12, 2023):

Can you share the command line you used to start the server and a snippet of code to reproduce the issue?

@fsouza commented on GitHub (Aug 12, 2023): Can you share the command line you used to start the server and a snippet of code to reproduce the issue?

kerem commented

2026-03-03 12:09:00 +03:00

Author

Owner

@manoss96 commented on GitHub (Aug 12, 2023):

@fsouza Sure, the server was started as such:

docker run -d -p 4443:4443 --mount type=bind,src=/test,dst=/data/bucket fsouza/fake-gcs-server:1.47.0

Then, a resumable upload was attempted via the Python API as such:

from io import BytesIO
from requests import Session
from google.cloud.storage import Client
from google.api_core.client_options import ClientOptions
from google.auth.credentials import AnonymousCredentials
from google.resumable_media.requests import ResumableUpload

# 1. Create HTTP session and GCS client.
session = Session()
session.verify = False

client = Client(
    credentials=AnonymousCredentials(),
    project="test",
    _http=session,
    client_options=ClientOptions(api_endpoint="https://127.0.0.1:4443"))

# 2. Create and initiate a resumable upload session.
rus = ResumableUpload(
    upload_url="https://127.0.0.1:4443/upload/storage/v1/b/bucket/o?uploadType=resumable&name=testfile.txt",
    chunk_size=1024*1024)

with BytesIO() as buffer:
    rus.initiate(
        transport=client._http,
        content_type='application/octet-stream',
        stream=buffer,
        stream_final=False,
        metadata=None)

    # 3. Try writing some bytes to buffer and uploading them to server.
    buffer.write(b"x" * (1024 * 1024))
    buffer.seek(0)
   
    # This is where the program stalls.
    rus.transmit_next_chunk(transport=client._http)

Just make sure to pip install google-cloud-storage==2.10.0 on Python 3.10 and you're good. Also, I should note that this was attempted on a Windows 10 system. I'm only mentioning this because I believe that Linux systems treat 0.0.0.0 differently (https://stackoverflow.com/a/21057696/19957744). It might work just fine on a Linux machine, then again, I haven't tested it.

@manoss96 commented on GitHub (Aug 12, 2023): @fsouza Sure, the server was started as such: ```sh docker run -d -p 4443:4443 --mount type=bind,src=/test,dst=/data/bucket fsouza/fake-gcs-server:1.47.0 ``` Then, a resumable upload was attempted via the Python API as such: ```python from io import BytesIO from requests import Session from google.cloud.storage import Client from google.api_core.client_options import ClientOptions from google.auth.credentials import AnonymousCredentials from google.resumable_media.requests import ResumableUpload # 1. Create HTTP session and GCS client. session = Session() session.verify = False client = Client( credentials=AnonymousCredentials(), project="test", _http=session, client_options=ClientOptions(api_endpoint="https://127.0.0.1:4443")) # 2. Create and initiate a resumable upload session. rus = ResumableUpload( upload_url="https://127.0.0.1:4443/upload/storage/v1/b/bucket/o?uploadType=resumable&name=testfile.txt", chunk_size=1024*1024) with BytesIO() as buffer: rus.initiate( transport=client._http, content_type='application/octet-stream', stream=buffer, stream_final=False, metadata=None) # 3. Try writing some bytes to buffer and uploading them to server. buffer.write(b"x" * (1024 * 1024)) buffer.seek(0) # This is where the program stalls. rus.transmit_next_chunk(transport=client._http) ``` Just make sure to ``pip install google-cloud-storage==2.10.0`` on Python 3.10 and you're good. Also, I should note that this was attempted on a Windows 10 system. I'm only mentioning this because I believe that Linux systems treat ``0.0.0.0`` differently (https://stackoverflow.com/a/21057696/19957744). It might work just fine on a Linux machine, then again, I haven't tested it.

kerem commented

2026-03-03 12:09:00 +03:00

Author

Owner

@manuteleco commented on GitHub (Aug 19, 2023):

I stumbled upon the same issue. Turns out you have to pass the -external-url parameter when running fake-gcs-server. In your example I believe it should be -external-url https://127.0.0.1:4443.

For further detail, this is how the Location header is being built:

and here some documentation about the -external-url attribute:

@manuteleco commented on GitHub (Aug 19, 2023): I stumbled upon the same issue. Turns out you have to pass the `-external-url` parameter when running `fake-gcs-server`. In your example I believe it should be `-external-url https://127.0.0.1:4443`. For further detail, this is how the `Location` header is being built: * https://github.com/fsouza/fake-gcs-server/blob/87dfef0e58609b2768fd6493091d78cd56b2e0d7/fakestorage/upload.go#L436-L442 * https://github.com/fsouza/fake-gcs-server/blob/87dfef0e58609b2768fd6493091d78cd56b2e0d7/fakestorage/server.go#L440-L449 and here some documentation about the `-external-url` attribute: * https://github.com/fsouza/fake-gcs-server/blob/87dfef0e58609b2768fd6493091d78cd56b2e0d7/internal/config/config.go#L75 * https://github.com/fsouza/fake-gcs-server/blob/87dfef0e58609b2768fd6493091d78cd56b2e0d7/fakestorage/server.go#L89-L93

kerem commented

2026-03-03 12:09:00 +03:00

Author

Owner

@manoss96 commented on GitHub (Aug 20, 2023):

@manuteleco Tested your solution and it works perfect, thanks! @fsouza Want me to close this issue or are you planning to replace 0.0.0.0 with 127.0.0.1 no matter the external-url param?

@manoss96 commented on GitHub (Aug 20, 2023): @manuteleco Tested your solution and it works perfect, thanks! @fsouza Want me to close this issue or are you planning to replace ``0.0.0.0`` with ``127.0.0.1`` no matter the ``external-url`` param?

kerem commented

2026-03-03 12:09:00 +03:00

Author

Owner

@fsouza commented on GitHub (Sep 9, 2023):

@manoss96 I think we cannot make that replacement because not everyone uses it on localhost. Also, I'm curious, is the Python client not able to send request to https://0.0.0.0:4443 (I know it's not RFC compliant, but in my experience this just works heh)

@fsouza commented on GitHub (Sep 9, 2023): @manoss96 I think we cannot make that replacement because not everyone uses it on localhost. Also, I'm curious, is the Python client not able to send request to https://0.0.0.0:4443 (I know it's not RFC compliant, but in my experience this just works heh)

kerem commented

2026-03-03 12:09:00 +03:00

Author

Owner

@manoss96 commented on GitHub (Sep 9, 2023):

@fsouza I think it's more of an OS thing rather than a Python thing, but I'm not sure. Anyways, I've managed to get it to work by following @manuteleco 's instructions. Thefore, I am closing this issue.

@manoss96 commented on GitHub (Sep 9, 2023): @fsouza I think it's more of an OS thing rather than a Python thing, but I'm not sure. Anyways, I've managed to get it to work by following @manuteleco 's instructions. Thefore, I am closing this issue.

kerem referenced this issue

2026-03-03 12:29:54 +03:00

[PR #184] [MERGED] build(deps): bump google.golang.org/api from 0.18.0 to 0.19.0 #415