starred/fake-gcs-server

Fork 0

mirror of https://github.com/fsouza/fake-gcs-server.git synced 2026-04-25 13:45:52 +03:00

[GH-ISSUE #1149] Resuming a resumable upload from GCS C++ client does not work: fake-gcs-server rejects “Content-Range: bytes /” #169

New issue

Open

opened 2026-03-03 12:08:52 +03:00 by kerem · 4 comments

kerem commented

2026-03-03 12:08:52 +03:00

Owner

Originally created by @QrczakMK on GitHub (May 5, 2023).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1149

When resuming a resumable upload using gcs::RestoreResumableUploadSession(), the C++ client sends Content-Range: bytes */*:
github.com/googleapis/google-cloud-cpp@1178ef9bbb/google/cloud/storage/internal/rest_client.cc (L699)

fake-gcs-server rejects this: github.com/fsouza/fake-gcs-server@e2d2de125a/fakestorage/upload.go (L594)

AFAIK this syntax is valid.

Originally created by @QrczakMK on GitHub (May 5, 2023). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1149 When resuming a resumable upload using `gcs::RestoreResumableUploadSession()`, the C++ client sends `Content-Range: bytes */*`: https://github.com/googleapis/google-cloud-cpp/blob/1178ef9bbb0257e940dc2d9af475802d8bb37038/google/cloud/storage/internal/rest_client.cc#L699 fake-gcs-server rejects this: https://github.com/fsouza/fake-gcs-server/blob/e2d2de125a3c14829c3f063681f37aa7e8095444/fakestorage/upload.go#L594 AFAIK this syntax is valid.

kerem added the

bug

label

2026-03-03 12:08:52 +03:00

kerem commented

2026-03-03 12:08:53 +03:00

Author

Owner

@fsouza commented on GitHub (May 6, 2023):

@QrczakMK thanks for reporting, can you give #1150 a shot?

@fsouza commented on GitHub (May 6, 2023): @QrczakMK thanks for reporting, can you give #1150 a shot?

kerem commented

2026-03-03 12:08:53 +03:00

Author

Owner

@QrczakMK commented on GitHub (May 7, 2023):

It helps in the sense that the request syntax is no longer rejected, but a resumable upload still fails. The same test succeeds against the real GCS server.

I don’t have that test open sourced (it is in Google), but here is what happens. I begin writing to a new object with gcs::NewResumableUploadSession():

… \"POST /upload/storage/v1/b/test-bucket/o?uploadType=resumable&name=riegeli-test-thcivcvtyawjewoe-object HTTP/1.1\" 200 423"

I write the first piece, capture resumable_session_id, suspend writing, and open a new writing session with gcs::RestoreResumableUploadSession(resumable_session_id) and gcs::DisableCrc32cChecksum(true):

… \"PUT /upload/resumable/9e63eeafa4ee28ae17ac7ce5a84cff9a HTTP/1.1\" 200 598"

But gcs::ObjectWriteStream::IsOpen() is false and gcs::ObjectWriteStream::metadata().ok() is true, which according to the documentation indicates that the previous upload was finished: github.com/googleapis/google-cloud-cpp@1178ef9bbb/google/cloud/storage/object_write_stream.h (L175-L192).

@QrczakMK commented on GitHub (May 7, 2023): It helps in the sense that the request syntax is no longer rejected, but a resumable upload still fails. The same test succeeds against the real GCS server. I don’t have that test open sourced (it is in Google), but here is what happens. I begin writing to a new object with `gcs::NewResumableUploadSession()`: ``` … \"POST /upload/storage/v1/b/test-bucket/o?uploadType=resumable&name=riegeli-test-thcivcvtyawjewoe-object HTTP/1.1\" 200 423" ``` I write the first piece, capture `resumable_session_id`, suspend writing, and open a new writing session with `gcs::RestoreResumableUploadSession(resumable_session_id)` and `gcs::DisableCrc32cChecksum(true)`: ``` … \"PUT /upload/resumable/9e63eeafa4ee28ae17ac7ce5a84cff9a HTTP/1.1\" 200 598" ``` But `gcs::ObjectWriteStream::IsOpen()` is `false` and `gcs::ObjectWriteStream::metadata().ok()` is `true`, which according to the documentation indicates that the previous upload was finished: https://github.com/googleapis/google-cloud-cpp/blob/1178ef9bbb0257e940dc2d9af475802d8bb37038/google/cloud/storage/object_write_stream.h#L175-L192.

kerem commented

2026-03-03 12:08:53 +03:00

Author

Owner

@fsouza commented on GitHub (May 7, 2023):

Gotcha, thanks for providing additional info. I'll try to create a reproducer.

@fsouza commented on GitHub (May 7, 2023): Gotcha, thanks for providing additional info. I'll try to create a reproducer.

kerem commented

2026-03-03 12:08:54 +03:00

Author

Owner

@QrczakMK commented on GitHub (May 7, 2023):

Here is a strace log of socket communication (of a similar but different run — this is a randomized test):

sendto(12, "POST /storage/v1/b?project= HTTP/1.1\r\nHost: localhost:27454\r\nUser-Agent: gcloud-cpp/v2.11.0-rc.g3+piper (Clang-9999.0.0; noex) libcurl/8.0.1-DEV BoringSSL zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.41.0\r\nAccept: */*\r\nx-goog-api-client: gl-cpp/Clang-9999.0.0-noex-2017 gccl/v2.11.0-rc.g3+piper\r\ncontent-type: application/json\r\nContent-Length: 52\r\n\r\n", 354, MSG_NOSIGNAL, NULL, 0) = 354
sendto(12, "{\"defaultEventBasedHold\":false,\"name\":\"test-bucket\"}", 52, MSG_NOSIGNAL, NULL, 0) = 52
recvfrom(12, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nDate: Sun, 07 May 2023 16:29:46 GMT\r\nContent-Length: 156\r\n\r\n{\"kind\":\"storage#bucket\",\"id\":\"test-bucket\",\"name\":\"test-bucket\",\"versioning\":{},\"timeCreated\":\"2023-05-07T09:29:46.703865-07:00\",\"location\":\"US-CENTRAL1\"}\n", 131072, 0, NULL, NULL) = 265
sendto(12, "POST /upload/storage/v1/b/test-bucket/o?uploadType=resumable&name=riegeli-test-ihnwgelmfdmhgvgu-object HTTP/1.1\r\nHost: localhost:27454\r\nUser-Agent: gcloud-cpp/v2.11.0-rc.g3+piper (Clang-9999.0.0; noex) libcurl/8.0.1-DEV BoringSSL zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.41.0\r\nAccept: */*\r\nx-goog-api-client: gl-cpp/Clang-9999.0.0-noex-2017 gccl/v2.11.0-rc.g3+piper\r\ncontent-type: application/json; charset=UTF-8\r\nContent-Length: 0\r\n\r\n", 443, MSG_NOSIGNAL, NULL, 0) = 443
recvfrom(12, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nLocation: http://localhost:27454/upload/resumable/353fee57963c299593b3504a00aecdef\r\nDate: Sun, 07 May 2023 16:29:46 GMT\r\nContent-Length: 423\r\n\r\n{\"kind\":\"storage#object\",\"name\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"id\":\"test-bucket/riegeli-test-ihnwgelmfdmhgvgu-object\",\"bucket\":\"test-bucket\",\"size\":\"0\",\"acl\":[{\"bucket\":\"test-bucket\",\"entity\":\"projectOwner\",\"object\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"projectTeam\":{},\"role\":\"OWNER\"}],\"timeCreated\":\"0001-01-01T00:00:00Z\",\"timeDeleted\":\"0001-01-01T00:00:00Z\",\"updated\":\"0001-01-01T00:00:00Z\",\"generation\":\"0\"}\n", 65536, 0, NULL, NULL) = 616
sendto(12, "PUT /upload/resumable/353fee57963c299593b3504a00aecdef HTTP/1.1\r\nHost: localhost:27454\r\nUser-Agent: gcloud-cpp/v2.11.0-rc.g3+piper (Clang-9999.0.0; noex) libcurl/8.0.1-DEV BoringSSL zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.41.0\r\nAccept: */*\r\nx-goog-api-client: gl-cpp/Clang-9999.0.0-noex-2017 gccl/v2.11.0-rc.g3+piper\r\ncontent-type: application/octet-stream\r\ncontent-range: bytes */*\r\nContent-Length: 0\r\n\r\n", 414, MSG_NOSIGNAL, NULL, 0) = 414
recvfrom(12, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nRange: bytes=0-0\r\nDate: Sun, 07 May 2023 16:29:46 GMT\r\nContent-Length: 598\r\n\r\n{\"kind\":\"storage#object\",\"name\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"id\":\"test-bucket/riegeli-test-ihnwgelmfdmhgvgu-object\",\"bucket\":\"test-bucket\",\"size\":\"0\",\"contentType\":\"application/octet-stream\",\"crc32c\":\"AAAAAA==\",\"acl\":[{\"bucket\":\"test-bucket\",\"entity\":\"projectOwner\",\"object\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"projectTeam\":{},\"role\":\"OWNER\"}],\"md5Hash\":\"1B2M2Y8AsgTpgAmY7PhCfg==\",\"etag\":\"\\\"1B2M2Y8AsgTpgAmY7PhCfg==\\\"\",\"timeCreated\":\"2023-05-07T09:29:46.712809-07:00\",\"timeDeleted\":\"0001-01-01T00:00:00Z\",\"updated\":\"2023-05-07T09:29:46.712811-07:00\",\"generation\":\"1683476986712812\"}\n", 65536, 0, NULL, NULL) = 725

I am not familiar with the protocol so I am not sure what it should be there.

No data were actually written because the length did not reach 256 KiB. Still it should resume starting from 0.

I don’t have an easy comparison with a real GCS server because in that case the communication goes via https.

@QrczakMK commented on GitHub (May 7, 2023): Here is a strace log of socket communication (of a similar but different run — this is a randomized test): ``` sendto(12, "POST /storage/v1/b?project= HTTP/1.1\r\nHost: localhost:27454\r\nUser-Agent: gcloud-cpp/v2.11.0-rc.g3+piper (Clang-9999.0.0; noex) libcurl/8.0.1-DEV BoringSSL zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.41.0\r\nAccept: */*\r\nx-goog-api-client: gl-cpp/Clang-9999.0.0-noex-2017 gccl/v2.11.0-rc.g3+piper\r\ncontent-type: application/json\r\nContent-Length: 52\r\n\r\n", 354, MSG_NOSIGNAL, NULL, 0) = 354 sendto(12, "{\"defaultEventBasedHold\":false,\"name\":\"test-bucket\"}", 52, MSG_NOSIGNAL, NULL, 0) = 52 recvfrom(12, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nDate: Sun, 07 May 2023 16:29:46 GMT\r\nContent-Length: 156\r\n\r\n{\"kind\":\"storage#bucket\",\"id\":\"test-bucket\",\"name\":\"test-bucket\",\"versioning\":{},\"timeCreated\":\"2023-05-07T09:29:46.703865-07:00\",\"location\":\"US-CENTRAL1\"}\n", 131072, 0, NULL, NULL) = 265 sendto(12, "POST /upload/storage/v1/b/test-bucket/o?uploadType=resumable&name=riegeli-test-ihnwgelmfdmhgvgu-object HTTP/1.1\r\nHost: localhost:27454\r\nUser-Agent: gcloud-cpp/v2.11.0-rc.g3+piper (Clang-9999.0.0; noex) libcurl/8.0.1-DEV BoringSSL zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.41.0\r\nAccept: */*\r\nx-goog-api-client: gl-cpp/Clang-9999.0.0-noex-2017 gccl/v2.11.0-rc.g3+piper\r\ncontent-type: application/json; charset=UTF-8\r\nContent-Length: 0\r\n\r\n", 443, MSG_NOSIGNAL, NULL, 0) = 443 recvfrom(12, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nLocation: http://localhost:27454/upload/resumable/353fee57963c299593b3504a00aecdef\r\nDate: Sun, 07 May 2023 16:29:46 GMT\r\nContent-Length: 423\r\n\r\n{\"kind\":\"storage#object\",\"name\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"id\":\"test-bucket/riegeli-test-ihnwgelmfdmhgvgu-object\",\"bucket\":\"test-bucket\",\"size\":\"0\",\"acl\":[{\"bucket\":\"test-bucket\",\"entity\":\"projectOwner\",\"object\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"projectTeam\":{},\"role\":\"OWNER\"}],\"timeCreated\":\"0001-01-01T00:00:00Z\",\"timeDeleted\":\"0001-01-01T00:00:00Z\",\"updated\":\"0001-01-01T00:00:00Z\",\"generation\":\"0\"}\n", 65536, 0, NULL, NULL) = 616 sendto(12, "PUT /upload/resumable/353fee57963c299593b3504a00aecdef HTTP/1.1\r\nHost: localhost:27454\r\nUser-Agent: gcloud-cpp/v2.11.0-rc.g3+piper (Clang-9999.0.0; noex) libcurl/8.0.1-DEV BoringSSL zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.41.0\r\nAccept: */*\r\nx-goog-api-client: gl-cpp/Clang-9999.0.0-noex-2017 gccl/v2.11.0-rc.g3+piper\r\ncontent-type: application/octet-stream\r\ncontent-range: bytes */*\r\nContent-Length: 0\r\n\r\n", 414, MSG_NOSIGNAL, NULL, 0) = 414 recvfrom(12, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nRange: bytes=0-0\r\nDate: Sun, 07 May 2023 16:29:46 GMT\r\nContent-Length: 598\r\n\r\n{\"kind\":\"storage#object\",\"name\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"id\":\"test-bucket/riegeli-test-ihnwgelmfdmhgvgu-object\",\"bucket\":\"test-bucket\",\"size\":\"0\",\"contentType\":\"application/octet-stream\",\"crc32c\":\"AAAAAA==\",\"acl\":[{\"bucket\":\"test-bucket\",\"entity\":\"projectOwner\",\"object\":\"riegeli-test-ihnwgelmfdmhgvgu-object\",\"projectTeam\":{},\"role\":\"OWNER\"}],\"md5Hash\":\"1B2M2Y8AsgTpgAmY7PhCfg==\",\"etag\":\"\\\"1B2M2Y8AsgTpgAmY7PhCfg==\\\"\",\"timeCreated\":\"2023-05-07T09:29:46.712809-07:00\",\"timeDeleted\":\"0001-01-01T00:00:00Z\",\"updated\":\"2023-05-07T09:29:46.712811-07:00\",\"generation\":\"1683476986712812\"}\n", 65536, 0, NULL, NULL) = 725 ``` I am not familiar with the protocol so I am not sure what it should be there. No data were actually written because the length did not reach 256 KiB. Still it should resume starting from 0. I don’t have an easy comparison with a real GCS server because in that case the communication goes via https.

kerem referenced this issue

2026-03-03 12:29:50 +03:00

[PR #169] [MERGED] build(deps): bump golang from 1.13.6 to 1.13.7 #400