[GH-ISSUE #1080] Minimum chunk size not enforced #164

New issue

Open

opened 2026-03-03 12:08:51 +03:00 by kerem · 1 comment

kerem commented

2026-03-03 12:08:51 +03:00

Owner

Originally created by @ianhomer on GitHub (Feb 21, 2023).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1080

We see a different behaviour on fake GCS server when chunk size is below the minimum of 262144 - https://cloud.google.com/storage/docs/performing-resumable-uploads

On fake GCS

curl -i -X PUT --data-binary "@$FILE_NAME" \
              -H "Content-Length: 500"               \
              -H "Content-Range: bytes 0-499/700000"    \
              "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID"

accepts the partial upload and responds status code 308 with

{"kind":"storage#object","name":"my-upload-1.txt",... }

On real GCS

curl --http1.1 -v -i -X PUT --data-binary "@$FILE_NAME" \
              -H "Content-Length: 500"               \
              -H "Content-Range: bytes 0-499/700000"    \
              "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID"

we get a 400 with response

Invalid request.  The number of bytes uploaded is required to be equal or greater than 262144, 
except for the final request (it's recommended to be the exact multiple of 262144).  
The received request contained 500 bytes, which does not meet this requirement.

When chunk size is multiple of 262144 both behave in a similar manner.

The small chunk size is handy for some test scenarios, however since it does not match the behaviour of real GCS, it masks a failure scenario that would be good to test with fake gcs.

Should fake GCS mirror this real GCS behaviour where a minimum chunk size is enforced?

Originally created by @ianhomer on GitHub (Feb 21, 2023). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1080 We see a different behaviour on fake GCS server when chunk size is below the minimum of 262144 - https://cloud.google.com/storage/docs/performing-resumable-uploads On fake GCS ``` curl -i -X PUT --data-binary "@$FILE_NAME" \ -H "Content-Length: 500" \ -H "Content-Range: bytes 0-499/700000" \ "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID" ``` accepts the partial upload and responds status code 308 with ``` {"kind":"storage#object","name":"my-upload-1.txt",... } ``` On real GCS ``` curl --http1.1 -v -i -X PUT --data-binary "@$FILE_NAME" \ -H "Content-Length: 500" \ -H "Content-Range: bytes 0-499/700000" \ "$UPLOAD_URL/o?uploadType=resumable&name=$FILE_NAME&upload_id=$UPLOAD_ID" ``` we get a 400 with response ``` Invalid request. The number of bytes uploaded is required to be equal or greater than 262144, except for the final request (it's recommended to be the exact multiple of 262144). The received request contained 500 bytes, which does not meet this requirement. ``` When chunk size is multiple of 262144 both behave in a similar manner. The small chunk size is handy for some test scenarios, however since it does not match the behaviour of real GCS, it masks a failure scenario that would be good to test with fake gcs. Should fake GCS mirror this real GCS behaviour where a minimum chunk size is enforced?

kerem added the

bug

compatibility-issue

labels

2026-03-03 12:08:51 +03:00

kerem commented

2026-03-03 12:08:52 +03:00

Author

Owner

@fsouza commented on GitHub (Feb 21, 2023):

Hey @ianhomer, thanks for reporting. We should definitely mirror GCS' behavior.

@fsouza commented on GitHub (Feb 21, 2023): Hey @ianhomer, thanks for reporting. We should definitely mirror GCS' behavior.

kerem referenced this issue

2026-03-03 12:29:48 +03:00

[PR #164] [MERGED] build(deps): bump google-cloud-storage from 1.24.1 to 1.25.0 in /examples/python #393