starred/fake-gcs-server

Fork 0

mirror of https://github.com/fsouza/fake-gcs-server.git synced 2026-04-25 21:55:56 +03:00

[GH-ISSUE #371] Invalid Request when calling create_file with Ruby GCS SDK #2222

New issue

Closed

opened 2026-03-15 18:05:52 +03:00 by kerem · 4 comments

kerem commented

2026-03-15 18:05:52 +03:00

Owner

Originally created by @mheffner on GitHub (Nov 26, 2020).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/371

When using the googe-cloud-storage Ruby gem, version 1.29.1, to try and upload a file to fake-gcs-server we are seeing an Invalid Request error. This is the request as seen from the fake-gcs-server logs:

nf-originci-gcs       | time="2020-11-26T15:17:57Z" level=info msg="172.31.0.9 - - [26/Nov/2020:15:17:57 +0000] \"POST /upload/storage/v1/b/deploy-manifests/o?name=my-key HTTP/1.1\" 400 19"

It would appear that we are hitting this error case:
github.com/fsouza/fake-gcs-server@e8bd4c7f4a/fakestorage/upload.go (L71)
which makes sense as we are not using a policy nor is there an uploadType POST param.

I can reproduce this with the following code:

require "google/cloud/storage"

storage = Google::Cloud::Storage.anonymous(endpoint: "http://gcs:4443/")
bucket = storage.bucket("deploy-manifests", skip_lookup: true)

data = StringIO.new('{"hi":"bye"}')

r = bucket.create_file(data, "my-key", content_type: "application/json")

This will throw an exception in:

/usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/http_command.rb:228:in `check_status': Invalid request (Google::Apis::ClientError)
	from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/api_command.rb:119:in `check_status'
	from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/http_command.rb:194:in `process_response'
	from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/upload.rb:174:in `process_response'
	from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/upload.rb:250:in `execute_once'

If I run a netcat locally and point the endpoint at it, this is the full initial POST request:

POST /upload/storage/v1/b/deploy-manifests/o?name=my-key HTTP/1.1
User-Agent: gcloud-ruby/1.29.1 google-api-ruby-client/0.48.0 Linux/5.8.15-101.fc31.x86_64 (gzip)
X-Goog-Api-Client: gl-ruby/2.7.2 gdcl/0.48.0
Content-Type: application/json
x-goog-api-client: gl-ruby/2.7.2 gccl/1.29.1
Accept-Encoding: gzip
X-Goog-Upload-Protocol: resumable
X-Goog-Upload-Command: start
X-Goog-Upload-Header-Content-Length: 12
X-Goog-Upload-Header-Content-Type: application/json
Accept: */*
Date: Thu, 26 Nov 2020 14:04:58 GMT
Content-Length: 34
Host: 172.31.0.9:12345

{"contentType":"application/json"}

This doesn't seem to follow the documented API, so I'm not exactly sure how it's supposed to work. Have others seen this before?

Cheers!

Originally created by @mheffner on GitHub (Nov 26, 2020). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/371 When using the [googe-cloud-storage](https://rubygems.org/gems/google-cloud-storage/versions/0.20.0) Ruby gem, version 1.29.1, to try and upload a file to fake-gcs-server we are seeing an Invalid Request error. This is the request as seen from the fake-gcs-server logs: ``` nf-originci-gcs | time="2020-11-26T15:17:57Z" level=info msg="172.31.0.9 - - [26/Nov/2020:15:17:57 +0000] \"POST /upload/storage/v1/b/deploy-manifests/o?name=my-key HTTP/1.1\" 400 19" ``` It would appear that we are hitting this error case: https://github.com/fsouza/fake-gcs-server/blob/e8bd4c7f4a67f6c06a70e73be879cc981d4c84bc/fakestorage/upload.go#L71 which makes sense as we are not using a policy nor is there an `uploadType` POST param. I can reproduce this with the following code: ```ruby require "google/cloud/storage" storage = Google::Cloud::Storage.anonymous(endpoint: "http://gcs:4443/") bucket = storage.bucket("deploy-manifests", skip_lookup: true) data = StringIO.new('{"hi":"bye"}') r = bucket.create_file(data, "my-key", content_type: "application/json") ``` This will throw an exception in: ``` /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/http_command.rb:228:in `check_status': Invalid request (Google::Apis::ClientError) from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/api_command.rb:119:in `check_status' from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/http_command.rb:194:in `process_response' from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/upload.rb:174:in `process_response' from /usr/local/bundle/ruby/2.7.0/gems/google-api-client-0.48.0/lib/google/apis/core/upload.rb:250:in `execute_once' ``` If I run a netcat locally and point the endpoint at it, this is the full initial POST request: ``` POST /upload/storage/v1/b/deploy-manifests/o?name=my-key HTTP/1.1 User-Agent: gcloud-ruby/1.29.1 google-api-ruby-client/0.48.0 Linux/5.8.15-101.fc31.x86_64 (gzip) X-Goog-Api-Client: gl-ruby/2.7.2 gdcl/0.48.0 Content-Type: application/json x-goog-api-client: gl-ruby/2.7.2 gccl/1.29.1 Accept-Encoding: gzip X-Goog-Upload-Protocol: resumable X-Goog-Upload-Command: start X-Goog-Upload-Header-Content-Length: 12 X-Goog-Upload-Header-Content-Type: application/json Accept: */* Date: Thu, 26 Nov 2020 14:04:58 GMT Content-Length: 34 Host: 172.31.0.9:12345 {"contentType":"application/json"} ``` This doesn't seem to follow the documented API, so I'm not exactly sure how it's supposed to work. Have others seen this before? Cheers!

kerem closed this issue

2026-03-15 18:05:57 +03:00

kerem commented

2026-03-15 18:06:03 +03:00

Author

Owner

@mheffner commented on GitHub (Nov 28, 2020):

Digging further on this, the implementation stems from the ResumableUpload protocol here: https://github.com/googleapis/google-api-ruby-client/blob/master/lib/google/apis/core/upload.rb

I did not see it documented for GCS, but it does seem to closely mirror the protocol documented for the Photos API: https://developers.google.com/photos/library/guides/resumable-uploads

I was able to spike enough of an server-side implementation to get the Ruby SDK working, though it is by far a full version. I'll push a PR if it is useful for others.

@mheffner commented on GitHub (Nov 28, 2020): Digging further on this, the implementation stems from the ResumableUpload protocol here: https://github.com/googleapis/google-api-ruby-client/blob/master/lib/google/apis/core/upload.rb I did not see it documented for GCS, but it does seem to closely mirror the protocol documented for the Photos API: https://developers.google.com/photos/library/guides/resumable-uploads I was able to spike enough of an server-side implementation to get the Ruby SDK working, though it is by far a full version. I'll push a PR if it is useful for others.

kerem commented

2026-03-15 18:06:09 +03:00

Author

Owner

@mheffner commented on GitHub (Nov 28, 2020):

PR here: https://github.com/mheffner/fake-gcs-server/pull/1. Not pulled on this repo as it requires some lint fixes first.

@mheffner commented on GitHub (Nov 28, 2020): PR here: https://github.com/mheffner/fake-gcs-server/pull/1. Not pulled on this repo as it requires some lint fixes first.

kerem commented

2026-03-15 18:06:14 +03:00

Author

Owner

@fsouza commented on GitHub (Nov 30, 2020):

There are other undocumented things that we support here, so this should be fine 🙈 I merged your other PR and cleaned up the main branch, please feel free to send your PR :D

Thank you very much for reporting and investigating it!

@fsouza commented on GitHub (Nov 30, 2020): There are other undocumented things that we support here, so this should be fine 🙈 I merged your other PR and cleaned up the main branch, please feel free to send your PR :D Thank you very much for reporting and investigating it!

kerem commented

2026-03-15 18:06:19 +03:00

Author

Owner

@mheffner commented on GitHub (Nov 30, 2020):

Thanks @fsouza , I'll clean it up and submit it soon.

I did finally find this, though it doesn't provide a lot more context. 😆 https://github.com/googleapis/google-api-ruby-client/issues/311

@mheffner commented on GitHub (Nov 30, 2020): Thanks @fsouza , I'll clean it up and submit it soon. I did finally find this, though it doesn't provide a lot more context. :laughing: https://github.com/googleapis/google-api-ruby-client/issues/311