starred/fake-gcs-server

Fork 0

mirror of https://github.com/fsouza/fake-gcs-server.git synced 2026-04-25 13:45:52 +03:00

[GH-ISSUE #1202] Issue with PublicHost and Golang client #173

New issue

Open

opened 2026-03-03 12:08:55 +03:00 by kerem · 6 comments

kerem commented

2026-03-03 12:08:55 +03:00

Owner

Originally created by @ivanovaleksey on GitHub (Jun 7, 2023).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1202

Hey!

Thank you for the great GCS emulator!
I am using this server as a local storage while working with DuckDB.

It has a feature working with files in S3 storages directly https://duckdb.org/docs/guides/import/s3_import
To use it I need to configure endpoint SET s3_endpoint='some-host';
As far as I understand that should be the same host that is specified in public-host.

I just picked up value for public host from source code comment, i.e. storage.gcs.127.0.0.1.nip.io:4443
It works fine when I run fake-gcs-server as a binary but fails when I run it within Golang unit-test.
I took this example https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/example_test.go#LL41C13-L41C13 and added PublicHost: "storage.gcs.127.0.0.1.nip.io:8081" and then got object not found error.

After some debugging I realized that the issue is probably here https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/server.go#L418
r.Host was storage.googleapis.com while s.publicHost was storage.gcs.127.0.0.1.nip.io:8081.

I got it working by initialing GCS client manually

client, err := storage.NewClient(context.Background(), option.WithEndpoint("http://storage.gcs.127.0.0.1.nip.io:8081/storage/v1/"))

and specifying Scheme = "http"

I wonder if it's intended behaviour? My impression was that (*Server).Client() method should have handled this automatically as it was stated in the comment

Client returns a GCS client configured to talk to the server.

This issue might be related to #982

Originally created by @ivanovaleksey on GitHub (Jun 7, 2023). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1202 Hey! Thank you for the great GCS emulator! I am using this server as a local storage while working with DuckDB. It has a feature working with files in S3 storages directly https://duckdb.org/docs/guides/import/s3_import To use it I need to configure endpoint `SET s3_endpoint='some-host';` As far as I understand that should be the same host that is specified in `public-host`. I just picked up value for public host from source code comment, i.e. `storage.gcs.127.0.0.1.nip.io:4443` It works fine when I run `fake-gcs-server` as a binary but fails when I run it within Golang unit-test. I took this example https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/example_test.go#LL41C13-L41C13 and added `PublicHost: "storage.gcs.127.0.0.1.nip.io:8081"` and then got `object not found` error. After some debugging I realized that the issue is probably here https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/server.go#L418 `r.Host` was `storage.googleapis.com` while `s.publicHost` was `storage.gcs.127.0.0.1.nip.io:8081`. I got it working by initialing GCS client manually ``` client, err := storage.NewClient(context.Background(), option.WithEndpoint("http://storage.gcs.127.0.0.1.nip.io:8081/storage/v1/")) ``` and specifying `Scheme = "http"` I wonder if it's intended behaviour? My impression was that `(*Server).Client()` method should have handled this automatically as it was stated in the comment > Client returns a GCS client configured to talk to the server. This issue might be related to #982

kerem added the

bug

label

2026-03-03 12:08:55 +03:00

kerem commented

2026-03-03 12:08:56 +03:00

Author

Owner

@fsouza commented on GitHub (Jun 7, 2023):

I wonder if it's intended behaviour? My impression was that (*Server).Client() method should have handled this automatically as it was stated in the comment

Hmm, that's interesting. (*Server).Client() should definitely handle this correctly. Do you have an example of a unit test that fails with that error? We should add those tests to fake-gcs-server itself and fix the created client to make sure it either sets all headers or accepts any requests.

@fsouza commented on GitHub (Jun 7, 2023): > I wonder if it's intended behaviour? My impression was that (*Server).Client() method should have handled this automatically as it was stated in the comment Hmm, that's interesting. `(*Server).Client()` should definitely handle this correctly. Do you have an example of a unit test that fails with that error? We should add those tests to fake-gcs-server itself and fix the created client to make sure it either sets all headers or accepts any requests.

kerem commented

2026-03-03 12:08:56 +03:00

Author

Owner

@ivanovaleksey commented on GitHub (Jun 7, 2023):

@fsouza thank you for the reply!
Please see my commit here

I am not sure what is happening but in ExampleServer_with_public_host function panic doesn't really work. If you debug it you will see that err != nil but panic just handle it silently.
I changed it to log.Fatal and also added another "proper" test. There panic actually works as expected.

Weird thing is that if I copy ExampleServer_with_public_host to my actual project it fails. Maybe something is wrong with my environment? Can't even imagine what it could be.

Could you please run it yourself to see if it works/fails?

P.S.

I guess it has something do to with it being example. This doesn't panic

func ExamplePanic() {
	panic("error")
	// Output: whatever
}

Still don't understand why it fails in some files and not in the others.
If I put the code above to internal/backend/backend_test.go it fails but if I put it any *_test.go file under fakestorage directory it just runs silently

@ivanovaleksey commented on GitHub (Jun 7, 2023): @fsouza thank you for the reply! Please see my commit [here](https://github.com/ivanovaleksey/fake-gcs-server/commit/7627dd8b276c58b6157cd22b9c5e228caa9045bf) I am not sure what is happening but in `ExampleServer_with_public_host` function `panic` doesn't really work. If you debug it you will see that `err != nil` but panic just handle it silently. I changed it to `log.Fatal` and also added another "proper" test. There `panic` actually works as expected. Weird thing is that if I copy `ExampleServer_with_public_host` to my actual project it fails. Maybe something is wrong with my environment? Can't even imagine what it could be. Could you please run it yourself to see if it works/fails? P.S. I guess it has something do to with it being example. This doesn't panic ``` func ExamplePanic() { panic("error") // Output: whatever } ``` Still don't understand why it fails in some files and not in the others. If I put the code above to `internal/backend/backend_test.go` it fails but if I put it any `*_test.go` file under `fakestorage` directory it just runs silently

kerem commented

2026-03-03 12:08:56 +03:00

Author

Owner

@jboyens commented on GitHub (Jul 21, 2023):

I had a very very similar issue with Google's Go client. I don't know that it's the same issue, but this stumped me for longer than I'll care to admit.

By default, the Go client code will use the XML range reader rather than the JSON range reader (see github.com/googleapis/google-cloud-go@cf3452eba7/storage/http_client.go (L780-L790)).

If I, on the official client, say storage.WithJSONReads() as an option alongside option.WithEndpoint() everything is fine and I receive no 404.

@jboyens commented on GitHub (Jul 21, 2023): I had a very very similar issue with Google's Go client. I don't know that it's the same issue, but this stumped me for longer than I'll care to admit. By default, the Go client code will use the XML range reader rather than the JSON range reader (see https://github.com/googleapis/google-cloud-go/blob/cf3452eba7d02b28343c9e6c2425288f80e42d3c/storage/http_client.go#L780-L790). If I, on the official client, say `storage.WithJSONReads()` as an option alongside `option.WithEndpoint()` everything is fine and I receive no `404`.

kerem commented

2026-03-03 12:08:56 +03:00

Author

Owner

@bliuchak commented on GitHub (Jul 27, 2023):

@jboyens you've saved me a lot of time! storage.WithJSONReads() works like a charm.

@bliuchak commented on GitHub (Jul 27, 2023): @jboyens you've saved me a lot of time! `storage.WithJSONReads()` works like a charm.

kerem commented

2026-03-03 12:08:56 +03:00

Author

Owner

@pocockn commented on GitHub (Jan 20, 2024):

Thanks @jboyens !

@pocockn commented on GitHub (Jan 20, 2024): Thanks @jboyens !

kerem commented

2026-03-03 12:08:56 +03:00

Author

Owner

@glenn-smith-0 commented on GitHub (Apr 9, 2025):

Thanks @jboyens, this has fixed issue we were having https://github.com/fsouza/fake-gcs-server/issues/1952

@glenn-smith-0 commented on GitHub (Apr 9, 2025): Thanks @jboyens, this has fixed issue we were having https://github.com/fsouza/fake-gcs-server/issues/1952

kerem referenced this issue

2026-03-03 12:29:50 +03:00

[PR #173] [MERGED] build(deps): bump google.golang.org/api from 0.16.0 to 0.17.0 in /examples/go #402