[GH-ISSUE #982] Unable to access uploaded file (go + testcontainers) #154

New issue

Open

opened 2026-03-03 12:08:46 +03:00 by kerem · 8 comments

kerem commented 2026-03-03 12:08:46 +03:00

Owner

Copy link

Originally created by @rtrzebinski-usc on GitHub (Nov 6, 2022).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/982

Hi, here is my issue with the emulator - I use testcontainers for integration test, uploading works fine (I see the file in the container) but there is no way to access it via HTTP or from the code, here is the script to reproduce:

package fsouza

import (
	"bytes"
	"cloud.google.com/go/storage"
	"context"
	"fmt"
	"github.com/docker/go-connections/nat"
	"github.com/stretchr/testify/assert"
	"github.com/testcontainers/testcontainers-go"
	"github.com/testcontainers/testcontainers-go/wait"
	"google.golang.org/api/option"
	"io"
	"log"
	"os"
	"testing"
	"time"
)

func TestUpload(t *testing.T) {
	ctx := context.Background()

	var port = "4443/tcp"
	req := testcontainers.GenericContainerRequest{
		ContainerRequest: testcontainers.ContainerRequest{
			Image:        "fsouza/fake-gcs-server",
			ExposedPorts: []string{port},
			Cmd:          []string{"-scheme", "http", "-port", "4443", "-public-host", "localhost:4443"},
			WaitingFor: wait.ForAll(
				wait.ForLog("server started at"),
			),
		},
		Started: true,
	}
	container, err := testcontainers.GenericContainer(ctx, req)
	defer container.Terminate(ctx)
	if err != nil {
		log.Fatal(err)
	}

	assert.True(t, container.IsRunning(), "container is not running")

	mappedPort, err := container.MappedPort(ctx, nat.Port(port))
	if err != nil {
		log.Fatal(err)
	}

	log.Println("gcs container ready and running at port: ", mappedPort.Port())

	// Set STORAGE_EMULATOR_HOST environment variable.
	err = os.Setenv("STORAGE_EMULATOR_HOST", fmt.Sprintf("%s:%s", "http://localhost", mappedPort.Port()))
	if err != nil {
		log.Fatalf("os.Setenv: %v", err)
	}

	endpoint := fmt.Sprintf("http://localhost:%s/storage/v1/", mappedPort.Port())
	log.Println("Endpoint:", endpoint)

	s, err := storage.NewClient(ctx, option.WithEndpoint(endpoint))
	if err != nil {
		t.Error(err)
	}

	// file to be uploaded
	var buffer bytes.Buffer
	buffer.WriteString("foo bar")
	bucketName := "foo"
	objectName := "bar"

	bucket := s.Bucket(bucketName)
	if err := bucket.Create(ctx, "abc", nil); err != nil {
		log.Printf("Failed to create bucket: %v", err)
	}

	o := s.Bucket(bucketName).Object(objectName)

	wc := o.NewWriter(ctx)

	written, err := io.Copy(wc, &buffer)

	if err != nil {
		log.Printf("failed to upload file, %v", err)
	}

	fmt.Printf("uploader - written: %v", written)

	err = wc.Close()

	if err != nil {
		t.Error(err)
	}

	log.Println("Creating reader")

	rd, err := bucket.Object(objectName).NewReader(ctx)

	if err != nil {
		log.Printf("failed to create reader, %v", err)
	}

	println("sleeping..")
	time.Sleep(1000 * time.Second)

	res, err := io.ReadAll(rd)

	if err != nil {
		t.Errorf("failed to read, %v", err)
	}

	fmt.Print(res)
}

When you run it will print the base url like http://localhost:49405/storage/v1 - I try to add bucked and object name and access like http://localhost:49405/storage/v1/foo/bar - getting 404.

Then since script is paused by time.Sleep(1000 * time.Second) I can inspect the image - I see that:

• file is uploaded
• API requests are in web logs, but with 404

invoice-srv $ docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS          PORTS                                         NAMES
ca2561b006f1   fsouza/fake-gcs-server      "/bin/fake-gcs-serve…"   24 seconds ago   Up 22 seconds   0.0.0.0:49407->4443/tcp, :::49402->4443/tcp   eloquent_cohen
e495202a08a7   testcontainers/ryuk:0.3.4   "/app"                   27 seconds ago   Up 25 seconds   0.0.0.0:49406->8080/tcp, :::49401->8080/tcp   condescending_gould
invoice-srv $ docker exec -it ca2561b006f1 sh
/ # ls /storage/foo/bar
/storage/foo/bar
/ # cat /storage/foo/bar
foo bar/ # exit
invoice-srv $ docker logs ca2561b006f1
time="2022-11-06T13:51:13Z" level=info msg="couldn't load any objects or buckets from \"/data\", starting empty"
time="2022-11-06T13:51:13Z" level=info msg="server started at http://[::]:4443"
time="2022-11-06T13:51:16Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:16 +0000] \"GET /storage/v1/ HTTP/1.1\" 404 10"
time="2022-11-06T13:51:16Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:16 +0000] \"GET /favicon.ico HTTP/1.1\" 404 10"
time="2022-11-06T13:51:19Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:19 +0000] \"POST /storage/v1/b?alt=json&prettyPrint=false&project=abc HTTP/1.1\" 200 131"
time="2022-11-06T13:51:19Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:19 +0000] \"POST /upload/storage/v1/b/foo/o?alt=json&name=bar&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1\" 200 442"
time="2022-11-06T13:51:19Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:19 +0000] \"GET /foo/bar HTTP/1.1\" 404 10"
time="2022-11-06T13:51:20Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:20 +0000] \"GET /storage/v1/foo/bar HTTP/1.1\" 404 10"

What I do wrong? Any help will be appreciated, thank you :)

Originally created by @rtrzebinski-usc on GitHub (Nov 6, 2022). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/982 Hi, here is my issue with the emulator - I use `testcontainers` for integration test, uploading works fine (I see the file in the container) but there is no way to access it via HTTP or from the code, here is the script to reproduce: ``` package fsouza import ( "bytes" "cloud.google.com/go/storage" "context" "fmt" "github.com/docker/go-connections/nat" "github.com/stretchr/testify/assert" "github.com/testcontainers/testcontainers-go" "github.com/testcontainers/testcontainers-go/wait" "google.golang.org/api/option" "io" "log" "os" "testing" "time" ) func TestUpload(t *testing.T) { ctx := context.Background() var port = "4443/tcp" req := testcontainers.GenericContainerRequest{ ContainerRequest: testcontainers.ContainerRequest{ Image: "fsouza/fake-gcs-server", ExposedPorts: []string{port}, Cmd: []string{"-scheme", "http", "-port", "4443", "-public-host", "localhost:4443"}, WaitingFor: wait.ForAll( wait.ForLog("server started at"), ), }, Started: true, } container, err := testcontainers.GenericContainer(ctx, req) defer container.Terminate(ctx) if err != nil { log.Fatal(err) } assert.True(t, container.IsRunning(), "container is not running") mappedPort, err := container.MappedPort(ctx, nat.Port(port)) if err != nil { log.Fatal(err) } log.Println("gcs container ready and running at port: ", mappedPort.Port()) // Set STORAGE_EMULATOR_HOST environment variable. err = os.Setenv("STORAGE_EMULATOR_HOST", fmt.Sprintf("%s:%s", "http://localhost", mappedPort.Port())) if err != nil { log.Fatalf("os.Setenv: %v", err) } endpoint := fmt.Sprintf("http://localhost:%s/storage/v1/", mappedPort.Port()) log.Println("Endpoint:", endpoint) s, err := storage.NewClient(ctx, option.WithEndpoint(endpoint)) if err != nil { t.Error(err) } // file to be uploaded var buffer bytes.Buffer buffer.WriteString("foo bar") bucketName := "foo" objectName := "bar" bucket := s.Bucket(bucketName) if err := bucket.Create(ctx, "abc", nil); err != nil { log.Printf("Failed to create bucket: %v", err) } o := s.Bucket(bucketName).Object(objectName) wc := o.NewWriter(ctx) written, err := io.Copy(wc, &buffer) if err != nil { log.Printf("failed to upload file, %v", err) } fmt.Printf("uploader - written: %v", written) err = wc.Close() if err != nil { t.Error(err) } log.Println("Creating reader") rd, err := bucket.Object(objectName).NewReader(ctx) if err != nil { log.Printf("failed to create reader, %v", err) } println("sleeping..") time.Sleep(1000 * time.Second) res, err := io.ReadAll(rd) if err != nil { t.Errorf("failed to read, %v", err) } fmt.Print(res) } ``` When you run it will print the base url like `http://localhost:49405/storage/v1` - I try to add bucked and object name and access like `http://localhost:49405/storage/v1/foo/bar` - getting 404. Then since script is paused by `time.Sleep(1000 * time.Second)` I can inspect the image - I see that: - file is uploaded - API requests are in web logs, but with 404 ``` invoice-srv $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ca2561b006f1 fsouza/fake-gcs-server "/bin/fake-gcs-serve…" 24 seconds ago Up 22 seconds 0.0.0.0:49407->4443/tcp, :::49402->4443/tcp eloquent_cohen e495202a08a7 testcontainers/ryuk:0.3.4 "/app" 27 seconds ago Up 25 seconds 0.0.0.0:49406->8080/tcp, :::49401->8080/tcp condescending_gould invoice-srv $ docker exec -it ca2561b006f1 sh / # ls /storage/foo/bar /storage/foo/bar / # cat /storage/foo/bar foo bar/ # exit invoice-srv $ docker logs ca2561b006f1 time="2022-11-06T13:51:13Z" level=info msg="couldn't load any objects or buckets from \"/data\", starting empty" time="2022-11-06T13:51:13Z" level=info msg="server started at http://[::]:4443" time="2022-11-06T13:51:16Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:16 +0000] \"GET /storage/v1/ HTTP/1.1\" 404 10" time="2022-11-06T13:51:16Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:16 +0000] \"GET /favicon.ico HTTP/1.1\" 404 10" time="2022-11-06T13:51:19Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:19 +0000] \"POST /storage/v1/b?alt=json&prettyPrint=false&project=abc HTTP/1.1\" 200 131" time="2022-11-06T13:51:19Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:19 +0000] \"POST /upload/storage/v1/b/foo/o?alt=json&name=bar&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1\" 200 442" time="2022-11-06T13:51:19Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:19 +0000] \"GET /foo/bar HTTP/1.1\" 404 10" time="2022-11-06T13:51:20Z" level=info msg="172.17.0.1 - - [06/Nov/2022:13:51:20 +0000] \"GET /storage/v1/foo/bar HTTP/1.1\" 404 10" ``` What I do wrong? Any help will be appreciated, thank you :)

kerem commented 2026-03-03 12:08:47 +03:00

Author

Owner

Copy link

@rtrzebinski-usc commented on GitHub (Nov 6, 2022):

I found out that the url to download the file would be http://localhost:49409/download/storage/v1/b/foo/o/bar - but how do I check whether file was uploaded using the code (reader) so I don't receive failed to create reader, storage: object doesn't exist error?

<!-- gh-comment-id:1304815969 --> @rtrzebinski-usc commented on GitHub (Nov 6, 2022): I found out that the url to download the file would be `http://localhost:49409/download/storage/v1/b/foo/o/bar` - but how do I check whether file was uploaded using the code (reader) so I don't receive `failed to create reader, storage: object doesn't exist` error?

kerem commented 2026-03-03 12:08:47 +03:00

Author

Owner

Copy link

@dezyh commented on GitHub (Nov 13, 2022):

I'm also able to reproduce the above (with my own implementation). The issue is related to the -public-host argument not being set.

If I adapt the go example to create an object and then read that object, this only works -public-host localhost:8080 is set (ci script). Without it, I also get storage: object doesn't exist when executing it.

Solution

- Cmd:          []string{"-scheme", "http", "-port", "4443", "-public-host", "localhost:4443"}
+ Cmd:          []string{"-scheme", "http", "-port", "4443", "-public-host", "0.0.0.0"}

Journey

Exploring the source code, you can see that one of the downloadObject's MatcherFunc's uses the publicHostMatcher (which uses the public-host argument and publicHost config.
github.com/fsouza/fake-gcs-server@071372eda6/fakestorage/server.go (L274-L277)

So taking a look at this MatcherFunc, it will match requests only for a specific port if one is given, but falls back to matching any port if none is given.
github.com/fsouza/fake-gcs-server@071372eda6/fakestorage/server.go (L293-L300)

<!-- gh-comment-id:1312672609 --> @dezyh commented on GitHub (Nov 13, 2022): I'm also able to reproduce the above (with my own implementation). The issue is related to the `-public-host` argument not being set. If I adapt the go example to create an object and then read that object, this only works `-public-host localhost:8080` is set ([ci script](https://github.com/fsouza/fake-gcs-server/blob/main/ci/run-go-example.sh#L8)). Without it, I also get `storage: object doesn't exist` when executing it. ### Solution ``` - Cmd: []string{"-scheme", "http", "-port", "4443", "-public-host", "localhost:4443"} + Cmd: []string{"-scheme", "http", "-port", "4443", "-public-host", "0.0.0.0"} ``` ### Journey Exploring the source code, you can see that one of the `downloadObject`'s `MatcherFunc`'s uses the `publicHostMatcher` (which uses the `public-host` argument and `publicHost` config. https://github.com/fsouza/fake-gcs-server/blob/071372eda6e3cfd4db175a83ef4fa6cb75f024de/fakestorage/server.go#L274-L277 So taking a look at this `MatcherFunc`, it will match requests only for a specific port if one is given, but falls back to matching any port if none is given. https://github.com/fsouza/fake-gcs-server/blob/071372eda6e3cfd4db175a83ef4fa6cb75f024de/fakestorage/server.go#L293-L300

kerem commented 2026-03-03 12:08:47 +03:00

Author

Owner

Copy link

@dezyh commented on GitHub (Nov 13, 2022):

The documentation in the README mentions setting -public-host but doesn't mention that

1. It is required to set -public-host for downloadObject (just for pre-signed urls).
2. It's possible to omit a port when specifying -public-host

It might be nice to document this, because it also cost me about 2 hours to work through everything (especially because I initially thought it was an implementation bug in my code)?

<!-- gh-comment-id:1312682882 --> @dezyh commented on GitHub (Nov 13, 2022): The documentation in the README [mentions](https://github.com/fsouza/fake-gcs-server#using-with-signed-urls) setting `-public-host` but doesn't mention that 1. It is required to set `-public-host` for `downloadObject` (just for pre-signed urls). 2. It's possible to omit a port when specifying `-public-host` It might be nice to document this, because it also cost me about 2 hours to work through everything (especially because I initially thought it was an implementation bug in my code)?

kerem commented 2026-03-03 12:08:47 +03:00

Author

Owner

Copy link

@fsouza commented on GitHub (Nov 16, 2022):

@dezyh thanks for digging into this. Since you did the investigation, do you also want to send a PR with improvements to the docs? I can do it if you prefer!

<!-- gh-comment-id:1316394797 --> @fsouza commented on GitHub (Nov 16, 2022): @dezyh thanks for digging into this. Since you did the investigation, do you also want to send a PR with improvements to the docs? I can do it if you prefer!

kerem commented 2026-03-03 12:08:47 +03:00

Author

Owner

Copy link

@dezyh commented on GitHub (Nov 16, 2022):

I'll make a readme PR, just wanted to check that I wasn't missing something obvious.

<!-- gh-comment-id:1317237445 --> @dezyh commented on GitHub (Nov 16, 2022): I'll make a readme PR, just wanted to check that I wasn't missing something obvious.

kerem commented 2026-03-03 12:08:48 +03:00

Author

Owner

Copy link

@fsouza commented on GitHub (Nov 16, 2022):

@dezyh oh, that sounds good. I'll dig into that sample code to see if there's a better fix, but yeah sounds like we need to clarify the role of public-host in the docs.

<!-- gh-comment-id:1317247652 --> @fsouza commented on GitHub (Nov 16, 2022): @dezyh oh, that sounds good. I'll dig into that sample code to see if there's a better fix, but yeah sounds like we need to clarify the role of `public-host` in the docs.

kerem commented 2026-03-03 12:08:48 +03:00

Author

Owner

Copy link

@sourabhsparkala commented on GitHub (Nov 22, 2022):

@fsouza @dezyh I followed the instructions given here and I just ran the code given above and set the public-host to 0.0.0.0. But I still face the same issue. Am I missing anything?

2022/11/22 12:12:41 failed to create reader, storage: object doesn't exist
sleeping..

@rtrzebinski-usc were you able to run this test case successfully?

<!-- gh-comment-id:1323509013 --> @sourabhsparkala commented on GitHub (Nov 22, 2022): @fsouza @dezyh I followed the instructions given here and I just ran the code given above and set the `public-host` to `0.0.0.0`. But I still face the same issue. Am I missing anything? ``` 2022/11/22 12:12:41 failed to create reader, storage: object doesn't exist sleeping.. ``` @rtrzebinski-usc were you able to run this test case successfully?

kerem commented 2026-03-03 12:08:48 +03:00

Author

Owner

Copy link

@dezyh commented on GitHub (Dec 2, 2022):

Sorry, I was busy for a while.

I'm personally using dockertest and running containers inside a docker network. This works perfectly both local and on GitHub Actions with my above comment.

One small subtlety, is that connecting through the network's hostname for the container (not sure on terminology) doesn't work in GitHub Actions. While this is partly dockertest specific, it might also apply with testcontainers.

CloudStorageEndpoint = fmt.Sprintf("%s:%s", cloudStorageContainer.Container.NetworkSettings.Networks["test"].IPAddress, CloudStoragePort)

I have to connect through a port that's bound to the default network (again, sorry don't know the correct terminology)

CloudStorageEndpoint = fmt.Sprintf("0.0.0.0:%s", cloudStorageContainer.GetPort(fmt.Sprintf("%s/tcp", CloudStoragePort)))

<!-- gh-comment-id:1334850669 --> @dezyh commented on GitHub (Dec 2, 2022): Sorry, I was busy for a while. I'm personally using dockertest and running containers inside a docker network. This works perfectly both local and on GitHub Actions with my above comment. One small subtlety, is that connecting through the network's hostname for the container (not sure on terminology) doesn't work in GitHub Actions. While this is partly dockertest specific, it might also apply with testcontainers. ``` CloudStorageEndpoint = fmt.Sprintf("%s:%s", cloudStorageContainer.Container.NetworkSettings.Networks["test"].IPAddress, CloudStoragePort) ``` I have to connect through a port that's bound to the default network (again, sorry don't know the correct terminology) ``` CloudStorageEndpoint = fmt.Sprintf("0.0.0.0:%s", cloudStorageContainer.GetPort(fmt.Sprintf("%s/tcp", CloudStoragePort))) ```

kerem referenced this issue 2026-03-03 12:29:45 +03:00

[PR #154] [MERGED] Support serving HTTP #384

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

dependabot/npm_and_yarn/examples/node/fast-xml-parser-5.7.1

dependabot/github_actions/goreleaser/goreleaser-action-7.1.0

dependabot/github_actions/actions/cache-5.0.5

v1.54.0

v1.53.1

v1.53.0

v1.52.3

v1.52.2

v1.52.1

v1.52.0

v1.51.0

v1.50.2

v1.50.1

v1.50.0

v1.49.3

v1.49.2

v1.49.1

v1.49.0

v1.48.0

v1.47.8

v1.47.7

v1.47.6

v1.47.5

v1.47.4

v1.47.3

v1.47.2

v1.47.1

v1.47.0

v1.46.0

v1.45.2

v1.45.1

v1.45.0

v1.44.2

v1.44.1

v1.44.0

v1.43.0

v1.42.2

v1.42.1

v1.42.0

v1.41.0

v1.40.3

v1.40.2

v1.40.1

v1.40.0

v1.39.0

v1.38.4

v1.38.3

v1.38.2

v1.38.1

v1.38.0

v1.37.12

v1.37.11

v1.37.10

v1.37.9

v1.37.8

v1.37.3

v1.37.2

v1.37.1

v1.37.0

v1.36.3

v1.36.2

v1.36.1

v1.36.0

v1.35.0

v1.34.1

v1.34.0

v1.33.1

v1.33.0

v1.32.2

v1.32.1

v1.32.0

v1.31.1

v1.31.0

v1.30.2

v1.30.1

v1.30.0

1.29.2

v1.29.1

v1.29.0

v1.28.0

v1.27.0

v1.26.0

v1.25.2

v1.25.1

v1.25.0

v1.24.0

v1.23.1

v1.23.0

v1.22.7

v1.22.6

v1.22.5

v1.22.4

v1.22.3

v1.22.2

v1.22.1

v1.22.0

v1.21.2

v1.21.1

v1.21.0

v1.20.0

v1.19.5

v1.19.4

v1.19.3

v1.19.2

v1.19.1

v1.19.0

v1.18.4

v1.18.3

v1.18.2

v1.18.1

v1.18.0

v1.17.1

v1.17.0

v1.16.3

v1.16.2

v1.16.1

v1.16.0

v1.15.0

v1.14.1

v1.14.0

v1.13.1

v1.13.0

v1.12.1

v1.12.0

v1.11.6

v1.11.5

v1.11.4

v1.11.3

v1.11.2

v1.11.1

v1.11.0

v1.10.1

v1.10.0

v1.9.0

v1.8.0

v1.7.1

v1.7.0

v1.6.2

v1.6.1

v1.6.0

v1.5.1

v1.5.0

v1.4.2

v1.4.1

v1.4.0

v1.3.1

v1.3.0

v1.2.0

v1.1.0

v1.0.0

Labels

Clear labels   

bug

  

compatibility-issue

  

docker

  

documentation

  

enhancement

  

help wanted

  

needs information

  

pull-request

Mirrored from GitHub Pull Request

  

question

  

stale

  

unfortunate

  

wontfix

No labels

bug

compatibility-issue

docker

documentation

enhancement

help wanted

needs information

pull-request

question

stale

unfortunate

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/fake-gcs-server#154

No description provided.