starred/fake-gcs-server

Fork 0

mirror of https://github.com/fsouza/fake-gcs-server.git synced 2026-04-26 06:05:54 +03:00

[GH-ISSUE #458] Failing to read from an object I just wrote #88

New issue

Closed

opened 2026-03-03 12:08:10 +03:00 by kerem · 5 comments

kerem commented

2026-03-03 12:08:10 +03:00

Owner

Originally created by @mipnw on GitHub (Mar 19, 2021).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/458

When I run the following unit test which tries to round trip an objet to GCP storage (upload then download and compare equality of the object content) I get failure
Error: Expected nil, but got: &url.Error{Op:"Get", URL:"https://storage.googleapis.com/my-bucket/my-object", Err:(*errors.errorString)(...)}

I was expecting that test to pass. Having configured the fake gcs server for the HTTP scheme, I'm surprised by the "https" present in the URL of the error.

Is this my misunderstanding of how to use fakestorage.NewServerWithOptions, or is this a bug in this Go package?

package fakestore_test

import (
	"bytes"
	"context"
	"io"
	"io/ioutil"
	"testing"

	"github.com/fsouza/fake-gcs-server/fakestorage"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
)

func TestFake(t *testing.T) {
	mockServer, err := fakestorage.NewServerWithOptions(
		fakestorage.Options{
			Scheme:     "http",
			Host:       "localhost",
			Port:       uint16(8080),
			PublicHost: "localhost:8080",
		},
	)
	require.Nil(t, err)
	require.NotNil(t, mockServer)
	defer mockServer.Stop()

	mockClient := mockServer.Client()
	bucketName := "my-bucket"
	objectName := "my-object"
	ctx := context.Background()
	writer := mockClient.Bucket(bucketName).Object(objectName).NewWriter(ctx)

	expectedContent := []byte("my content")
	bytesReader := bytes.NewReader(expectedContent)
	countBytesWritten, err := io.Copy(writer, bytesReader)
	require.Nil(t, err)
	require.Equal(t, int64(len(expectedContent)), countBytesWritten)

	reader, err := mockClient.Bucket(bucketName).Object(objectName).NewReader(ctx)
	require.Nil(t, err)
	require.NotNil(t, reader)
	actualContent, err := ioutil.ReadAll(reader)
	require.Nil(t, err)
	assert.Equal(t, expectedContent, actualContent)
}

My go.mod references github.com/fsouza/fake-gcs-server v1.22.7. Using go version go1.15.6 darwin/amd64

Originally created by @mipnw on GitHub (Mar 19, 2021). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/458 When I run the following unit test which tries to round trip an objet to GCP storage (upload then download and compare equality of the object content) I get failure `Error: Expected nil, but got: &url.Error{Op:"Get", URL:"https://storage.googleapis.com/my-bucket/my-object", Err:(*errors.errorString)(...)}` I was expecting that test to pass. Having configured the fake gcs server for the HTTP scheme, I'm surprised by the "https" present in the URL of the error. Is this my misunderstanding of how to use `fakestorage.NewServerWithOptions`, or is this a bug in this Go package? ``` package fakestore_test import ( "bytes" "context" "io" "io/ioutil" "testing" "github.com/fsouza/fake-gcs-server/fakestorage" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) func TestFake(t *testing.T) { mockServer, err := fakestorage.NewServerWithOptions( fakestorage.Options{ Scheme: "http", Host: "localhost", Port: uint16(8080), PublicHost: "localhost:8080", }, ) require.Nil(t, err) require.NotNil(t, mockServer) defer mockServer.Stop() mockClient := mockServer.Client() bucketName := "my-bucket" objectName := "my-object" ctx := context.Background() writer := mockClient.Bucket(bucketName).Object(objectName).NewWriter(ctx) expectedContent := []byte("my content") bytesReader := bytes.NewReader(expectedContent) countBytesWritten, err := io.Copy(writer, bytesReader) require.Nil(t, err) require.Equal(t, int64(len(expectedContent)), countBytesWritten) reader, err := mockClient.Bucket(bucketName).Object(objectName).NewReader(ctx) require.Nil(t, err) require.NotNil(t, reader) actualContent, err := ioutil.ReadAll(reader) require.Nil(t, err) assert.Equal(t, expectedContent, actualContent) } ``` My go.mod references `github.com/fsouza/fake-gcs-server v1.22.7`. Using `go version go1.15.6 darwin/amd64`

kerem

2026-03-03 12:08:10 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-03 12:08:12 +03:00

Author

Owner

@fsouza commented on GitHub (Mar 19, 2021):

Yeah this is a bug in the client. If you change the server to "https" instead of "http" it should work.

I can push a fix later today (or if you're willing to give it a shot, we need to implement Dial in addition to DialTLS in the http.Transport instance).

@fsouza commented on GitHub (Mar 19, 2021): Yeah this is a bug in the client. If you change the server to "https" instead of "http" it should work. I can push a fix later today (or if you're willing to give it a shot, we need to implement `Dial` in addition to `DialTLS` in the http.Transport instance).

kerem commented

2026-03-03 12:08:12 +03:00

Author

Owner

@mipnw commented on GitHub (Mar 19, 2021):

ok I've modified the fakestorage.Options to use Scheme: "https". The test still fails at reader, err := mockClient.Bucket(bucketName).Object(objectName).NewReader(ctx) with error "storage: object doesn't exist"

@mipnw commented on GitHub (Mar 19, 2021): ok I've modified the `fakestorage.Options` to use `Scheme: "https"`. The test still fails at `reader, err := mockClient.Bucket(bucketName).Object(objectName).NewReader(ctx)` with error "storage: object doesn't exist"

kerem commented

2026-03-03 12:08:12 +03:00

Author

Owner

@fsouza commented on GitHub (Mar 19, 2021):

@mipnw you need to close the writer for it to create the object: https://pkg.go.dev/cloud.google.com/go/storage#ObjectHandle.NewWriter

@fsouza commented on GitHub (Mar 19, 2021): @mipnw you need to close the writer for it to create the object: https://pkg.go.dev/cloud.google.com/go/storage#ObjectHandle.NewWriter

kerem commented

2026-03-03 12:08:12 +03:00

Author

Owner

@mipnw commented on GitHub (Mar 19, 2021):

I've fixed that test bug by closing the writer.

I'm still getting an error that "storage: object doesn't exist" but now it's not coming from a failure to write the object because I can see that object show up in my filesystem if I pass a StorageRoot option. So part 1 (write the object) seems to work. But part 2 (read back the object), still gives me trouble.

package fakestore_test

import (
	"context"
	"io/ioutil"
	"testing"

	"github.com/fsouza/fake-gcs-server/fakestorage"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
)

func TestFake(t *testing.T) {
	mockServer, err := fakestorage.NewServerWithOptions(
		fakestorage.Options{
			//StorageRoot: "./",
			Scheme:     "https",
			Host:       "localhost",
			Port:       uint16(8080),
			PublicHost: "localhost:8080",
		},
	)
	require.Nil(t, err)
	require.NotNil(t, mockServer)
	defer mockServer.Stop()

	projectId := "my-project"
	bucketName := "my-bucket"
	objectName := "my-object"

	mockClient := mockServer.Client()
	ctx := context.Background()
	err = mockClient.Bucket(bucketName).Create(ctx, projectId, &storage.BucketAttrs{})
	require.Nil(t, err)
	writer := mockClient.Bucket(bucketName).Object(objectName).NewWriter(ctx)

	expectedContent := []byte("my content")
	countBytesWritten, err := writer.Write(expectedContent)
	require.Nil(t, err)
	require.Equal(t, len(expectedContent), countBytesWritten)
	err = writer.Close()
	require.Nil(t, err)

	reader, err := mockClient.Bucket(bucketName).Object(objectName).NewReader(ctx)
	require.Nil(t, err)
	require.NotNil(t, reader)
	actualContent, err := ioutil.ReadAll(reader)
	require.Nil(t, err)
	assert.Equal(t, expectedContent, actualContent)
}

@mipnw commented on GitHub (Mar 19, 2021): I've fixed that test bug by closing the writer. I'm still getting an error that "storage: object doesn't exist" but now it's not coming from a failure to write the object because I can see that object show up in my filesystem if I pass a `StorageRoot` option. So part 1 (write the object) seems to work. But part 2 (read back the object), still gives me trouble. ``` package fakestore_test import ( "context" "io/ioutil" "testing" "github.com/fsouza/fake-gcs-server/fakestorage" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) func TestFake(t *testing.T) { mockServer, err := fakestorage.NewServerWithOptions( fakestorage.Options{ //StorageRoot: "./", Scheme: "https", Host: "localhost", Port: uint16(8080), PublicHost: "localhost:8080", }, ) require.Nil(t, err) require.NotNil(t, mockServer) defer mockServer.Stop() projectId := "my-project" bucketName := "my-bucket" objectName := "my-object" mockClient := mockServer.Client() ctx := context.Background() err = mockClient.Bucket(bucketName).Create(ctx, projectId, &storage.BucketAttrs{}) require.Nil(t, err) writer := mockClient.Bucket(bucketName).Object(objectName).NewWriter(ctx) expectedContent := []byte("my content") countBytesWritten, err := writer.Write(expectedContent) require.Nil(t, err) require.Equal(t, len(expectedContent), countBytesWritten) err = writer.Close() require.Nil(t, err) reader, err := mockClient.Bucket(bucketName).Object(objectName).NewReader(ctx) require.Nil(t, err) require.NotNil(t, reader) actualContent, err := ioutil.ReadAll(reader) require.Nil(t, err) assert.Equal(t, expectedContent, actualContent) } ```

kerem commented

2026-03-03 12:08:12 +03:00

Author

Owner

@fsouza commented on GitHub (Mar 20, 2021):

@mipnw I think the problem is the public host, if you remove that everything should work.

I'll fix the Client() method to return a client that supports both http and the custom public host.

@fsouza commented on GitHub (Mar 20, 2021): @mipnw I think the problem is the public host, if you remove that everything should work. I'll fix the Client() method to return a client that supports both http and the custom public host.

kerem referenced this issue

2026-03-03 12:29:31 +03:00

[PR #88] [MERGED] build(deps): bump google.golang.org/api from 0.10.0 to 0.11.0 #330