[GH-ISSUE #1226] bug: invalid CRC computed #178

New issue

Closed

opened 2026-03-03 12:08:56 +03:00 by kerem · 1 comment

kerem commented

2026-03-03 12:08:56 +03:00

Owner

Originally created by @hairyhenderson on GitHub (Jun 26, 2023).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1226

In previous versions of the GCS storage client (cloud.google.com/go/storage, up to v1.29.0), there was a parsing bug that caused the client to skip CRC validation.

However, in v1.30.0, a fix for that bug was released, and since then I've received this error while reading:

storage: bad CRC on read: got 1077016486, want 2266108419

My hunch is the CRC calculation in fsouza/fake-gcs-server is incorrect (compared to how GCS does it); when the feature was added (https://github.com/fsouza/fake-gcs-server/pull/307), this comment in the description stands out:

I compared the results against actually uploading to GCS (and using gsutil hash -m) and they differ for some reason. Whilst this is somewhat annoying it isn't a problem for me as all I need is something consistent.

I think this wasn't a problem then, since the header wasn't used, but now that cloud.google.com/go/storage has fixed that bug this is now a problem!

/cc @antonydenyer @fsouza

Originally created by @hairyhenderson on GitHub (Jun 26, 2023). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1226 In previous versions of the GCS storage client (`cloud.google.com/go/storage`, up to v1.29.0), there was a parsing bug that caused the client to skip CRC validation. However, in v1.30.0, [a fix](https://github.com/googleapis/google-cloud-go/commit/574a86c614445f8c3f5a54446820df774c31cd47) for that bug was released, and since then I've received this error while reading: ``` storage: bad CRC on read: got 1077016486, want 2266108419 ``` My hunch is the CRC calculation in `fsouza/fake-gcs-server` is incorrect (compared to how GCS does it); when the feature was added (https://github.com/fsouza/fake-gcs-server/pull/307), this comment in the description stands out: > I compared the results against actually uploading to GCS (and using `gsutil hash -m`) and they differ for some reason. Whilst this is somewhat annoying it isn't a problem for me as all I need is something consistent. I think this wasn't a problem then, since the header wasn't used, but now that `cloud.google.com/go/storage` has fixed that bug this is now a problem! /cc @antonydenyer @fsouza

kerem

2026-03-03 12:08:56 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-03 12:08:57 +03:00

Author

Owner

@hairyhenderson commented on GitHub (Jun 26, 2023):

I've done a bit more digging and it looks like the issue is due to compression - I'm gzip-compressing the object on upload, and when downloading the object Decompressive transcoding is occurring.

When trying to determine if the server uncompressed the content, the client expects there to be a X-Goog-Stored-Content-Encoding header set to gzip, without a Content-Encoding header set to gzip:

https://github.com/googleapis/google-cloud-go/blob/main/storage/reader.go#L135

It appears that fake-gcs-server is decompressing the content, but it is not setting the X-Goog-Stored-Content-Encoding header. I think that's the root of this bug. I'll see if I can find a fix.

@hairyhenderson commented on GitHub (Jun 26, 2023): I've done a bit more digging and it looks like the issue is due to compression - I'm gzip-compressing the object on upload, and when downloading the object [Decompressive transcoding](https://cloud.google.com/storage/docs/transcoding#decompressive_transcoding) is occurring. When trying to determine if the server uncompressed the content, the client expects there to be a `X-Goog-Stored-Content-Encoding` header set to `gzip`, without a `Content-Encoding` header set to `gzip`: https://github.com/googleapis/google-cloud-go/blob/main/storage/reader.go#L135 It appears that `fake-gcs-server` _is_ decompressing the content, but it is _not_ setting the `X-Goog-Stored-Content-Encoding` header. I think that's the root of this bug. I'll see if I can find a fix.

kerem referenced this issue

2026-03-03 12:29:52 +03:00

[PR #178] [MERGED] build(deps): bump golang from 1.13.7 to 1.13.8 #407