[GH-ISSUE #315] Crc32c and Md5Hash do not match hashses created from GCS #63

New issue

Closed

opened 2026-03-03 12:07:56 +03:00 by kerem · 3 comments

kerem commented

2026-03-03 12:07:56 +03:00

Owner

Originally created by @antonydenyer on GitHub (Aug 27, 2020).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/315

When hashing the file the hash does not match the one generated by GCS. To reproduce the problem upload a file to GCS and upload the same file to fake-gcs-server. Fetch the object and look metadata and you will notice a different MD5 hash.

You can also reproduce this locally with gsutil hash -m

Essentially:

encodedMd5Hash([]byte("content1")) == "myuzIfXdGoeFfrh1ziL34Q==

https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/upload.go#L217

Originally created by @antonydenyer on GitHub (Aug 27, 2020). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/315 When hashing the file the hash does not match the one generated by GCS. To reproduce the problem upload a file to GCS and upload the same file to fake-gcs-server. Fetch the object and look metadata and you will notice a different MD5 hash. You can also reproduce this locally with `gsutil hash -m` Essentially: `encodedMd5Hash([]byte("content1")) == "myuzIfXdGoeFfrh1ziL34Q==` https://github.com/fsouza/fake-gcs-server/blob/main/fakestorage/upload.go#L217

kerem closed this issue

2026-03-03 12:07:56 +03:00

kerem commented

2026-03-03 12:07:57 +03:00

Author

Owner

@fsouza commented on GitHub (Aug 27, 2020):

@antonydenyer can you share the script or commands you're using to upload the files? Like, is the process to upload to GCS and fake-gcs-server exactly the same? "myuzIfXdGoeFfrh1ziL34Q==" is not the correct hash for "content1", it's the hash for "content1\n":

％ echo "content1" > file-with-new-line
％ echo -n "content1" > file-with-no-new-line
％ gsutil hash -m file-with-no-new-line file-with-new-line
Hashes [base64] for file-with-no-new-line:
        Hash (md5):             flXbAB0xmpSwtxNSmnVmIw==
Hashes [base64] for file-with-new-line:
        Hash (md5):             myuzIfXdGoeFfrh1ziL34Q==

Operation completed over 2 objects/17.0 B.

@fsouza commented on GitHub (Aug 27, 2020): @antonydenyer can you share the script or commands you're using to upload the files? Like, is the process to upload to GCS and fake-gcs-server exactly the same? `"myuzIfXdGoeFfrh1ziL34Q=="` is not the correct hash for `"content1"`, it's the hash for `"content1\n"`: ``` ％ echo "content1" > file-with-new-line ％ echo -n "content1" > file-with-no-new-line ％ gsutil hash -m file-with-no-new-line file-with-new-line Hashes [base64] for file-with-no-new-line: Hash (md5): flXbAB0xmpSwtxNSmnVmIw== Hashes [base64] for file-with-new-line: Hash (md5): myuzIfXdGoeFfrh1ziL34Q== Operation completed over 2 objects/17.0 B. ```

kerem commented

2026-03-03 12:07:57 +03:00

Author

Owner

@fsouza commented on GitHub (Aug 27, 2020):

(perhaps there's a bug in our backend and it's not preserving the new line at the end of the file, I'll double check that)

@fsouza commented on GitHub (Aug 27, 2020): (perhaps there's a bug in our backend and it's not preserving the new line at the end of the file, I'll double check that)

kerem commented

2026-03-03 12:07:57 +03:00

Author

Owner

@antonydenyer commented on GitHub (Aug 27, 2020):

I take it back! Can't reproduce, must have fat fingered a newline during initial testing.

@antonydenyer commented on GitHub (Aug 27, 2020): I take it back! Can't reproduce, must have fat fingered a newline during initial testing.

kerem referenced this issue

2026-03-03 12:10:11 +03:00

[PR #63] [MERGED] build(deps): bump golang from 1.12.8-alpine to 1.12.9-alpine #307