[GH-ISSUE #684] uploaded file is crashed

kerem commented

2026-03-03 12:08:26 +03:00

Owner

Originally created by @ri8ika on GitHub (Feb 13, 2022).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/684

In mac machine, you can compare this issue by getting info of the file ("Get Info"). Under "More Info", there's no any value compared to original file. Additionally in "Preview", you can not see the preview (We can understand it is crashed). If the file is of text type, then the preview section will show the information in json format like metadata etc.

Now, let's analyze what happens on both mac and linux machines:

Mac machine:
When serving the file for the first time it loads the actual content. (Correct behavior)
After restarting the server and serving the file it loads the actual content. (Correct behavior)

Linux machine:
When serving the file for the first time it loads the actual content. (Correct behavior)
After restarting the server and serving the file it loads json content with the metadata etc information. (Wrong behavior)

Even though I have mentioned Correct behavior, in fact is not correct behavior. Because file content is uploaded / modified with some kind of json data (metadata, etc.) than actual content which is not correct - is obviously a bug.

This looks like the issue is similar to: https://github.com/fsouza/fake-gcs-server/issues/296

But I have created this issue so that you guys can debug and solve the issue. Hope this issue helps fixing the bug!

Originally created by @ri8ika on GitHub (Feb 13, 2022). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/684 In mac machine, you can compare this issue by getting info of the file ("Get Info"). Under "More Info", there's no any value compared to original file. Additionally in "Preview", you can not see the preview (We can understand it is crashed). If the file is of text type, then the preview section will show the information in json format like metadata etc. Now, let's analyze what happens on both mac and linux machines: Mac machine: When serving the file for the first time it loads the actual content. (Correct behavior) After restarting the server and serving the file it loads the actual content. (Correct behavior) Linux machine: When serving the file for the first time it loads the actual content. (Correct behavior) After restarting the server and serving the file it loads json content with the metadata etc information. (Wrong behavior) Even though I have mentioned Correct behavior, in fact is not correct behavior. Because file content is uploaded / modified with some kind of json data (metadata, etc.) than actual content which is not correct - is obviously a bug. This looks like the issue is similar to: https://github.com/fsouza/fake-gcs-server/issues/296 But I have created this issue so that you guys can debug and solve the issue. Hope this issue helps fixing the bug!

kerem added the

bug

label

2026-03-03 12:08:26 +03:00

kerem commented

2026-03-03 12:08:26 +03:00

Author

Owner

@fsouza commented on GitHub (Feb 15, 2022):

Yeah this is related to us not persisting metadata properly I think. I believe the work @gaul is doing in #683 should fix this. My plan is to review that PR before the end of this week. It should be good to go though, as @gaul definitely know what he's doing. It's just a matter of me finding the time (let me know if you can help test it!)

@fsouza commented on GitHub (Feb 15, 2022): Yeah this is related to us not persisting metadata properly I think. I believe the work @gaul is doing in #683 should fix this. My plan is to review that PR before the end of this week. It should be good to go though, as @gaul definitely know what he's doing. It's just a matter of me finding the time (let me know if you can help test it!)

kerem commented

2026-03-03 12:08:26 +03:00

Author

Owner

@ri8ika commented on GitHub (Feb 15, 2022):

Definitely. I have checked that as well and continuously watch the changes. If something is wrong (ie. result is not as expected) then I'll point out and let you know here.

By the way, I'm also debugging on this, if I could do something then I'd be glad to contribute.

@ri8ika commented on GitHub (Feb 15, 2022): Definitely. I have checked that as well and continuously watch the changes. If something is wrong (ie. result is not as expected) then I'll point out and let you know here. By the way, I'm also debugging on this, if I could do something then I'd be glad to contribute.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@ri8ika commented on GitHub (Feb 15, 2022):

Hi @fsouza

I have tested #683 PR and it's working great! 🙇 However, I have listed out some issues.

Currently, I have tested in Mac machine only:

Issue 1:

Run server with docker, and upload file.
user.metadata: operation not supported (xattr.Set)

Issue 2:

When serving video (tested only with .mp4 format),
it's being downloaded instead of loading

Issue 3:

Step 1: run server with go command
Step 2: uploaded file
Step 3: stop server
Step 4: run server with docker
uploaded file is there but it starts with empty,
that is to say uploaded file is not available while running with docker

Issue 4:

When serving pdf file, and svg image: (Other formats are working fine)
It's not loading actual content, it's showing some encoded characters.
However, restarting the server, it's showing actual content, no issue.

Issue 5:

Not able to play audio file when serving in browser, tested only in latest chrome browser.
However, it's being played locally.
Tested only with .mp3

I will let you know if I find any issue later.

@ri8ika commented on GitHub (Feb 15, 2022): Hi @fsouza I have tested #683 PR and it's working great! 🙇 However, I have listed out some issues. Currently, I have tested in **Mac machine** only: - Issue 1: ``` Run server with docker, and upload file. user.metadata: operation not supported (xattr.Set) ``` - Issue 2: ``` When serving video (tested only with .mp4 format), it's being downloaded instead of loading ``` - Issue 3: ``` Step 1: run server with go command Step 2: uploaded file Step 3: stop server Step 4: run server with docker uploaded file is there but it starts with empty, that is to say uploaded file is not available while running with docker ``` - Issue 4: ``` When serving pdf file, and svg image: (Other formats are working fine) It's not loading actual content, it's showing some encoded characters. However, restarting the server, it's showing actual content, no issue. ``` - Issue 5: ``` Not able to play audio file when serving in browser, tested only in latest chrome browser. However, it's being played locally. Tested only with .mp3 ``` I will let you know if I find any issue later.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@gaul commented on GitHub (Feb 15, 2022):

Issue 1: Run server with docker, and upload file. user.metadata: operation not supported (xattr.Set)

Docker has a variety of configurations so please be precise about what you are doing. Which path do you configure your -filesystem-root to point to? Is it inside the Docker container or does it point to a path on the macOS host? Does the path exist in /tmp which does not support xattr? If so you must you a different path and perhaps fake-gcs-server could warn more eagerly about this.

Issue 2: When serving video (tested only with .mp4 format), it's being downloaded instead of loading

Did you create these files with the current version of fake-gcs-server or were they externally created and exposed by fake-gcs-server? If the latter, you want to look at github.com/fsouza/fake-gcs-server@0d523fe7c3/internal/backend/fs.go (L205) which can do something more intelligent when the xattr are missing or blank. Specifically you want to guess the Content-Type from the file extension, e.g., .pdf becomes application/pdf. There are a few libraries that can do this.

I believe that issues 3-5 are similar to 2 so let's work through that first. This is a new feature so it may take some more debugging to understand. If you are comfortable programming in Go please submit any PRs to help improve fake-gcs-server!

@gaul commented on GitHub (Feb 15, 2022): > Issue 1: Run server with docker, and upload file. user.metadata: operation not supported (xattr.Set) Docker has a variety of configurations so please be precise about what you are doing. Which path do you configure your `-filesystem-root` to point to? Is it inside the Docker container or does it point to a path on the macOS host? Does the path exist in `/tmp` which does not support xattr? If so you must you a different path and perhaps fake-gcs-server could warn more eagerly about this. > Issue 2: When serving video (tested only with .mp4 format), it's being downloaded instead of loading Did you create these files with the current version of fake-gcs-server or were they externally created and exposed by fake-gcs-server? If the latter, you want to look at https://github.com/fsouza/fake-gcs-server/blob/0d523fe7c3677a92dc0bab97ee97777a91588d58/internal/backend/fs.go#L205 which can do something more intelligent when the xattr are missing or blank. Specifically you want to guess the Content-Type from the file extension, e.g., `.pdf` becomes `application/pdf`. There are a few libraries that can do this. I believe that issues 3-5 are similar to 2 so let's work through that first. This is a new feature so it may take some more debugging to understand. If you are comfortable programming in Go please submit any PRs to help improve fake-gcs-server!

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@ri8ika commented on GitHub (Feb 15, 2022):

Oops, sorry. It's my mistake. I din't set content type. After checking it again:

Issue 1:
Data mounted on docker:

services:
  storage:
    volumes:
          - /Users/ri8ika/Documents/data:/Users/ri8ika/Documents/data

And yes, I was using wrong data path in ENTRYPOINT by mistake. Now, it shows error while starting the server. Not sure about /tmp, it's created by running server by golang automatically, right? Can you please help me fix on this?

Issue 2: (Has issue)
I have set content type to "video/mp4"
Not able to play video.

Issue 3: (Issue persist)
Maybe fixing issue 1 may help this?

Issue 4: (No issue)
Ah, application/pdf worked fine, I was using octet stream - my mistake.
Same mistake did with svg file - din't define content type.

Issue 5: (Issue persist)
Content type is set to "audio/mpeg"
Also tried with "audio/mp3"

I am fresher in Golang. Will contribute anytime if I can do something. Thanks.

@ri8ika commented on GitHub (Feb 15, 2022): Oops, sorry. It's my mistake. I din't set content type. After checking it again: Issue 1: Data mounted on docker: ``` services: storage: volumes: - /Users/ri8ika/Documents/data:/Users/ri8ika/Documents/data ``` And yes, I was using wrong data path in ENTRYPOINT by mistake. Now, it shows error while starting the server. Not sure about `/tmp`, it's created by running server by golang automatically, right? Can you please help me fix on this? Issue 2: (Has issue) I have set content type to "video/mp4" Not able to play video. Issue 3: (Issue persist) Maybe fixing issue 1 may help this? Issue 4: (No issue) Ah, application/pdf worked fine, I was using octet stream - my mistake. Same mistake did with svg file - din't define content type. Issue 5: (Issue persist) Content type is set to "audio/mpeg" Also tried with "audio/mp3" I am fresher in Golang. Will contribute anytime if I can do something. Thanks.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@gaul commented on GitHub (Feb 16, 2022):

I believe that issue 1 relates to the lack of macOS Docker filesystem sharing (osxfs): https://docker-docs.netlify.app/docker-for-mac/osxfs/#extended-attributes. You can try the referenced PR that communicates errors more completely.

Your experience suggests that the Windows fallback logic might be useful for macOS and I guess Linux tmpfs as well.

@gaul commented on GitHub (Feb 16, 2022): I believe that issue 1 relates to the lack of macOS Docker filesystem sharing (osxfs): https://docker-docs.netlify.app/docker-for-mac/osxfs/#extended-attributes. You can try the referenced PR that communicates errors more completely. Your experience suggests that the Windows fallback logic might be useful for macOS and I guess Linux tmpfs as well.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@gaul commented on GitHub (Feb 16, 2022):

You might try running fake-gcs-server -data to copy from the macOS bind mount into a directory inside the Docker image as a workaround.

@gaul commented on GitHub (Feb 16, 2022): You might try running `fake-gcs-server -data` to copy from the macOS bind mount into a directory inside the Docker image as a workaround.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@ri8ika commented on GitHub (Feb 16, 2022):

Didn't work:

server started at http://[::]:4443

Linux tmpfs as well.

rootDir does not support user extended attributes:
Consider using a different filesystem type, e.g., not tmpfs on Linux

I think we need to create a different issue for solving Issue 3, it's about persisting data between normal server start and start with docker. And work fixing issue 2 and 5 here. I am currently trying to figure out why video and audio is not being played in browser.

@ri8ika commented on GitHub (Feb 16, 2022): Didn't work: ``` server started at http://[::]:4443 ``` > Linux tmpfs as well. ``` rootDir does not support user extended attributes: Consider using a different filesystem type, e.g., not tmpfs on Linux ``` I think we need to create a different issue for solving Issue 3, it's about persisting data between normal server start and start with docker. And work fixing issue 2 and 5 here. I am currently trying to figure out why video and audio is not being played in browser.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@gaul commented on GitHub (Feb 17, 2022):

@ri8ika Could you file an issue for oxsfs xattr support at https://github.com/docker/for-mac/issues and link to it here? I lack the correct setup and this requires inputting some live diagnostic metadata or else they will just close the issue.

@gaul commented on GitHub (Feb 17, 2022): @ri8ika Could you file an issue for oxsfs xattr support at https://github.com/docker/for-mac/issues and link to it here? I lack the correct setup and this requires inputting some live diagnostic metadata or else they will just close the issue.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@ri8ika commented on GitHub (Feb 17, 2022):

I am not pretty sure about diagnosing. I would be glad if you could guide me on this? Can we talk somewhere on Slack?

I tried self diagnose tool, but failed: https://docs.docker.com/desktop/mac/troubleshoot/#self-diagnose-tool

Failed with: vm has not started: vm has not started

Should I install VM or what steps need to take?
Ouch, they have some issue here: https://github.com/docker/for-mac/issues/6091

@ri8ika commented on GitHub (Feb 17, 2022): I am not pretty sure about diagnosing. I would be glad if you could guide me on this? Can we talk somewhere on Slack? I tried self diagnose tool, but failed: https://docs.docker.com/desktop/mac/troubleshoot/#self-diagnose-tool > Failed with: vm has not started: vm has not started Should I install VM or what steps need to take? Ouch, they have some issue here: https://github.com/docker/for-mac/issues/6091

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@fsouza commented on GitHub (Feb 17, 2022):

Hmm interesting. In my use cases I'd usually not mount the volume from Docker (I also haven't used docker4mac in a while). If this is a common use case, perhaps we should generalize the Windows fallback, but no need to go for that now I think, we can watch for other issues / comments from other users.

@fsouza commented on GitHub (Feb 17, 2022): Hmm interesting. In my use cases I'd usually not mount the volume from Docker (I also haven't used docker4mac in a while). If this is a common use case, perhaps we should generalize the Windows fallback, but no need to go for that now I think, we can watch for other issues / comments from other users.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@ri8ika commented on GitHub (Feb 17, 2022):

Yup! You are right. We can ignore docker issue for sometime. But what about the issue of video/audio not playing in browser. I don't think this should be ignored. There could be simple solution for this like fixing header or so.

@ri8ika commented on GitHub (Feb 17, 2022): Yup! You are right. We can ignore docker issue for sometime. But what about the issue of video/audio not playing in browser. I don't think this should be ignored. There could be simple solution for this like fixing header or so.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@fsouza commented on GitHub (Feb 17, 2022):

@ri8ika oh yeah absolutely. We can fix it to return whatever GCS returns. Curious if GCS does it based on metadata or automatically.

@fsouza commented on GitHub (Feb 17, 2022): @ri8ika oh yeah absolutely. We can fix it to return whatever GCS returns. Curious if GCS does it based on metadata or automatically.

kerem referenced this issue

2026-03-03 12:29:36 +03:00

[PR #117] [MERGED] build(deps): bump @google-cloud/storage from 4.0.1 to 4.1.0 in /examples/node #350

Rows
Columns

[GH-ISSUE #684] uploaded file is crashed #117