[GH-ISSUE #669] Efficiently handle large objects #116

New issue

Open

opened 2026-03-03 12:08:26 +03:00 by kerem · 2 comments

kerem commented

2026-03-03 12:08:26 +03:00

Owner

Originally created by @gaul on GitHub (Feb 2, 2022).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/669

Currently multipartUpload and uploadFileContent calls loadContent which calls ioutil.ReadAll which loads the entire request body into memory. Instead it should stream the body to disk when using the filesystem backend. This will reduce memory usage when testing large objects, e.g., 2 GB.

Originally created by @gaul on GitHub (Feb 2, 2022). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/669 Currently `multipartUpload` and `uploadFileContent` calls `loadContent` which calls `ioutil.ReadAll` which loads the entire request body into memory. Instead it should stream the body to disk when using the filesystem backend. This will reduce memory usage when testing large objects, e.g., 2 GB.

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@fsouza commented on GitHub (Feb 2, 2022):

@gaul do you think we can close this now that #670 is merged and #671 is open to track the move away from JSON?

@fsouza commented on GitHub (Feb 2, 2022): @gaul do you think we can close this now that #670 is merged and #671 is open to track the move away from JSON?

kerem commented

2026-03-03 12:08:27 +03:00

Author

Owner

@gaul commented on GitHub (Feb 3, 2022):

These issues relate but have a somewhat different scope. This one should include any memory optimization for large files, e.g., removing the call to append(obj.Content) in uploadFileContent which doubles memory usage.

@gaul commented on GitHub (Feb 3, 2022): These issues relate but have a somewhat different scope. This one should include any memory optimization for large files, e.g., removing the call to `append(obj.Content)` in `uploadFileContent` which doubles memory usage.