[GH-ISSUE #828] Efficiently handle large objects when downloading #136

New issue

Closed

opened 2026-03-03 12:08:36 +03:00 by kerem · 0 comments

kerem commented

2026-03-03 12:08:36 +03:00

Owner

Originally created by @dnerdy on GitHub (Jun 15, 2022).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/828

This issue is like #669 except that it's specific to downloading. When downloading an object fake-gcs-server loads an entire object into memory. This can be problematic when there are concurrent requests of large objects. It's especially problematic when there are concurrent range reads. We have code that performs a bunch of small concurrent range reads on a large file, and responses that are at most a few kilobytes in size cause tens of gigabytes of data to be loaded into memory.

Originally created by @dnerdy on GitHub (Jun 15, 2022). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/828 This issue is like #669 except that it's specific to downloading. When downloading an object fake-gcs-server loads an entire object into memory. This can be problematic when there are concurrent requests of large objects. It's especially problematic when there are concurrent range reads. We have code that performs a bunch of small concurrent range reads on a large file, and responses that are at most a few kilobytes in size cause tens of gigabytes of data to be loaded into memory.