[PR #829] [MERGED] Stream object downloads instead of buffering objects in memory #951

New issue

Closed

opened 2026-03-03 12:32:22 +03:00 by kerem · 0 comments

kerem commented

2026-03-03 12:32:22 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/fsouza/fake-gcs-server/pull/829
Author: @dnerdy
Created: 6/15/2022
Status: ✅ Merged
Merged: 9/2/2022
Merged by: @fsouza

Base: main ← Head: streaming-downloads

📝 Commits (10+)

0cd6c3b Stream object downloads instead of buffering object in memory
1c2962e Fix typo in xattr_windows.go
f354f3d Fix staticcheck lint errors.
07f8045 Attempt to fix renaming over open files on Windows
1c9ecbb Add build directive to rename_unix.go
d0e24d3 Close objects when listing objects and in tests
3d78ac9 Use compatOpenForWritingAllowingRename for temp file
227228f Close temp file
b15c8a5 Move the Windows "temporary delete path" to the temp dir
4f18d8d Close initial objects

📊 Changes

19 files changed (+653 additions, -219 deletions)

View changed files

📝 fakestorage/object.go (+200 -75)
📝 fakestorage/server.go (+2 -2)
📝 fakestorage/server_test.go (+7 -4)
📝 fakestorage/upload.go (+41 -41)
📝 go.mod (+1 -0)
📝 go.sum (+15 -0)
📝 internal/backend/backend_test.go (+36 -13)
📝 internal/backend/fs.go (+127 -47)
📝 internal/backend/memory.go (+49 -22)
📝 internal/backend/metadata.go (+1 -0)
📝 internal/backend/metadata_file.go (+4 -0)
📝 internal/backend/metadata_xattr.go (+4 -0)
📝 internal/backend/object.go (+38 -0)
➕ internal/backend/rename_unix.go (+21 -0)
➕ internal/backend/rename_windows.go (+54 -0)
📝 internal/backend/storage.go (+6 -6)
📝 internal/checksum/checksum.go (+29 -0)
📝 internal/notification/event.go (+5 -5)
📝 internal/notification/event_test.go (+13 -4)

📄 Description

Fixes #828.

This PR makes it so objects in the FS backend are streamed from disk
instead of being loaded into memory. This drastically reduces memory
usage when working with large files, especially when performing small
concurrent range reads on a large file.

A StreamingObject type has been added to both the fakestorage and
backend packages. This type contains a io.ReadSeekCloser that can be
used to access object data.

This change sets the stage for fixing #669 and #397. #669 isn't fixed
because it will require re-working of how resumable uploads are stored
(though other upload types have been made streaming in this PR). Also,
fixing #397 would require changing how initial objects are specified
when starting the server, which with this approach would require
changing the public API.

Regarding the public API: the server methods for working with objects
have been changed to use StreamingObject instead of Object. This is a
breaking change. It would be possible to make streaming backward
compatible by adding some metadata and methods to Object (to distinguish
between streaming and buffered objects), but I went with this approach
because it made identifying the code that needed to be updated easier;
the compiler pointed out the locations to update instead of needing to
track down runtime and test errors. An additional improvement, which is
compatible with the backward compatible approach, would be to add an
OpenForReading method on Object so callers that are currently
responsible for closing objects also explicitly open objects (which
would make the responsibility clearer). Anyway, I wanted to get this
initial approach up for feedback before putting in additional work.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/fsouza/fake-gcs-server/pull/829 **Author:** [@dnerdy](https://github.com/dnerdy) **Created:** 6/15/2022 **Status:** ✅ Merged **Merged:** 9/2/2022 **Merged by:** [@fsouza](https://github.com/fsouza) **Base:** `main` ← **Head:** `streaming-downloads` --- ### 📝 Commits (10+) - [`0cd6c3b`](https://github.com/fsouza/fake-gcs-server/commit/0cd6c3b30bcd2beef4321008faa4c29c7d505363) Stream object downloads instead of buffering object in memory - [`1c2962e`](https://github.com/fsouza/fake-gcs-server/commit/1c2962e284c88897b9536743e078e2b4bce98d7c) Fix typo in xattr_windows.go - [`f354f3d`](https://github.com/fsouza/fake-gcs-server/commit/f354f3d35686590203a0be1f70b02bad372519fe) Fix staticcheck lint errors. - [`07f8045`](https://github.com/fsouza/fake-gcs-server/commit/07f8045ed7e8d401b3a234528efc38ca74ccc4c7) Attempt to fix renaming over open files on Windows - [`1c9ecbb`](https://github.com/fsouza/fake-gcs-server/commit/1c9ecbb8f91295bf33c21a80a10b6b0344483f61) Add build directive to rename_unix.go - [`d0e24d3`](https://github.com/fsouza/fake-gcs-server/commit/d0e24d3b2ce0ebc4f056e14321964919f68dd636) Close objects when listing objects and in tests - [`3d78ac9`](https://github.com/fsouza/fake-gcs-server/commit/3d78ac91f3acc32f49a70e0d6e95238e8578a9db) Use compatOpenForWritingAllowingRename for temp file - [`227228f`](https://github.com/fsouza/fake-gcs-server/commit/227228ffa7db0beded8e3f073e024721bae47259) Close temp file - [`b15c8a5`](https://github.com/fsouza/fake-gcs-server/commit/b15c8a566285941e8917f3bc91dfaa2a8ab8a68b) Move the Windows "temporary delete path" to the temp dir - [`4f18d8d`](https://github.com/fsouza/fake-gcs-server/commit/4f18d8d56db399d5388a1c42d6f3133e6bce7be6) Close initial objects ### 📊 Changes **19 files changed** (+653 additions, -219 deletions) <details> <summary>View changed files</summary> 📝 `fakestorage/object.go` (+200 -75) 📝 `fakestorage/server.go` (+2 -2) 📝 `fakestorage/server_test.go` (+7 -4) 📝 `fakestorage/upload.go` (+41 -41) 📝 `go.mod` (+1 -0) 📝 `go.sum` (+15 -0) 📝 `internal/backend/backend_test.go` (+36 -13) 📝 `internal/backend/fs.go` (+127 -47) 📝 `internal/backend/memory.go` (+49 -22) 📝 `internal/backend/metadata.go` (+1 -0) 📝 `internal/backend/metadata_file.go` (+4 -0) 📝 `internal/backend/metadata_xattr.go` (+4 -0) 📝 `internal/backend/object.go` (+38 -0) ➕ `internal/backend/rename_unix.go` (+21 -0) ➕ `internal/backend/rename_windows.go` (+54 -0) 📝 `internal/backend/storage.go` (+6 -6) 📝 `internal/checksum/checksum.go` (+29 -0) 📝 `internal/notification/event.go` (+5 -5) 📝 `internal/notification/event_test.go` (+13 -4) </details> ### 📄 Description Fixes #828. This PR makes it so objects in the FS backend are streamed from disk instead of being loaded into memory. This drastically reduces memory usage when working with large files, especially when performing small concurrent range reads on a large file. A StreamingObject type has been added to both the fakestorage and backend packages. This type contains a io.ReadSeekCloser that can be used to access object data. This change sets the stage for fixing #669 and #397. #669 isn't fixed because it will require re-working of how resumable uploads are stored (though other upload types have been made streaming in this PR). Also, fixing #397 would require changing how initial objects are specified when starting the server, which with this approach would require changing the public API. Regarding the public API: the server methods for working with objects have been changed to use StreamingObject instead of Object. This is a breaking change. It would be possible to make streaming backward compatible by adding some metadata and methods to Object (to distinguish between streaming and buffered objects), but I went with this approach because it made identifying the code that needed to be updated easier; the compiler pointed out the locations to update instead of needing to track down runtime and test errors. An additional improvement, which is compatible with the backward compatible approach, would be to add an OpenForReading method on Object so callers that are currently responsible for closing objects also explicitly open objects (which would make the responsibility clearer). Anyway, I wanted to get this initial approach up for feedback before putting in additional work. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>