[PR #882] [CLOSED] Streaming downloads #1002

New issue

Closed

opened 2026-03-03 12:32:36 +03:00 by kerem · 0 comments

kerem commented

2026-03-03 12:32:36 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/fsouza/fake-gcs-server/pull/882
Author: @StevenACoffman
Created: 8/4/2022
Status: ❌ Closed

Base: main ← Head: streaming-downloads

📝 Commits (10+)

165bcd3 Stream object downloads instead of buffering object in memory
57d084c Fix typo in xattr_windows.go
ec9f547 Fix staticcheck lint errors.
beded88 Attempt to fix renaming over open files on Windows
c7d6f1d Add build directive to rename_unix.go
0cb0899 Close objects when listing objects and in tests
af77c57 Use compatOpenForWritingAllowingRename for temp file
4639463 Close temp file
71201ca Move the Windows "temporary delete path" to the temp dir
cbb36c8 Close initial objects

📊 Changes

21 files changed (+667 additions, -235 deletions)

View changed files

📝 .gitignore (+1 -0)
📝 fakestorage/object.go (+158 -65)
📝 fakestorage/object_test.go (+31 -17)
📝 fakestorage/server.go (+2 -2)
📝 fakestorage/server_test.go (+7 -4)
📝 fakestorage/upload.go (+41 -41)
📝 fakestorage/upload_test.go (+38 -9)
📝 go.mod (+1 -0)
📝 go.sum (+2 -0)
📝 internal/backend/backend_test.go (+36 -13)
📝 internal/backend/fs.go (+127 -47)
📝 internal/backend/memory.go (+49 -22)
📝 internal/backend/object.go (+38 -0)
➕ internal/backend/rename_unix.go (+21 -0)
➕ internal/backend/rename_windows.go (+54 -0)
📝 internal/backend/storage.go (+6 -6)
📝 internal/backend/xattr_unix.go (+4 -0)
📝 internal/backend/xattr_windows.go (+4 -0)
📝 internal/checksum/checksum.go (+29 -0)
📝 internal/notification/event.go (+5 -5)

...and 1 more files

📄 Description

Fixes #828. (This is #829 with the review requests applied)

This PR makes it so objects in the FS backend are streamed from disk instead of being loaded into memory. This drastically reduces memory usage when working with large files, especially when performing small concurrent range reads on a large file.

A StreamingObject type has been added to both the fakestorage and backend packages. This type contains a io.ReadSeekCloser that can be used to access object data.

This change sets the stage for fixing #669 and #397. #669 isn't fixed because it will require re-working of how resumable uploads are stored (though other upload types have been made streaming in this PR). Also, fixing #397 would require changing how initial objects are specified when starting the server, which with this approach would require changing the public API.

Regarding the public API: the server methods for working with objects have been changed to use StreamingObject instead of Object. This is a breaking change. It would be possible to make streaming backward compatible by adding some metadata and methods to Object (to distinguish between streaming and buffered objects), but I went with this approach because it made identifying the code that needed to be updated easier; the compiler pointed out the locations to update instead of needing to track down runtime and test errors. An additional improvement, which is compatible with the backward compatible approach, would be to add an OpenForReading method on Object so callers that are currently responsible for closing objects also explicitly open objects (which would make the responsibility clearer). Anyway, I wanted to get this initial approach up for feedback before putting in additional work.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/fsouza/fake-gcs-server/pull/882 **Author:** [@StevenACoffman](https://github.com/StevenACoffman) **Created:** 8/4/2022 **Status:** ❌ Closed **Base:** `main` ← **Head:** `streaming-downloads` --- ### 📝 Commits (10+) - [`165bcd3`](https://github.com/fsouza/fake-gcs-server/commit/165bcd3944849ae2055166e7edc36673feeeffc5) Stream object downloads instead of buffering object in memory - [`57d084c`](https://github.com/fsouza/fake-gcs-server/commit/57d084c9129b5700cabe76a27207227e3ac59eb9) Fix typo in xattr_windows.go - [`ec9f547`](https://github.com/fsouza/fake-gcs-server/commit/ec9f547011873a0a0fa6d74cafcc6c8800b0c3ad) Fix staticcheck lint errors. - [`beded88`](https://github.com/fsouza/fake-gcs-server/commit/beded88b7206fb3b4bc825f4a7c266c4ff2ddc78) Attempt to fix renaming over open files on Windows - [`c7d6f1d`](https://github.com/fsouza/fake-gcs-server/commit/c7d6f1d7356ad8217986ecf0293d7194bebfcce6) Add build directive to rename_unix.go - [`0cb0899`](https://github.com/fsouza/fake-gcs-server/commit/0cb0899a84780a0e403165a403c0efe694f87a39) Close objects when listing objects and in tests - [`af77c57`](https://github.com/fsouza/fake-gcs-server/commit/af77c571f8169f700f7a3ff3c7ecc9ad5d3fcc2b) Use compatOpenForWritingAllowingRename for temp file - [`4639463`](https://github.com/fsouza/fake-gcs-server/commit/463946347b64bcfd513655d25c30740c03a79867) Close temp file - [`71201ca`](https://github.com/fsouza/fake-gcs-server/commit/71201ca1eeb92416606430665faf48c3bb8d516b) Move the Windows "temporary delete path" to the temp dir - [`cbb36c8`](https://github.com/fsouza/fake-gcs-server/commit/cbb36c82f00e9ff083eaa3ebb8aa20d7716420d4) Close initial objects ### 📊 Changes **21 files changed** (+667 additions, -235 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+1 -0) 📝 `fakestorage/object.go` (+158 -65) 📝 `fakestorage/object_test.go` (+31 -17) 📝 `fakestorage/server.go` (+2 -2) 📝 `fakestorage/server_test.go` (+7 -4) 📝 `fakestorage/upload.go` (+41 -41) 📝 `fakestorage/upload_test.go` (+38 -9) 📝 `go.mod` (+1 -0) 📝 `go.sum` (+2 -0) 📝 `internal/backend/backend_test.go` (+36 -13) 📝 `internal/backend/fs.go` (+127 -47) 📝 `internal/backend/memory.go` (+49 -22) 📝 `internal/backend/object.go` (+38 -0) ➕ `internal/backend/rename_unix.go` (+21 -0) ➕ `internal/backend/rename_windows.go` (+54 -0) 📝 `internal/backend/storage.go` (+6 -6) 📝 `internal/backend/xattr_unix.go` (+4 -0) 📝 `internal/backend/xattr_windows.go` (+4 -0) 📝 `internal/checksum/checksum.go` (+29 -0) 📝 `internal/notification/event.go` (+5 -5) _...and 1 more files_ </details> ### 📄 Description Fixes #828. (This is #829 with the review requests applied) This PR makes it so objects in the FS backend are streamed from disk instead of being loaded into memory. This drastically reduces memory usage when working with large files, especially when performing small concurrent range reads on a large file. A StreamingObject type has been added to both the fakestorage and backend packages. This type contains a io.ReadSeekCloser that can be used to access object data. This change sets the stage for fixing #669 and #397. #669 isn't fixed because it will require re-working of how resumable uploads are stored (though other upload types have been made streaming in this PR). Also, fixing #397 would require changing how initial objects are specified when starting the server, which with this approach would require changing the public API. Regarding the public API: the server methods for working with objects have been changed to use StreamingObject instead of Object. This is a breaking change. It would be possible to make streaming backward compatible by adding some metadata and methods to Object (to distinguish between streaming and buffered objects), but I went with this approach because it made identifying the code that needed to be updated easier; the compiler pointed out the locations to update instead of needing to track down runtime and test errors. An additional improvement, which is compatible with the backward compatible approach, would be to add an OpenForReading method on Object so callers that are currently responsible for closing objects also explicitly open objects (which would make the responsibility clearer). Anyway, I wanted to get this initial approach up for feedback before putting in additional work. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>