[PR #2056] [fix] respect content-range header in resumable upload #2054

New issue

Open

opened 2026-03-03 13:29:34 +03:00 by kerem · 0 comments

kerem commented

2026-03-03 13:29:34 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/fsouza/fake-gcs-server/pull/2056
Author: @JornEngelbart
Created: 10/31/2025
Status: 🔄 Open

Base: main ← Head: respect-content-range-header-in-resumable-upload

📝 Commits (1)

1f3322e [fix] respect content-range header in resumable upload

📊 Changes

2 files changed (+257 additions, -13 deletions)

View changed files

📝 fakestorage/upload.go (+54 -13)
📝 fakestorage/upload_test.go (+203 -0)

📄 Description

Fix Content-Range header handling in resumable uploads

Problem

The resumable upload implementation was not correctly handling the Content-Range header when uploading chunks. The code was simply appending all chunks sequentially (obj.Content = append(obj.Content, content...)), which caused incorrect behavior when:

Overlapping ranges: Clients may re-upload the same byte range (e.g., for retry scenarios)
Out-of-order chunks: Though rare, chunks might arrive out of sequence
Non-sequential uploads: Some clients upload chunks that don't start at position 0

This resulted in:

Content being placed at incorrect byte positions
Duplicate content when ranges overlapped
Incorrect file sizes and corruption

Solution

The fix properly parses and respects the Content-Range header to place content at the exact byte positions specified:

Key Changes

Parse Content-Range before appending: The code now parses the Content-Range header first to determine where content should be placed

Place content at specified positions: Instead of appending, content is copied to the exact byte range specified:

// For known ranges (e.g., "bytes 0-262143/377464")
copy(obj.Content[contentStart:contentStart+len(content)], content)

Handle overlapping ranges: When the same range is uploaded multiple times, the content correctly overwrites the previous content at those positions
Extend buffer as needed: The buffer is automatically extended with zero-filled bytes when a range requires a larger buffer size
Validate content length: Added validation to ensure the actual content length matches the Content-Range header specification

Example Scenario

Previously, if a client sent:

Request 1: Content-Range: bytes 0-262143/377464 (262144 bytes)
Request 2: Content-Range: bytes 1-262144/377464 (262144 bytes, overlapping)
Request 3: Content-Range: bytes 1-377463/377464 (377463 bytes, overlapping)

The old code would append all chunks sequentially, resulting in an incorrect file.

The new code correctly:

Places first chunk at positions 0-262143
Places second chunk at positions 1-262144 (overwriting positions 1-262143 and adding new byte at 262144)
Places third chunk at positions 1-377463 (overwriting and extending as needed)
Final result: Correct file with all bytes in the right positions

Testing

Added comprehensive test coverage in TestResumableUploadWithContentRange that verifies:

Overlapping range uploads (same range uploaded twice)
Sequential non-overlapping range uploads
Proper content placement at specified byte positions
Checksum validation of final content

Impact

This fix ensures that fake-gcs-server correctly handles resumable uploads with Content-Range headers, making it compatible with clients that:

Retry uploads with overlapping ranges
Upload chunks in non-sequential order
Use the Content-Range header to specify exact byte positions

The fix is backward compatible and maintains the existing behavior for uploads without Content-Range headers (falls back to append behavior).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/fsouza/fake-gcs-server/pull/2056 **Author:** [@JornEngelbart](https://github.com/JornEngelbart) **Created:** 10/31/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `respect-content-range-header-in-resumable-upload` --- ### 📝 Commits (1) - [`1f3322e`](https://github.com/fsouza/fake-gcs-server/commit/1f3322efc2da269b0a0148d24c48a92a61360bba) [fix] respect content-range header in resumable upload ### 📊 Changes **2 files changed** (+257 additions, -13 deletions) <details> <summary>View changed files</summary> 📝 `fakestorage/upload.go` (+54 -13) 📝 `fakestorage/upload_test.go` (+203 -0) </details> ### 📄 Description # Fix Content-Range header handling in resumable uploads ## Problem The resumable upload implementation was not correctly handling the `Content-Range` header when uploading chunks. The code was simply appending all chunks sequentially (`obj.Content = append(obj.Content, content...)`), which caused incorrect behavior when: 1. **Overlapping ranges**: Clients may re-upload the same byte range (e.g., for retry scenarios) 2. **Out-of-order chunks**: Though rare, chunks might arrive out of sequence 3. **Non-sequential uploads**: Some clients upload chunks that don't start at position 0 This resulted in: - Content being placed at incorrect byte positions - Duplicate content when ranges overlapped - Incorrect file sizes and corruption ## Solution The fix properly parses and respects the `Content-Range` header to place content at the exact byte positions specified: ### Key Changes 1. **Parse Content-Range before appending**: The code now parses the `Content-Range` header first to determine where content should be placed 2. **Place content at specified positions**: Instead of appending, content is copied to the exact byte range specified: ```go // For known ranges (e.g., "bytes 0-262143/377464") copy(obj.Content[contentStart:contentStart+len(content)], content) ``` 3. **Handle overlapping ranges**: When the same range is uploaded multiple times, the content correctly overwrites the previous content at those positions 4. **Extend buffer as needed**: The buffer is automatically extended with zero-filled bytes when a range requires a larger buffer size 5. **Validate content length**: Added validation to ensure the actual content length matches the Content-Range header specification ### Example Scenario Previously, if a client sent: - Request 1: `Content-Range: bytes 0-262143/377464` (262144 bytes) - Request 2: `Content-Range: bytes 1-262144/377464` (262144 bytes, overlapping) - Request 3: `Content-Range: bytes 1-377463/377464` (377463 bytes, overlapping) The old code would append all chunks sequentially, resulting in an incorrect file. The new code correctly: - Places first chunk at positions 0-262143 - Places second chunk at positions 1-262144 (overwriting positions 1-262143 and adding new byte at 262144) - Places third chunk at positions 1-377463 (overwriting and extending as needed) - Final result: Correct file with all bytes in the right positions ## Testing Added comprehensive test coverage in `TestResumableUploadWithContentRange` that verifies: - Overlapping range uploads (same range uploaded twice) - Sequential non-overlapping range uploads - Proper content placement at specified byte positions - Checksum validation of final content ## Impact This fix ensures that fake-gcs-server correctly handles resumable uploads with Content-Range headers, making it compatible with clients that: - Retry uploads with overlapping ranges - Upload chunks in non-sequential order - Use the Content-Range header to specify exact byte positions The fix is backward compatible and maintains the existing behavior for uploads without Content-Range headers (falls back to append behavior). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>