[PR #60] [MERGED] Initial object versioning support - retention of old generations if version is enabled, and getObject by generation #304

New issue

Closed

opened 2026-03-03 12:10:11 +03:00 by kerem · 0 comments

kerem commented

2026-03-03 12:10:11 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/fsouza/fake-gcs-server/pull/60
Author: @dcaba
Created: 8/13/2019
Status: ✅ Merged
Merged: 12/12/2019
Merged by: @fsouza

Base: master ← Head: BasicVersioningSupport

📝 Commits (10+)

1cf1190 Object supporting updated/created/deleted date, as first step to support versions
c3f7ce0 Dockerfile reproducing with stages what drone is doing the CI, so we can reproduce locally. fmt-ed the files to pass the tests
7049962 rebase conflict issue - back to main golang image to run tests
9f2196c rebase conflict issue - back to main golang image to run tests
a53df2f attr related tests refactor
4b186ae creation and modification dates should be generated and returned if not provided as inputs. Generation now also part of the output, following the same policy
cb1e0c7 some IaC state/config files we dont want to commit or embed in docker images
dba2d10 this allows us to create a bucket, for some extra manual tests
79a7c6c fmt, so docker build now passes
76b75b4 unifing gitignore. renaming some functions

📊 Changes

16 files changed (+980 additions, -340 deletions)

View changed files

📝 .dockerignore (+1 -0)
📝 Dockerfile (+19 -3)
📝 fakestorage/bucket.go (+37 -10)
📝 fakestorage/bucket_test.go (+71 -34)
📝 fakestorage/object.go (+74 -2)
📝 fakestorage/object_test.go (+354 -133)
📝 fakestorage/response.go (+27 -13)
📝 fakestorage/upload.go (+1 -1)
📝 fakestorage/upload_test.go (+5 -5)
📝 internal/backend/backend_test.go (+169 -76)
➕ internal/backend/bucket.go (+7 -0)
📝 internal/backend/fs.go (+18 -6)
📝 internal/backend/memory.go (+150 -51)
📝 internal/backend/object.go (+42 -2)
📝 internal/backend/storage.go (+4 -3)
📝 main.go (+1 -1)

📄 Description

Finally the first version of the versioning support is ready for review. First I will describe what has been covered, a functionality I want to add really soon (object listing that considers archived versions) and the ones I dont expect to cover in short term (I don't need them yet, and I think is not critical functionality to enjoy versioning), and I will add inline comments to highlight some decisions I've made that probably are worth to bear in mind during the review.

What we have covered:

Related to versioning implementation:
- We have targeted the in memory backend, extending the backend interface: fs backend will not store the new fields, and will refuse/fail on any operation that targets versioning or specific generations
- Object storing updated/created/deleted dates, we can create or just inherit from inputs (but delete time, that will be always consistent)
- Return consistent ObjectAttrs.Generation
- Bucket versioning flag (BucketAttrs.VersioningEnabled) available for the in memory engine, and modifying the CreateBucket function (with a mandatory parameter, versioning enabled or not), and also implying a bucket is no longer just the bucketName, but an object with several fields.
  - Bucket listing / props in bucket level responses considering the versioning status
- Keeping multiple object versions when bucket versioning is enabled, returning always the last
- Implement deletion with bucket versioning is enabled, returning consistent results (archived vs live versions)
- Implement lookups of specific generations, that can target archived version
- Returning a consistent object ID (that should contain name, bucket and generation)
Non related to the functionality, but accelerates testing:
- Execute linting+tests as part of the Dockerbuild
  - This may be rolled back depending on the owner point of view: we are replicating, and sadly duplicating, logic we have in CI to reproduce tests locally

In the next PR we expect to cover:

Implement object listing with Query.Versions = true

Things we expect not cover under this PR neither short term (if the previous implementation does not require to proceed in the end)

Deletion of specific generations / archived objects
Adding support to Conditions (in order to perform conditional operations as described here ), specifically GenerationMatch
Bucket metageneration management
Object metageneration management
lifecycles rules
Support of versions in the filesystem backend (that can be fun...)

Thanks for your patience, thanks so much for your project and hope this does not generate a big headache.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/fsouza/fake-gcs-server/pull/60 **Author:** [@dcaba](https://github.com/dcaba) **Created:** 8/13/2019 **Status:** ✅ Merged **Merged:** 12/12/2019 **Merged by:** [@fsouza](https://github.com/fsouza) **Base:** `master` ← **Head:** `BasicVersioningSupport` --- ### 📝 Commits (10+) - [`1cf1190`](https://github.com/fsouza/fake-gcs-server/commit/1cf1190c1ee28d0f6083cfa37865929eddef5ef7) Object supporting updated/created/deleted date, as first step to support versions - [`c3f7ce0`](https://github.com/fsouza/fake-gcs-server/commit/c3f7ce0c44216a26d6e512078891c4ac26aeb507) Dockerfile reproducing with stages what drone is doing the CI, so we can reproduce locally. fmt-ed the files to pass the tests - [`7049962`](https://github.com/fsouza/fake-gcs-server/commit/704996294da05b024c3dd19f3fd1426d766a97ff) rebase conflict issue - back to main golang image to run tests - [`9f2196c`](https://github.com/fsouza/fake-gcs-server/commit/9f2196c5869b0880d618436036f98243173bd5c3) rebase conflict issue - back to main golang image to run tests - [`a53df2f`](https://github.com/fsouza/fake-gcs-server/commit/a53df2f2f4d026e1f6ae0348351a0f603113cc76) attr related tests refactor - [`4b186ae`](https://github.com/fsouza/fake-gcs-server/commit/4b186aed425a44ae71504ca27ceebb885701ff3d) creation and modification dates should be generated and returned if not provided as inputs. Generation now also part of the output, following the same policy - [`cb1e0c7`](https://github.com/fsouza/fake-gcs-server/commit/cb1e0c71fb7df6fb8f1970d95e89d864c73f0f0d) some IaC state/config files we dont want to commit or embed in docker images - [`dba2d10`](https://github.com/fsouza/fake-gcs-server/commit/dba2d106f778d8d6ce846aa4f31e3a8d70d555e6) this allows us to create a bucket, for some extra manual tests - [`79a7c6c`](https://github.com/fsouza/fake-gcs-server/commit/79a7c6cf616d4e51ef37eccdee2571cc397d20d2) fmt, so docker build now passes - [`76b75b4`](https://github.com/fsouza/fake-gcs-server/commit/76b75b4bdb10dba5912b80b781c93bb5a182ad5d) unifing gitignore. renaming some functions ### 📊 Changes **16 files changed** (+980 additions, -340 deletions) <details> <summary>View changed files</summary> 📝 `.dockerignore` (+1 -0) 📝 `Dockerfile` (+19 -3) 📝 `fakestorage/bucket.go` (+37 -10) 📝 `fakestorage/bucket_test.go` (+71 -34) 📝 `fakestorage/object.go` (+74 -2) 📝 `fakestorage/object_test.go` (+354 -133) 📝 `fakestorage/response.go` (+27 -13) 📝 `fakestorage/upload.go` (+1 -1) 📝 `fakestorage/upload_test.go` (+5 -5) 📝 `internal/backend/backend_test.go` (+169 -76) ➕ `internal/backend/bucket.go` (+7 -0) 📝 `internal/backend/fs.go` (+18 -6) 📝 `internal/backend/memory.go` (+150 -51) 📝 `internal/backend/object.go` (+42 -2) 📝 `internal/backend/storage.go` (+4 -3) 📝 `main.go` (+1 -1) </details> ### 📄 Description Finally the first version of the versioning support is ready for review. First I will describe what has been covered, a functionality I want to add really soon (object listing that considers archived versions) and the ones I dont expect to cover in short term (I don't need them yet, and I think is not critical functionality to enjoy versioning), and I will add inline comments to highlight some decisions I've made that probably are worth to bear in mind during the review. What we have covered: * Related to versioning implementation: * We have targeted the in memory backend, extending the backend interface: fs backend will not store the new fields, and will refuse/fail on any operation that targets versioning or specific generations * Object storing updated/created/deleted dates, we can create or just inherit from inputs (but delete time, that will be always consistent) * Return consistent ObjectAttrs.Generation * Bucket versioning flag (BucketAttrs.VersioningEnabled) available for the in memory engine, and modifying the CreateBucket function (with a mandatory parameter, versioning enabled or not), and also implying a bucket is no longer just the bucketName, but an object with several fields. * Bucket listing / props in bucket level responses considering the versioning status * Keeping multiple object versions when bucket versioning is enabled, returning always the last * Implement deletion with bucket versioning is enabled, returning consistent results (archived vs live versions) * Implement lookups of specific generations, that can target archived version * Returning a consistent object ID (that should contain name, bucket and generation) * Non related to the functionality, but accelerates testing: * Execute linting+tests as part of the Dockerbuild * This may be rolled back depending on the owner point of view: we are replicating, and sadly duplicating, logic we have in CI to reproduce tests locally In the next PR we expect to cover: * Implement object listing with Query.Versions = true Things we expect not cover under this PR neither short term (if the previous implementation does not require to proceed in the end) * Deletion of specific generations / archived objects * Adding support to Conditions (in order to perform conditional operations as described [here](https://godoc.org/cloud.google.com/go/storage#hdr-Conditions) ), specifically GenerationMatch * Bucket metageneration management * Object metageneration management * lifecycles rules * Support of versions in the filesystem backend (that can be fun...) Thanks for your patience, thanks so much for your project and hope this does not generate a big headache. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>