[GH-ISSUE #1854] Unable to list bucket with alpakka client lib due to missing field(s) #223

New issue

Open

opened 2026-03-03 12:09:15 +03:00 by kerem · 2 comments

kerem commented

2026-03-03 12:09:15 +03:00

Owner

Originally created by @marcuslinke on GitHub (Dec 17, 2024).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1854

I've uploaded some objects into a test bucket with the latest release version v1.50.2 and try to list the test bucket contents via alpakka client lib (https://doc.akka.io/libraries/alpakka/current/google-cloud-storage.html#list-bucket-contents). The lib assumes the following structure for the "acl" field:

  private final case class ObjectAccessControls(kind: String,
                                                id: String,
                                                selfLink: String,
                                                bucket: String,
                                                `object`: String,
                                                generation: String,
                                                entity: String,
                                                role: String,
                                                email: String,
                                                entityId: String,
                                                domain: String,
                                                projectTeam: ProjectTeam,
                                                etag: String)

but its missing some of these fields like "id" for example in the response and therefore throws an exception. Do I miss something? The official gcs documentation for this object type is here: https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls#resource-representations

Response from the fake-gcs-server:

{
  "items" : [ {
    "acl" : [ {
      "bucket" : "test_bucket",
      "entity" : "projectOwner-test-project",
      "etag" : "RVRhZw==",
      "kind" : "storage#objectAccessControl",
      "object" : "abwicklungsmengen/DataForAggregationTesting_2022-01-03-00-00-00.txt",
      "projectTeam" : { },
      "role" : "OWNER"
    } ],
    "bucket" : "test_bucket",
    "contentType" : "application/octet-stream",
    "crc32c" : "4rJcjA==",
    "etag" : "CS14W2cpr698UlqnbhH0hQ==",
    "generation" : "1734441772622141",
    "id" : "test_bucket/abwicklungsmengen/DataForAggregationTesting_2022-01-03-00-00-00.txt",
    "kind" : "storage#object",
    "md5Hash" : "CS14W2cpr698UlqnbhH0hQ==",
    "mediaLink" : "http://0.0.0.0:4443/download/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-03-00-00-00.txt?alt=media",
    "metageneration" : "1",
    "name" : "abwicklungsmengen/DataForAggregationTesting_2022-01-03-00-00-00.txt",
    "selfLink" : "http://0.0.0.0:4443/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-03-00-00-00.txt",
    "size" : "138",
    "storageClass" : "STANDARD",
    "timeCreated" : "2024-12-17T13:22:52.622135Z",
    "timeStorageClassUpdated" : "2024-12-17T13:22:52.62214Z",
    "updated" : "2024-12-17T13:22:52.62214Z"
  }, {
    "acl" : [ {
      "bucket" : "test_bucket",
      "entity" : "projectOwner-test-project",
      "etag" : "RVRhZw==",
      "kind" : "storage#objectAccessControl",
      "object" : "abwicklungsmengen/DataForAggregationTesting_2022-01-04-00-00-00.txt",
      "projectTeam" : { },
      "role" : "OWNER"
    } ],
    "bucket" : "test_bucket",
    "contentType" : "application/octet-stream",
    "crc32c" : "q/rcDw==",
    "etag" : "3iYL8URA0SdAj/hX7JYTIw==",
    "generation" : "1734441772635009",
    "id" : "test_bucket/abwicklungsmengen/DataForAggregationTesting_2022-01-04-00-00-00.txt",
    "kind" : "storage#object",
    "md5Hash" : "3iYL8URA0SdAj/hX7JYTIw==",
    "mediaLink" : "http://0.0.0.0:4443/download/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-04-00-00-00.txt?alt=media",
    "metageneration" : "1",
    "name" : "abwicklungsmengen/DataForAggregationTesting_2022-01-04-00-00-00.txt",
    "selfLink" : "http://0.0.0.0:4443/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-04-00-00-00.txt",
    "size" : "137",
    "storageClass" : "STANDARD",
    "timeCreated" : "2024-12-17T13:22:52.635004Z",
    "timeStorageClassUpdated" : "2024-12-17T13:22:52.635008Z",
    "updated" : "2024-12-17T13:22:52.635008Z"
  }, {
    "acl" : [ {
      "bucket" : "test_bucket",
      "entity" : "projectOwner-test-project",
      "etag" : "RVRhZw==",
      "kind" : "storage#objectAccessControl",
      "object" : "abwicklungsmengen/LMA_AWM_Export_2022-11-05-00-00-00.txt",
      "projectTeam" : { },
      "role" : "OWNER"
    } ],
    "bucket" : "test_bucket",
    "contentType" : "application/octet-stream",
    "crc32c" : "JilXew==",
    "etag" : "Dye/u06a+x/r3G6L7gvw6g==",
    "generation" : "1734441772597178",
    "id" : "test_bucket/abwicklungsmengen/LMA_AWM_Export_2022-11-05-00-00-00.txt",
    "kind" : "storage#object",
    "md5Hash" : "Dye/u06a+x/r3G6L7gvw6g==",
    "mediaLink" : "http://0.0.0.0:4443/download/storage/v1/b/test_bucket/o/abwicklungsmengen%2FLMA_AWM_Export_2022-11-05-00-00-00.txt?alt=media",
    "metageneration" : "1",
    "name" : "abwicklungsmengen/LMA_AWM_Export_2022-11-05-00-00-00.txt",
    "selfLink" : "http://0.0.0.0:4443/storage/v1/b/test_bucket/o/abwicklungsmengen%2FLMA_AWM_Export_2022-11-05-00-00-00.txt",
    "size" : "144",
    "storageClass" : "STANDARD",
    "timeCreated" : "2024-12-17T13:22:52.597174Z",
    "timeStorageClassUpdated" : "2024-12-17T13:22:52.597178Z",
    "updated" : "2024-12-17T13:22:52.597178Z"
  } ],
  "kind" : "storage#objects"
}

Originally created by @marcuslinke on GitHub (Dec 17, 2024). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1854 I've uploaded some objects into a test bucket with the latest release version v1.50.2 and try to list the test bucket contents via alpakka client lib (https://doc.akka.io/libraries/alpakka/current/google-cloud-storage.html#list-bucket-contents). The lib assumes the following structure for the "acl" field: ``` private final case class ObjectAccessControls(kind: String, id: String, selfLink: String, bucket: String, `object`: String, generation: String, entity: String, role: String, email: String, entityId: String, domain: String, projectTeam: ProjectTeam, etag: String) ``` but its missing some of these fields like "id" for example in the response and therefore throws an exception. Do I miss something? The official gcs documentation for this object type is here: https://cloud.google.com/storage/docs/json_api/v1/bucketAccessControls#resource-representations Response from the fake-gcs-server: ``` { "items" : [ { "acl" : [ { "bucket" : "test_bucket", "entity" : "projectOwner-test-project", "etag" : "RVRhZw==", "kind" : "storage#objectAccessControl", "object" : "abwicklungsmengen/DataForAggregationTesting_2022-01-03-00-00-00.txt", "projectTeam" : { }, "role" : "OWNER" } ], "bucket" : "test_bucket", "contentType" : "application/octet-stream", "crc32c" : "4rJcjA==", "etag" : "CS14W2cpr698UlqnbhH0hQ==", "generation" : "1734441772622141", "id" : "test_bucket/abwicklungsmengen/DataForAggregationTesting_2022-01-03-00-00-00.txt", "kind" : "storage#object", "md5Hash" : "CS14W2cpr698UlqnbhH0hQ==", "mediaLink" : "http://0.0.0.0:4443/download/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-03-00-00-00.txt?alt=media", "metageneration" : "1", "name" : "abwicklungsmengen/DataForAggregationTesting_2022-01-03-00-00-00.txt", "selfLink" : "http://0.0.0.0:4443/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-03-00-00-00.txt", "size" : "138", "storageClass" : "STANDARD", "timeCreated" : "2024-12-17T13:22:52.622135Z", "timeStorageClassUpdated" : "2024-12-17T13:22:52.62214Z", "updated" : "2024-12-17T13:22:52.62214Z" }, { "acl" : [ { "bucket" : "test_bucket", "entity" : "projectOwner-test-project", "etag" : "RVRhZw==", "kind" : "storage#objectAccessControl", "object" : "abwicklungsmengen/DataForAggregationTesting_2022-01-04-00-00-00.txt", "projectTeam" : { }, "role" : "OWNER" } ], "bucket" : "test_bucket", "contentType" : "application/octet-stream", "crc32c" : "q/rcDw==", "etag" : "3iYL8URA0SdAj/hX7JYTIw==", "generation" : "1734441772635009", "id" : "test_bucket/abwicklungsmengen/DataForAggregationTesting_2022-01-04-00-00-00.txt", "kind" : "storage#object", "md5Hash" : "3iYL8URA0SdAj/hX7JYTIw==", "mediaLink" : "http://0.0.0.0:4443/download/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-04-00-00-00.txt?alt=media", "metageneration" : "1", "name" : "abwicklungsmengen/DataForAggregationTesting_2022-01-04-00-00-00.txt", "selfLink" : "http://0.0.0.0:4443/storage/v1/b/test_bucket/o/abwicklungsmengen%2FDataForAggregationTesting_2022-01-04-00-00-00.txt", "size" : "137", "storageClass" : "STANDARD", "timeCreated" : "2024-12-17T13:22:52.635004Z", "timeStorageClassUpdated" : "2024-12-17T13:22:52.635008Z", "updated" : "2024-12-17T13:22:52.635008Z" }, { "acl" : [ { "bucket" : "test_bucket", "entity" : "projectOwner-test-project", "etag" : "RVRhZw==", "kind" : "storage#objectAccessControl", "object" : "abwicklungsmengen/LMA_AWM_Export_2022-11-05-00-00-00.txt", "projectTeam" : { }, "role" : "OWNER" } ], "bucket" : "test_bucket", "contentType" : "application/octet-stream", "crc32c" : "JilXew==", "etag" : "Dye/u06a+x/r3G6L7gvw6g==", "generation" : "1734441772597178", "id" : "test_bucket/abwicklungsmengen/LMA_AWM_Export_2022-11-05-00-00-00.txt", "kind" : "storage#object", "md5Hash" : "Dye/u06a+x/r3G6L7gvw6g==", "mediaLink" : "http://0.0.0.0:4443/download/storage/v1/b/test_bucket/o/abwicklungsmengen%2FLMA_AWM_Export_2022-11-05-00-00-00.txt?alt=media", "metageneration" : "1", "name" : "abwicklungsmengen/LMA_AWM_Export_2022-11-05-00-00-00.txt", "selfLink" : "http://0.0.0.0:4443/storage/v1/b/test_bucket/o/abwicklungsmengen%2FLMA_AWM_Export_2022-11-05-00-00-00.txt", "size" : "144", "storageClass" : "STANDARD", "timeCreated" : "2024-12-17T13:22:52.597174Z", "timeStorageClassUpdated" : "2024-12-17T13:22:52.597178Z", "updated" : "2024-12-17T13:22:52.597178Z" } ], "kind" : "storage#objects" } ```

kerem added the

compatibility-issue

label

2026-03-03 12:09:15 +03:00

kerem commented

2026-03-03 12:09:16 +03:00

Author

Owner

@fsouza commented on GitHub (Jan 4, 2025):

Yeah I think we're missing those fields. Marking as a compat issue.

@fsouza commented on GitHub (Jan 4, 2025): Yeah I think we're missing those fields. Marking as a compat issue.

kerem commented

2026-03-03 12:09:17 +03:00

Author

Owner

@mdedetrich commented on GitHub (Mar 30, 2025):

Also experiencing this when adding support to fake-gcs-server in the pekko-connectors google cloud storage client

The future returned an exception of type: spray.json.DeserializationException, with message: Object is missing required member 'selfLink'.

which was meant to be solved in https://github.com/fsouza/fake-gcs-server/issues/758 but I guess the field wasn't universally added in all responses? In this specific case its on creation of buckets.

I created an issue at https://github.com/fsouza/fake-gcs-server/issues/1943

@mdedetrich commented on GitHub (Mar 30, 2025): Also experiencing this when adding support to fake-gcs-server in the pekko-connectors google cloud storage client ``` The future returned an exception of type: spray.json.DeserializationException, with message: Object is missing required member 'selfLink'. ``` which was meant to be solved in https://github.com/fsouza/fake-gcs-server/issues/758 but I guess the field wasn't universally added in all responses? In this specific case its on creation of buckets. I created an issue at https://github.com/fsouza/fake-gcs-server/issues/1943

kerem referenced this issue

2026-03-03 12:30:00 +03:00

[PR #223] [MERGED] build(deps): bump google-cloud-storage from 1.27.0 to 1.28.0 in /examples/python #437