[GH-ISSUE #1898] Pubsub Integration Sanity Check #225

New issue

Open

opened 2026-03-03 12:09:17 +03:00 by kerem · 2 comments

kerem commented

2026-03-03 12:09:17 +03:00

Owner

Originally created by @ihandley-efx on GitHub (Feb 13, 2025).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1898

Can someone please verify fake-gcs-server still publishes notification messages to a pubsub emulator?

I've scoured this repo looking for information:

I followed the thread discussion with @Ortega-Dan (see)
I followed the thread started by @rogierlommers (see)
I looked over the notification module

I just can't get fake-gcs-server to publish a message.

I'm doing this in Docker; here is the relevant portion of my docker-compose.yml:

storage:
    image: fsouza/fake-gcs-server:latest
    depends_on:
      pubsub-init:
        condition: service_completed_successfully
    volumes:
      - ./integration_tests/cloudstorage/:/data
    environment:
      PUBSUB_EMULATOR_HOST: localhost:8085
    ports:
      - '0.0.0.0:9023:9023'
    command: >
      -data=/data
      -scheme=http
      -port=9023
      -event.pubsub-project-id=test-project
      -event.pubsub-topic=test-topic
    healthcheck:
      test: wget --no-verbose --tries=1 --spider localhost:9023/_internal/healthcheck
        || exit 1
      interval: 10s
      timeout: 10s
      retries: 5
      start_period: 10s
    platform: linux/amd64

  pubsub:
    image: us-docker.pkg.dev/iaas-gcr-reg-prd-ad3d/build/google/google-toolbox-emulator:1.0
    environment:
      GCP_PROJECT_ID: test-project
    ports:
      - '0.0.0.0:8085:8085'
    command: >
      bash -c '
        gcloud beta emulators pubsub start --host-port=0.0.0.0:8085 --project=$$GCP_PROJECT_ID
      '
    healthcheck:
      test: curl --fail http://localhost:8085/
      interval: 10s
      timeout: 10s
      retries: 15
      start_period: 10s
    platform: linux/amd64

  pubsub-init:
    image: us-docker.pkg.dev/iaas-gcr-reg-prd-ad3d/build/google/google-toolbox-emulator:1.0
    depends_on:
      pubsub:
        condition: service_healthy
    environment:
      GCP_PROJECT_ID: test-project
      GCP_PUBSUB_SUBSCRIPTION_ID: test-subscription
      GCP_PUBSUB_TOPIC_ID: test-topic
      PUBSUB_EMULATOR_HOST: pubsub:8085
    command: >
      bash -c '
        curl -f -H "Content-Type: application/json" -X PUT "http://$$PUBSUB_EMULATOR_HOST/v1/projects/$$GCP_PROJECT_ID/topics/$$GCP_PUBSUB_TOPIC_ID"
        curl -f -H "Content-Type: application/json" -X PUT -d "{
            \"topic\": \"projects/$$GCP_PROJECT_ID/topics/$$GCP_PUBSUB_TOPIC_ID\"
        }" "http://$$PUBSUB_EMULATOR_HOST/v1/projects/$$GCP_PROJECT_ID/subscriptions/$$GCP_PUBSUB_SUBSCRIPTION_ID"
      '
    platform: linux/amd64

The test I'm running writes an object to the fake-gcs-server's bucket, and it seems to be doing that as seen here in the logs:

2025-02-13 06:57:31 storage-1      | time=2025-02-13T13:57:31.096Z level=INFO msg="172.20.0.1 - - [13/Feb/2025:13:57:31 +0000] \"POST /upload/storage/v1/b/test-bucket/o?alt=json&name=test-rdr-file.csv&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1\" 200 823\n"
2025-02-13 06:57:34 storage-1      | time=2025-02-13T13:57:34.747Z level=INFO msg="::1 - - [13/Feb/2025:13:57:34 +0000] \"GET /_internal/healthcheck HTTP/1.1\" 200 0\n"

I don't see any log message(s) written to indicate a message has been published nor has my pubsub listener picked anything up.

Any ideas what I'm missing here? Thanks in advance!

Originally created by @ihandley-efx on GitHub (Feb 13, 2025). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/1898 Can someone please verify `fake-gcs-server` still publishes notification messages to a pubsub emulator? I've scoured this repo looking for information: - I followed the thread discussion with @Ortega-Dan ([see](https://github.com/fsouza/fake-gcs-server/pull/490#issuecomment-1546489345)) - I followed the thread started by @rogierlommers ([see](https://github.com/fsouza/fake-gcs-server/issues/24#issue-426691652)) - I looked over the [notification module](https://github.com/fsouza/fake-gcs-server/tree/main/internal/notification) I just can't get `fake-gcs-server` to publish a message. I'm doing this in Docker; here is the relevant portion of my `docker-compose.yml`: ``` storage: image: fsouza/fake-gcs-server:latest depends_on: pubsub-init: condition: service_completed_successfully volumes: - ./integration_tests/cloudstorage/:/data environment: PUBSUB_EMULATOR_HOST: localhost:8085 ports: - '0.0.0.0:9023:9023' command: > -data=/data -scheme=http -port=9023 -event.pubsub-project-id=test-project -event.pubsub-topic=test-topic healthcheck: test: wget --no-verbose --tries=1 --spider localhost:9023/_internal/healthcheck || exit 1 interval: 10s timeout: 10s retries: 5 start_period: 10s platform: linux/amd64 pubsub: image: us-docker.pkg.dev/iaas-gcr-reg-prd-ad3d/build/google/google-toolbox-emulator:1.0 environment: GCP_PROJECT_ID: test-project ports: - '0.0.0.0:8085:8085' command: > bash -c ' gcloud beta emulators pubsub start --host-port=0.0.0.0:8085 --project=$$GCP_PROJECT_ID ' healthcheck: test: curl --fail http://localhost:8085/ interval: 10s timeout: 10s retries: 15 start_period: 10s platform: linux/amd64 pubsub-init: image: us-docker.pkg.dev/iaas-gcr-reg-prd-ad3d/build/google/google-toolbox-emulator:1.0 depends_on: pubsub: condition: service_healthy environment: GCP_PROJECT_ID: test-project GCP_PUBSUB_SUBSCRIPTION_ID: test-subscription GCP_PUBSUB_TOPIC_ID: test-topic PUBSUB_EMULATOR_HOST: pubsub:8085 command: > bash -c ' curl -f -H "Content-Type: application/json" -X PUT "http://$$PUBSUB_EMULATOR_HOST/v1/projects/$$GCP_PROJECT_ID/topics/$$GCP_PUBSUB_TOPIC_ID" curl -f -H "Content-Type: application/json" -X PUT -d "{ \"topic\": \"projects/$$GCP_PROJECT_ID/topics/$$GCP_PUBSUB_TOPIC_ID\" }" "http://$$PUBSUB_EMULATOR_HOST/v1/projects/$$GCP_PROJECT_ID/subscriptions/$$GCP_PUBSUB_SUBSCRIPTION_ID" ' platform: linux/amd64 ``` The test I'm running writes an object to the `fake-gcs-server`'s bucket, and it seems to be doing that as seen here in the logs: ``` 2025-02-13 06:57:31 storage-1 | time=2025-02-13T13:57:31.096Z level=INFO msg="172.20.0.1 - - [13/Feb/2025:13:57:31 +0000] \"POST /upload/storage/v1/b/test-bucket/o?alt=json&name=test-rdr-file.csv&prettyPrint=false&projection=full&uploadType=multipart HTTP/1.1\" 200 823\n" 2025-02-13 06:57:34 storage-1 | time=2025-02-13T13:57:34.747Z level=INFO msg="::1 - - [13/Feb/2025:13:57:34 +0000] \"GET /_internal/healthcheck HTTP/1.1\" 200 0\n" ``` I don't see any log message(s) written to indicate a message has been published nor has my pubsub listener picked anything up. Any ideas what I'm missing here? Thanks in advance!

kerem commented

2026-03-03 12:09:18 +03:00

Author

Owner

@zimkha commented on GitHub (Mar 17, 2025):

Hello, I see on your storage You have set PUBSUB_EMULATOR_HOST: localhost:8085 in the storage container. localhost in this context refers to the storage container itself, not the container. You can try to change the "localhost" on pusub in your storage container

@zimkha commented on GitHub (Mar 17, 2025): Hello, I see on your storage You have set PUBSUB_EMULATOR_HOST: localhost:8085 in the storage container. localhost in this context refers to the storage container itself, not the container. You can try to change the "localhost" on pusub in your storage container

kerem commented

2026-03-03 12:09:18 +03:00

Author

Owner

@Goldziher commented on GitHub (Jun 4, 2025):

Same issue. This appears to be a real bug.

@Goldziher commented on GitHub (Jun 4, 2025): Same issue. This appears to be a real bug.