starred/fake-gcs-server

Fork 0

mirror of https://github.com/fsouza/fake-gcs-server.git synced 2026-04-25 13:45:52 +03:00

[GH-ISSUE #199] python gcs client will throw ValueError because of wrong timestamp format #2181

New issue

Closed

opened 2026-03-15 17:55:53 +03:00 by kerem · 5 comments

kerem commented

2026-03-15 17:55:53 +03:00

Owner

Originally created by @kc-panw on GitHub (Mar 20, 2020).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/199

The python GCS client somehow takes its format very similar to RFC3339 but not exactly the same so it will throw.

The format it uses is actually:

"2006-01-02T15:04:05.999999Z07:00"

not

time.RFC3339

would this be fixed?

Originally created by @kc-panw on GitHub (Mar 20, 2020). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/199 The python GCS client somehow takes its format very similar to RFC3339 but not exactly the same so it will throw. The format it uses is actually: `"2006-01-02T15:04:05.999999Z07:00"` not `time.RFC3339` would this be fixed?

kerem closed this issue

2026-03-15 17:55:58 +03:00

kerem commented

2026-03-15 17:56:03 +03:00

Author

Owner

@fsouza commented on GitHub (Mar 21, 2020):

@kc-panw hey there, thanks for reporting this issue. It can definitely be fixed. Can you provide a reproducer, just so we can write a test to prevent regressions?

Thanks!

@fsouza commented on GitHub (Mar 21, 2020): @kc-panw hey there, thanks for reporting this issue. It can definitely be fixed. Can you provide a reproducer, just so we can write a test to prevent regressions? Thanks!

kerem commented

2026-03-15 17:56:08 +03:00

Author

Owner

@linjun9 commented on GitHub (May 7, 2020):

@fsouza Yea, I'm getting the same problem. You can reproduce by running the following sample code.

The problem is it using RFC3339Microsecond in python GCS client, but it's using RFC3339 in fake-gcs-server. It requires following format 2006-01-02T15:04:05.999999Z07:00. Thanks in advance.

import os

import requests
import urllib3
from google.api_core.client_options import ClientOptions
from google.auth.credentials import AnonymousCredentials
from google.cloud import storage

EXTERNAL_URL = os.getenv("EXTERNAL_URL", "https://127.0.0.1:4443")
PUBLIC_HOST = os.getenv("PUBLIC_HOST", "storage.gcs.127.0.0.1.nip.io:4443")

storage.blob._API_ACCESS_ENDPOINT = "https://" + PUBLIC_HOST
storage.blob._DOWNLOAD_URL_TEMPLATE = (
    u"%s/download/storage/v1{path}?alt=media" % EXTERNAL_URL
)
storage.blob._BASE_UPLOAD_TEMPLATE = (
    u"%s/upload/storage/v1{bucket_path}/o?uploadType=" % EXTERNAL_URL
)
storage.blob._MULTIPART_URL_TEMPLATE = storage.blob._BASE_UPLOAD_TEMPLATE + u"multipart"
storage.blob._RESUMABLE_URL_TEMPLATE = storage.blob._BASE_UPLOAD_TEMPLATE + u"resumable"

my_http = requests.Session()
my_http.verify = False  # disable SSL validation
urllib3.disable_warnings(
    urllib3.exceptions.InsecureRequestWarning
)  # disable https warnings for https insecure certs

client = storage.Client(
    credentials=AnonymousCredentials(),
    project="test",
    _http=my_http,
    client_options=ClientOptions(api_endpoint=EXTERNAL_URL),
)


bucket = client.bucket('sample-bucket')
b = bucket.get_blob('test_file.txt')
s = b.download_to_filename('/test.txt')

@linjun9 commented on GitHub (May 7, 2020): @fsouza Yea, I'm getting the same problem. You can reproduce by running the following sample code. The problem is it using RFC3339Microsecond in python GCS client, but it's using RFC3339 in `fake-gcs-server`. It requires following format `2006-01-02T15:04:05.999999Z07:00`. Thanks in advance. ``` import os import requests import urllib3 from google.api_core.client_options import ClientOptions from google.auth.credentials import AnonymousCredentials from google.cloud import storage EXTERNAL_URL = os.getenv("EXTERNAL_URL", "https://127.0.0.1:4443") PUBLIC_HOST = os.getenv("PUBLIC_HOST", "storage.gcs.127.0.0.1.nip.io:4443") storage.blob._API_ACCESS_ENDPOINT = "https://" + PUBLIC_HOST storage.blob._DOWNLOAD_URL_TEMPLATE = ( u"%s/download/storage/v1{path}?alt=media" % EXTERNAL_URL ) storage.blob._BASE_UPLOAD_TEMPLATE = ( u"%s/upload/storage/v1{bucket_path}/o?uploadType=" % EXTERNAL_URL ) storage.blob._MULTIPART_URL_TEMPLATE = storage.blob._BASE_UPLOAD_TEMPLATE + u"multipart" storage.blob._RESUMABLE_URL_TEMPLATE = storage.blob._BASE_UPLOAD_TEMPLATE + u"resumable" my_http = requests.Session() my_http.verify = False # disable SSL validation urllib3.disable_warnings( urllib3.exceptions.InsecureRequestWarning ) # disable https warnings for https insecure certs client = storage.Client( credentials=AnonymousCredentials(), project="test", _http=my_http, client_options=ClientOptions(api_endpoint=EXTERNAL_URL), ) bucket = client.bucket('sample-bucket') b = bucket.get_blob('test_file.txt') s = b.download_to_filename('/test.txt') ```

kerem commented

2026-03-15 17:56:14 +03:00

Author

Owner

@fsouza commented on GitHub (May 11, 2020):

@linjun9 awesome, thanks for the reproducer! The fix for this is small (use ``time.RFC3339Nanoinstead oftime.RFC3339`), but writing a test may be trickier. I can give it a shot later this week.

@fsouza commented on GitHub (May 11, 2020): @linjun9 awesome, thanks for the reproducer! The fix for this is small (use ``time.RFC3339Nano` instead of `time.RFC3339`), but writing a test may be trickier. I can give it a shot later this week.

kerem commented

2026-03-15 17:56:19 +03:00

Author

Owner

@linjun9 commented on GitHub (May 11, 2020):

@fsouza I tryied time.RFC3339Nano. It doesn't work, since RFC3339Nano gives us 9 decimal places, we only need 6 decimal places. Not a Golang expert, but yea, feel free to try it out. Thanks 👍

@linjun9 commented on GitHub (May 11, 2020): @fsouza I tryied `time.RFC3339Nano`. It doesn't work, since `RFC3339Nano` gives us 9 decimal places, we only need 6 decimal places. Not a Golang expert, but yea, feel free to try it out. Thanks 👍

kerem commented

2026-03-15 17:56:24 +03:00

Author

Owner

@chrisK824 commented on GitHub (May 11, 2024):

Hi @fsouza , thanks for this awesome project!

I have stumbled on the above issue. I am not sure if this has been re-introduced at some stage due to python client changes, but it is breaking in my trials. I am using python3.12 and latest version (1.49) of the emulator.

Here is a reproducing script:

from google.api_core.client_options import ClientOptions
from google.auth.credentials import AnonymousCredentials
from google.cloud import storage


client = storage.Client(
    credentials=AnonymousCredentials(),
    project="test",
    client_options=ClientOptions(api_endpoint="http://localhost:6000"),
)


bucket = client.create_bucket('sample-bucket')
b = bucket.blob("test")

with b.open('wb') as f:
    f.write('test'.encode())

updated_b = bucket.get_blob('test')

print(updated_b.updated)

this spits out:

➜  example python3.9 example.py
Traceback (most recent call last):
  File "/Users/chriskarvouniaris/Desktop/example/example.py", line 26, in <module>
    print(updated_b.updated)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 4224, in updated
    return _rfc3339_nanos_to_datetime(value)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/cloud/_helpers/__init__.py", line 273, in _rfc3339_nanos_to_datetime
    raise ValueError(
ValueError: Timestamp: '2024-05-11T22:29:31.335823+03:00', does not match pattern: '\n    (?P<no_fraction>\n        \\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}  # YYYY-MM-DDTHH:MM:SS\n    )\n    (                                        # Optional decimal part\n     \\.                                      # decimal point\n     (?P<nanos>\\d{1,9})                      # nanoseconds, maybe truncated\n    )?\n    Z

@chrisK824 commented on GitHub (May 11, 2024): Hi @fsouza , thanks for this awesome project! I have stumbled on the above issue. I am not sure if this has been re-introduced at some stage due to python client changes, but it is breaking in my trials. I am using python3.12 and latest version (1.49) of the emulator. Here is a reproducing script: ``` from google.api_core.client_options import ClientOptions from google.auth.credentials import AnonymousCredentials from google.cloud import storage client = storage.Client( credentials=AnonymousCredentials(), project="test", client_options=ClientOptions(api_endpoint="http://localhost:6000"), ) bucket = client.create_bucket('sample-bucket') b = bucket.blob("test") with b.open('wb') as f: f.write('test'.encode()) updated_b = bucket.get_blob('test') print(updated_b.updated) ``` this spits out: ``` ➜ example python3.9 example.py Traceback (most recent call last): File "/Users/chriskarvouniaris/Desktop/example/example.py", line 26, in <module> print(updated_b.updated) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 4224, in updated return _rfc3339_nanos_to_datetime(value) File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/cloud/_helpers/__init__.py", line 273, in _rfc3339_nanos_to_datetime raise ValueError( ValueError: Timestamp: '2024-05-11T22:29:31.335823+03:00', does not match pattern: '\n (?P<no_fraction>\n \\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2} # YYYY-MM-DDTHH:MM:SS\n )\n ( # Optional decimal part\n \\. # decimal point\n (?P<nanos>\\d{1,9}) # nanoseconds, maybe truncated\n )?\n Z ```