[GH-ISSUE #270] Internal Error when trying to upload with signed URLs #2204

New issue

Closed

opened 2026-03-15 18:00:57 +03:00 by kerem · 4 comments

kerem commented

2026-03-15 18:00:57 +03:00

Owner

Originally created by @ex-nerd on GitHub (Jun 13, 2020).
Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/270

Extracted from my (now-removed) comment on #217 since it seems to be related, but not identical to that problem.

I'm trying to mimic signed upload URLs. I know my code (with a URL generated by GCS/KMS) works against actual GCS but attempts to generate a URL that will work with fake-gcs-server results in Internal Error.

My replication code looks something like this:

import requests

session = requests.Session()
session.verify = False

resp = session.post(
    "https://my-public-uri.local:4443/my-bucket/my-test-key-to-create?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=fake-gcs&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=fake-gcs'
    files={
        "file": (
            "my-test-key",
            f"this is an upload test",
            "text/plain",
        )
    },
)
print(resp.status_code)

This results in a response with an error message claiming invalid character '-' in numeric literal\n

I've tracked this down to a problem in loadMetadata expecting some form of JSON data but my actual body looks like this:

--ea7516aedd980cfb6672d9ed6301a60b\r\nContent-Disposition: form-data; name="file"; filename="my-test-key"\r\nContent-Type: text/plain\r\n\r\nthis is an upload test\r\n--ea7516aedd980cfb6672d9ed6301a60b--\r\n

If I add &uploadType=multipart to my URL to force this into a multipart upload, the error changes to invalid character 'h' in literal true (expecting 'r')\n from the if err != io.EOF check in multipartUpload().

P.S. It'd be really handy if there was a way to do a stack trace for those errors ... I was lazy and added a bunch of prefix text to the error messages to figure out which was which, but it was a bit tedious.

Originally created by @ex-nerd on GitHub (Jun 13, 2020). Original GitHub issue: https://github.com/fsouza/fake-gcs-server/issues/270 Extracted from my (now-removed) comment on #217 since it seems to be related, but not identical to that problem. I'm trying to mimic signed upload URLs. I know my code (with a URL generated by GCS/KMS) works against actual GCS but attempts to generate a URL that will work with fake-gcs-server results in `Internal Error`. My replication code looks something like this: ```python import requests session = requests.Session() session.verify = False resp = session.post( "https://my-public-uri.local:4443/my-bucket/my-test-key-to-create?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=fake-gcs&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=fake-gcs' files={ "file": ( "my-test-key", f"this is an upload test", "text/plain", ) }, ) print(resp.status_code) ``` This results in a response with an error message claiming `invalid character '-' in numeric literal\n` I've tracked this down to a problem in `loadMetadata` expecting some form of JSON data but my actual body looks like this: ``` --ea7516aedd980cfb6672d9ed6301a60b\r\nContent-Disposition: form-data; name="file"; filename="my-test-key"\r\nContent-Type: text/plain\r\n\r\nthis is an upload test\r\n--ea7516aedd980cfb6672d9ed6301a60b--\r\n ``` If I add `&uploadType=multipart` to my URL to force this into a multipart upload, the error changes to `invalid character 'h' in literal true (expecting 'r')\n` from the `if err != io.EOF` check in `multipartUpload()`. P.S. It'd be really handy if there was a way to do a stack trace for those errors ... I was lazy and added a bunch of prefix text to the error messages to figure out which was which, but it was a bit tedious.

kerem

2026-03-15 18:00:57 +03:00

closed this issue
added the
bug

help wanted
labels

kerem commented

2026-03-15 18:01:13 +03:00

Author

Owner

@ex-nerd commented on GitHub (Jun 13, 2020):

One other minor point. GCS returns a 204 for signed POST requests, not a 200

@ex-nerd commented on GitHub (Jun 13, 2020): One other minor point. GCS returns a `204` for signed POST requests, not a `200`

kerem commented

2026-03-15 18:01:18 +03:00

Author

Owner

@ex-nerd commented on GitHub (Jun 17, 2020):

Additionally, the upload seems to be pulling the key from the name query string property (similarly to media) or from the file metadata, rather than from the URL path itself.

A standard signed URL request looks like this:

https://storage.googleapis.com/my-bucket-name/test-file-for-signed-urls?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=<MY-CREDENTIALS>&X-Goog-Date=20200617T035632Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=<LONG-SIGNATURE-HERE>

I assume the key name is embedded in the signature (at least for verification purposes), but the bucket and key name are also right there on the URL. It would make much more sense for fake-gcs-server to honor the key name on the URL path rather than require a separate nonstandard property that GCS seems happy to ignore.

Note: I discovered this issue when using uploadType=media as a workaround for this bug, so that may also need similar behavior (I don't know how that particular uploadType works in actual GCS).

@ex-nerd commented on GitHub (Jun 17, 2020): Additionally, the upload seems to be pulling the `key` from the `name` query string property (similarly to `media`) or from the file metadata, rather than from the URL path itself. A standard signed URL request looks like this: ``` https://storage.googleapis.com/my-bucket-name/test-file-for-signed-urls?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=<MY-CREDENTIALS>&X-Goog-Date=20200617T035632Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=<LONG-SIGNATURE-HERE> ``` I assume the key name is embedded in the signature (at least for verification purposes), but the bucket and key name are also right there on the URL. It would make much more sense for fake-gcs-server to honor the key name on the URL path rather than require a separate nonstandard property that GCS seems happy to ignore. _Note: I discovered this issue when using `uploadType=media` as a workaround for this bug, so that may also need similar behavior (I don't know how that particular `uploadType` works in actual GCS)._

kerem commented

2026-03-15 18:01:24 +03:00

Author

Owner

@Sytten commented on GitHub (Jun 28, 2021):

Same error here, is there a work around except the media?
The JSON body is really not standard, the form is the usual way of sending the parameters.

@Sytten commented on GitHub (Jun 28, 2021): Same error here, is there a work around except the media? The JSON body is really not standard, the form is the usual way of sending the parameters.

kerem commented

2026-03-15 18:01:29 +03:00

Author

Owner

@Sytten commented on GitHub (Jun 29, 2021):

Ok I understand it is because we use the form upload which is in the XML API: https://cloud.google.com/storage/docs/xml-api/post-object-forms#html

@Sytten commented on GitHub (Jun 29, 2021): Ok I understand it is because we use the form upload which is in the XML API: https://cloud.google.com/storage/docs/xml-api/post-object-forms#html