[GH-ISSUE #2085] File copies using multipart upload fail with SSE-C #1057

Closed
opened 2026-03-04 01:51:01 +03:00 by kerem · 5 comments
Owner

Originally created by @icassassin on GitHub (Jan 3, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2085

Additional Information

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.91 (commit:14eb1a7) with OpenSSL
Also tried with v1.90 that comes with Ubuntu 22.04; same issue

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

example: 2.9.9

Kernel information (uname -r)

5.15.0-56-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

Ubuntu 22.04.1 LTS

How to run s3fs, if applicable

from /etc/fstab:
[bucket name] [mount point] fuse.s3fs _netdev,allow_other,url=https://s3.wasabisys.com,passwd_file=,use_sse=custom:,use_path_request_style,dbglevel=info 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

I think the relevant syslog lines are:
Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl_multi.cpp:MultiRead(228): failed a request(400: https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH)
Jan 3 20:41:06 rdserv-staging s3fs[10246]: [tpath=/Godot_v3.5.1-stable_x11.64.zip][start=20971520][size=18752962][part=3]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: URL is https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH
Jan 3 20:41:06 rdserv-staging s3fs[10246]: URL changed is https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH
Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=1&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [89915b33b6ee9af99dacd4950c1c8fa543ffad962d887d5e97c7c58d4fa5eef1]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com
Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=2&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [63e1cc395ec600502bdb8809060272bb58fa0f745ee30512fd42be4059d463eb]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com
Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [aaad05e698850c86508eba9b86fbe6b6f210d020794039aa909f9090d3c575a3]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com
Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: #012InvalidRequestThe provided encryption parameters did not match the ones used originally.61385D6BF18725E2akP7Ctxs/meJqL6e/1IEVyrX7GmNg34f3HfrYbaLZIsazNdJ+yRhSEA98OSepBBcyvPltFv7OOxO
Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: #012InvalidRequestThe provided encryption parameters did not match the ones used originally.9087CD73CE679AA4RXSVMQyNIUmGQ2zbz/Ko0xzfVeggiohMgenY82tN1tLY+cJ/FvNulW/Cq+bVRNvH8o2SRdV4vWlQ
Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: #012InvalidRequestThe provided encryption parameters did not match the ones used originally.227B94B427F38170i9DXrceGjaBNMIB6qQr75dRz2Uju6RFW/kVAQ199AJEw3ZXJU1jK16EIzWNSXzUl3w7nEbri+QEL

Details about issue

I cannot copy large-ish (>10MB, or the multipart size limit I think) to an s3 bucket mounted with a custom key. Small files work fine. If I use "use_sse=1" instead, all is well. It looks like SSE-C mounts aren't sending encryption information for multipart uploads.

Originally created by @icassassin on GitHub (Jan 3, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2085 <!-- -------------------------------------------------------------------------- The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all. Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD. --------------------------------------------------------------------------- --> ### Additional Information #### Version of s3fs being used (`s3fs --version`) Amazon Simple Storage Service File System V1.91 (commit:14eb1a7) with OpenSSL Also tried with v1.90 that comes with Ubuntu 22.04; same issue #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) example: 2.9.9 #### Kernel information (`uname -r`) 5.15.0-56-generic #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) Ubuntu 22.04.1 LTS #### How to run s3fs, if applicable from /etc/fstab: [bucket name] [mount point] fuse.s3fs _netdev,allow_other,url=https://s3.wasabisys.com,passwd_file=<path>,use_sse=custom:<path2>,use_path_request_style,dbglevel=info 0 0 #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`) I think the relevant syslog lines are: Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl_multi.cpp:MultiRead(228): failed a request(400: https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH) Jan 3 20:41:06 rdserv-staging s3fs[10246]: [tpath=/Godot_v3.5.1-stable_x11.64.zip][start=20971520][size=18752962][part=3] Jan 3 20:41:06 rdserv-staging s3fs[10246]: URL is https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH Jan 3 20:41:06 rdserv-staging s3fs[10246]: URL changed is https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=1&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [89915b33b6ee9af99dacd4950c1c8fa543ffad962d887d5e97c7c58d4fa5eef1] Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=2&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [63e1cc395ec600502bdb8809060272bb58fa0f745ee30512fd42be4059d463eb] Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [aaad05e698850c86508eba9b86fbe6b6f210d020794039aa909f9090d3c575a3] Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>InvalidRequest</Code><Message>The provided encryption parameters did not match the ones used originally.</Message><RequestId>61385D6BF18725E2</RequestId><HostId>akP7Ctxs/meJqL6e/1IEVyrX7GmNg34f3HfrYbaLZIsazNdJ+yRhSEA98OSepBBcyvPltFv7OOxO</HostId></Error> Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>InvalidRequest</Code><Message>The provided encryption parameters did not match the ones used originally.</Message><RequestId>9087CD73CE679AA4</RequestId><HostId>RXSVMQyNIUmGQ2zbz/Ko0xzfVeggiohMgenY82tN1tLY+cJ/FvNulW/Cq+bVRNvH8o2SRdV4vWlQ</HostId></Error> Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>InvalidRequest</Code><Message>The provided encryption parameters did not match the ones used originally.</Message><RequestId>227B94B427F38170</RequestId><HostId>i9DXrceGjaBNMIB6qQr75dRz2Uju6RFW/kVAQ199AJEw3ZXJU1jK16EIzWNSXzUl3w7nEbri+QEL</HostId></Error> ### Details about issue I cannot copy large-ish (>10MB, or the multipart size limit I think) to an s3 bucket mounted with a custom key. Small files work fine. If I use "use_sse=1" instead, all is well. It looks like SSE-C mounts aren't sending encryption information for multipart uploads.
kerem 2026-03-04 01:51:01 +03:00
  • closed this issue
  • added the
    bug
    label
Author
Owner

@icassassin commented on GitHub (Jan 3, 2023):

I think this is related to https://github.com/s3fs-fuse/s3fs-fuse/issues/1816

<!-- gh-comment-id:1370214869 --> @icassassin commented on GitHub (Jan 3, 2023): I think this is related to https://github.com/s3fs-fuse/s3fs-fuse/issues/1816
Author
Owner

@ggtakec commented on GitHub (Jan 6, 2023):

@icassassin
I was able to reproduce the bug in this issue about SSE-C.
Since #1816 is also SSE-C, it seems to be the same cause.

I'm currently working on a fix, so please be patient.

<!-- gh-comment-id:1373384165 --> @ggtakec commented on GitHub (Jan 6, 2023): @icassassin I was able to reproduce the bug in this issue about SSE-C. _Since #1816 is also SSE-C, it seems to be the same cause._ I'm currently working on a fix, so please be patient.
Author
Owner

@ggtakec commented on GitHub (Jan 6, 2023):

I created PR #2088 for this issue.
Please try this code if you can, or wait a moment for it to be merged.

<!-- gh-comment-id:1373462416 --> @ggtakec commented on GitHub (Jan 6, 2023): I created PR #2088 for this issue. Please try this code if you can, or wait a moment for it to be merged.
Author
Owner

@icassassin commented on GitHub (Jan 7, 2023):

Just did a brief test with this and it looks to be working as expected now. Successfully wrote the same file I previously tried and verified it is encrypted. Thanks!

<!-- gh-comment-id:1374358308 --> @icassassin commented on GitHub (Jan 7, 2023): > Just did a brief test with this and it looks to be working as expected now. Successfully wrote the same file I previously tried and verified it is encrypted. Thanks!
Author
Owner

@ggtakec commented on GitHub (Jan 7, 2023):

@icassassin Thanks for checking.
The code has been merged, so the bug will be fixed in the next version.

<!-- gh-comment-id:1374369442 --> @ggtakec commented on GitHub (Jan 7, 2023): @icassassin Thanks for checking. The code has been merged, so the bug will be fixed in the next version.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1057
No description provided.