mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 13:26:00 +03:00
[GH-ISSUE #2085] File copies using multipart upload fail with SSE-C #1057
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#1057
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @icassassin on GitHub (Jan 3, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2085
Additional Information
Version of s3fs being used (
s3fs --version)Amazon Simple Storage Service File System V1.91 (commit:14eb1a7) with OpenSSL
Also tried with v1.90 that comes with Ubuntu 22.04; same issue
Version of fuse being used (
pkg-config --modversion fuse,rpm -qi fuseordpkg -s fuse)example: 2.9.9
Kernel information (
uname -r)5.15.0-56-generic
GNU/Linux Distribution, if applicable (
cat /etc/os-release)Ubuntu 22.04.1 LTS
How to run s3fs, if applicable
from /etc/fstab:
[bucket name] [mount point] fuse.s3fs _netdev,allow_other,url=https://s3.wasabisys.com,passwd_file=,use_sse=custom:,use_path_request_style,dbglevel=info 0 0
s3fs syslog messages (
grep s3fs /var/log/syslog,journalctl | grep s3fs, ors3fs outputs)I think the relevant syslog lines are:
Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl_multi.cpp:MultiRead(228): failed a request(400: https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH)
Jan 3 20:41:06 rdserv-staging s3fs[10246]: [tpath=/Godot_v3.5.1-stable_x11.64.zip][start=20971520][size=18752962][part=3]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: URL is https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH
Jan 3 20:41:06 rdserv-staging s3fs[10246]: URL changed is https://s3.wasabisys.com/[bucket name]/Godot_v3.5.1-stable_x11.64.zip?partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH
Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=1&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [89915b33b6ee9af99dacd4950c1c8fa543ffad962d887d5e97c7c58d4fa5eef1]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com
Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=2&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [63e1cc395ec600502bdb8809060272bb58fa0f745ee30512fd42be4059d463eb]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com
Jan 3 20:41:06 rdserv-staging s3fs[10246]: computing signature [PUT] [/Godot_v3.5.1-stable_x11.64.zip] [partNumber=3&uploadId=kVWQFryLsXmfu__SQmTnhVLwq2gL6s7CYhMAXzlVQZ9vSXF4ceiFE9FIRxleQExNdgwIvIo_x7HkjCPJAt6lj9XwAfTxynL5ED2otPBmmyTbY40_HjZXwS9Kkq-lT6AH] [aaad05e698850c86508eba9b86fbe6b6f210d020794039aa909f9090d3c575a3]
Jan 3 20:41:06 rdserv-staging s3fs[10246]: url is https://s3.wasabisys.com
Jan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: #012
InvalidRequestThe provided encryption parameters did not match the ones used originally.61385D6BF18725E2akP7Ctxs/meJqL6e/1IEVyrX7GmNg34f3HfrYbaLZIsazNdJ+yRhSEA98OSepBBcyvPltFv7OOxOJan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: #012
InvalidRequestThe provided encryption parameters did not match the ones used originally.9087CD73CE679AA4RXSVMQyNIUmGQ2zbz/Ko0xzfVeggiohMgenY82tN1tLY+cJ/FvNulW/Cq+bVRNvH8o2SRdV4vWlQJan 3 20:41:06 rdserv-staging s3fs[10246]: curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: #012
InvalidRequestThe provided encryption parameters did not match the ones used originally.227B94B427F38170i9DXrceGjaBNMIB6qQr75dRz2Uju6RFW/kVAQ199AJEw3ZXJU1jK16EIzWNSXzUl3w7nEbri+QELDetails about issue
I cannot copy large-ish (>10MB, or the multipart size limit I think) to an s3 bucket mounted with a custom key. Small files work fine. If I use "use_sse=1" instead, all is well. It looks like SSE-C mounts aren't sending encryption information for multipart uploads.
@icassassin commented on GitHub (Jan 3, 2023):
I think this is related to https://github.com/s3fs-fuse/s3fs-fuse/issues/1816
@ggtakec commented on GitHub (Jan 6, 2023):
@icassassin
I was able to reproduce the bug in this issue about SSE-C.
Since #1816 is also SSE-C, it seems to be the same cause.
I'm currently working on a fix, so please be patient.
@ggtakec commented on GitHub (Jan 6, 2023):
I created PR #2088 for this issue.
Please try this code if you can, or wait a moment for it to be merged.
@icassassin commented on GitHub (Jan 7, 2023):
Just did a brief test with this and it looks to be working as expected now. Successfully wrote the same file I previously tried and verified it is encrypted. Thanks!
@ggtakec commented on GitHub (Jan 7, 2023):
@icassassin Thanks for checking.
The code has been merged, so the bug will be fixed in the next version.