starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #39] Support for SSE-C #22

New issue

Closed

opened 2026-03-04 01:41:16 +03:00 by kerem · 8 comments

kerem commented

2026-03-04 01:41:16 +03:00

Owner

Originally created by @geoffjukes on GitHub (Jun 13, 2014).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/39

Amazon announced support for customer-managed encryption keys (referred to as 'SSE-C'). The upload and download requests must include the encryption keys, and Amazon encrypts/decrypts at their end.

This is an incredibly useful feature, if S3FS could support it, I'm sure a lot of people would find it useful.

More information on SSE-C is here:

http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html

Originally created by @geoffjukes on GitHub (Jun 13, 2014). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/39 Amazon announced support for customer-managed encryption keys (referred to as 'SSE-C'). The upload and download requests must include the encryption keys, and Amazon encrypts/decrypts at their end. This is an incredibly useful feature, if S3FS could support it, I'm sure a lot of people would find it useful. More information on SSE-C is here: http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html

kerem closed this issue

2026-03-04 01:41:16 +03:00

kerem commented

2026-03-04 01:41:17 +03:00

Author

Owner

@ggtakec commented on GitHub (Jun 20, 2014):

I think s3fs can support SSE-C, and try to implement into s3fs.
Please wait for make it.

Regards,

@ggtakec commented on GitHub (Jun 20, 2014): I think s3fs can support SSE-C, and try to implement into s3fs. Please wait for make it. Regards,

kerem commented

2026-03-04 01:41:17 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 19, 2014):

I push new branch "see-c" for support SSE-C.
You can see help("s3fs -h") or man page for "use-sse" option.
The use_sse option is changed and allowed parameter for see-c secret keys file.

The use_sse option is following:

"use_sse"
enable Server-Site Encryption(SSE).
"use_sse=1"
enable Server-Site Encryption(SSE). this is old format.
"use_sse=filepath"
enable Server-Site Encryption with Customer-Provided Encryption Keys(SSE-C).
The file can have some lines, each line is one of SSE-C secret key.
The first line secret key is used by uploading/copying objects which are made new object.
After first line secret keys are used by decrypt old objects.
The file permission must be 600.
Please take care for running s3fs, you must run s3fs with HTTPS for SSE-C.

The secret keys file has many SSE-C keys, it means you can change/add new key for SSE-C.
It can read/list objects which is encrypted by old key.

I made secret keys and tested by following:
$ pwgen -Bs 32 1 > ~/sseckeys
$ pwgen -Bs 32 1 >> /sseckeys
$ cat /tmp/aaa
U5VKgo4BkmE3grmQ1KtatGGoqfDwVcHV
Z7ImOlQ8aPulwft2iw0SJ7v8fCHCmNug
$ s3fs /mnt/s3 -o allow_other,use_cache=/tmp,use_sse=/sseckeys,url=https://s3.amazonaws.com -f &

If you can, please checkout see-c branch and test it.
Thanks in advance for your help.

@ggtakec commented on GitHub (Jul 19, 2014): I push new branch "see-c" for support SSE-C. You can see help("s3fs -h") or man page for "use-sse" option. The use_sse option is changed and allowed parameter for see-c secret keys file. The use_sse option is following: 1) "use_sse" enable Server-Site Encryption(SSE). 2) "use_sse=1" enable Server-Site Encryption(SSE). this is old format. 3) "use_sse=filepath" enable Server-Site Encryption with Customer-Provided Encryption Keys(SSE-C). The file can have some lines, each line is one of SSE-C secret key. The first line secret key is used by uploading/copying objects which are made new object. After first line secret keys are used by decrypt old objects. The file permission must be 600. Please take care for running s3fs, you must run s3fs with HTTPS for SSE-C. The secret keys file has many SSE-C keys, it means you can change/add new key for SSE-C. It can read/list objects which is encrypted by old key. I made secret keys and tested by following: $ pwgen -Bs 32 1 > ~/sseckeys $ pwgen -Bs 32 1 >> ~/sseckeys $ cat /tmp/aaa U5VKgo4BkmE3grmQ1KtatGGoqfDwVcHV Z7ImOlQ8aPulwft2iw0SJ7v8fCHCmNug $ s3fs <bucket> /mnt/s3 -o allow_other,use_cache=/tmp,use_sse=~/sseckeys,url=https://s3.amazonaws.com -f & If you can, please checkout see-c branch and test it. Thanks in advance for your help.

kerem commented

2026-03-04 01:41:17 +03:00

Author

Owner

@malloctrim commented on GitHub (Aug 18, 2014):

Hi ggtakec,

I've tried you sse-c patch, but if fails when trying to upload larges files (multipart), Amazon returns a '400 bad request' error.

@malloctrim commented on GitHub (Aug 18, 2014): Hi ggtakec, I've tried you sse-c patch, but if fails when trying to upload larges files (multipart), Amazon returns a '400 bad request' error.

kerem commented

2026-03-04 01:41:17 +03:00

Author

Owner

@malloctrim commented on GitHub (Aug 18, 2014):

Ok, got it working better on my side, couple of things I noted:

At two places you call this: AddSseKeyRequestHeader(value, ow_sse_flag), but this is incorrect, because ow_sse_flag is true and then you send the 'copy-source' headers instead of the normal one. Changing this to AddSseKeyRequestHeader(value, false) makes the upload itself get a proper 200 response from Amazon (this caused the 400 bad request error above).
After the upload is finished you check that the etag response from the multipart upload is the same as the md5 you made before, but with sse-c the etag of the response is not the md5 (http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html), so you have to take the etag from the return headers instead.

@malloctrim commented on GitHub (Aug 18, 2014): Ok, got it working better on my side, couple of things I noted: - At two places you call this: `AddSseKeyRequestHeader(value, ow_sse_flag)`, but this is incorrect, because ow_sse_flag is true and then you send the 'copy-source' headers instead of the normal one. Changing this to `AddSseKeyRequestHeader(value, false)` makes the upload itself get a proper 200 response from Amazon (this caused the 400 bad request error above). - After the upload is finished you check that the etag response from the multipart upload is the same as the md5 you made before, but with sse-c the etag of the response is not the md5 (http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html), so you have to take the etag from the return headers instead.

kerem commented

2026-03-04 01:41:17 +03:00

Author

Owner

@malloctrim commented on GitHub (Aug 18, 2014):

By the way, a suggestion. You read the key(s) now from a file, but it would be useful if you could also read the key from an environment variable (if defined)... then the key doesn't have to reside somewhere on the disk.

@malloctrim commented on GitHub (Aug 18, 2014): By the way, a suggestion. You read the key(s) now from a file, but it would be useful if you could also read the key from an environment variable (if defined)... then the key doesn't have to reside somewhere on the disk.

kerem commented

2026-03-04 01:41:17 +03:00

Author

Owner

@malloctrim commented on GitHub (Aug 18, 2014):

I have tested using rsync, and the chmod is failing when you call AddSseKeyRequestHeader with false all the time, as it also needs the copy-source headers, so I had to track separately when to send the 'copy-source' headers vs the normal ones. But you probably have a better way of solving that. Anyway, rsync is now properly working for me with this sse-c branch, which is what I needed.

@malloctrim commented on GitHub (Aug 18, 2014): I have tested using rsync, and the chmod is failing when you call `AddSseKeyRequestHeader` with false all the time, as it also needs the copy-source headers, so I had to track separately when to send the 'copy-source' headers vs the normal ones. But you probably have a better way of solving that. Anyway, rsync is now properly working for me with this sse-c branch, which is what I needed.

kerem commented

2026-03-04 01:41:17 +03:00

Author

Owner

@ggtakec commented on GitHub (Aug 26, 2014):

I committed new see-c branch.
It fixed about see, before revision is wrong because I did not understand backer policy with see.
Old revision used original see mode(key) as possible as s3fs can.
But it was wrong logic, it should be used see mode(key) if it is specifyed.
So when any operation s3fs(see-c branch) change(make)s the object with sse(key) if sse(key) is specified.

And I added AWSSSECKEYS environment.

Please check it and let me know.
Thanks,

@ggtakec commented on GitHub (Aug 26, 2014): I committed new see-c branch. It fixed about see, before revision is wrong because I did not understand backer policy with see. Old revision used original see mode(key) as possible as s3fs can. But it was wrong logic, it should be used see mode(key) if it is specifyed. So when any operation s3fs(see-c branch) change(make)s the object with sse(key) if sse(key) is specified. And I added AWSSSECKEYS environment. Please check it and let me know. Thanks,

kerem commented

2026-03-04 01:41:18 +03:00

Author

Owner

@ggtakec commented on GitHub (Sep 7, 2014):

Hi, malloctrim

I merged see-c branch to master, and I closed this issue.
If you find a issue about it, please let me know.(please post new issue)

Thanks in advance for your help.

@ggtakec commented on GitHub (Sep 7, 2014): Hi, malloctrim I merged see-c branch to master, and I closed this issue. If you find a issue about it, please let me know.(please post new issue) Thanks in advance for your help.

kerem referenced this issue

2026-03-04 01:53:20 +03:00

[PR #22] [MERGED] Fix typos in docs for max_stat_cache_size (the default is actually 1000) #1313