[GH-ISSUE #203] Change xattr from single JSON blob into per-attribute user metadata

kerem commented

2026-03-04 01:42:14 +03:00

Owner

Originally created by @gaul on GitHub (Jun 25, 2015).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/203

Presently s3fs stores all extended attributes in a single base64-encoded JSON blob. This prevents other tools like s3cmd and the AWS console from introspecting and modifying these. Instead s3fs should store each extended attrbutes in its own user metadata.

Originally created by @gaul on GitHub (Jun 25, 2015). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/203 Presently s3fs stores all extended attributes in a single base64-encoded JSON blob. This prevents other tools like s3cmd and the AWS console from introspecting and modifying these. Instead s3fs should store each extended attrbutes in its own user metadata.

kerem closed this issue

2026-03-04 01:42:14 +03:00

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 11, 2015):

As I've also commented on #206, I think it does not need to change the header name for xattr.

If s3fs user need to change xattr by other tools(s3cmd, was console, etc), I think he should be able to do it.
Now x-amz-meta-xattr value is formatted by json and base64, and it has some xattr key and value.

Considering from the point of view of each of the attribute, this is might be hard to use.
However, if it is thought as a whole extended attribute, it is easy to use.
And one of case is that entryptfs and encfs use some xattr that are accessed by only the system, another case there are xattrs that a user sets for their own.

But it is simply, it is possible to access each xattr through FUSE The filesystem.
Then we can choice which xattr is set one header or each key’s name header.

At the moment, although both cases will be possible, I think it does not need to save in the header to the corresponding xattr to each.

Regards,

@ggtakec commented on GitHub (Jul 11, 2015): As I've also commented on #206, I think it does not need to change the header name for xattr. If s3fs user need to change xattr by other tools(s3cmd, was console, etc), I think he should be able to do it. Now x-amz-meta-xattr value is formatted by json and base64, and it has some xattr key and value. Considering from the point of view of each of the attribute, this is might be hard to use. However, if it is thought as a whole extended attribute, it is easy to use. And one of case is that entryptfs and encfs use some xattr that are accessed by only the system, another case there are xattrs that a user sets for their own. But it is simply, it is possible to access each xattr through FUSE The filesystem. Then we can choice which xattr is set one header or each key’s name header. At the moment, although both cases will be possible, I think it does not need to save in the header to the corresponding xattr to each. Regards,

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@gaul commented on GitHub (Jul 12, 2015):

Storing extended attributes in per-key user metadata has other advantages -- s3fs could expose other metadata like storage class and content type as xattr which would allow standard tools to introspect and manipulate them. Imagine being able to toggle Glacier storage class with a simple command. I do not believe that most users can or will craft the base64-JSON format to do this.

@gaul commented on GitHub (Jul 12, 2015): Storing extended attributes in per-key user metadata has other advantages -- s3fs could expose other metadata like storage class and content type as xattr which would allow standard tools to introspect and manipulate them. Imagine being able to toggle Glacier storage class with a simple command. I do not believe that most users can or will craft the base64-JSON format to do this.

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 12, 2015):

Yes, I understand advantages.

But I worry about performance and complex logic and benefits commensurate with the cost.
For example, clients(users) can set any extended attribute name and value.

Then s3fs need to escape(or base64 encoding) those and set "x-amz-meta-xattr...".
We must always encode extended attribute value(and just maybe name), because the value can be set to binary data.
(Maybe the name can be set characters that are not allowed as an HTTP header too. However, I have not been confirmed completely.)
Thus, if s3fs makes each header for each extended attribute, HTTP header will be "x-amz-meta-xattr-".
And s3fs is even made of each of the header, we have to encode base64(or uri encode) for the value.

So I think also be divided into each of the header, but I did not think that there is an advantage commensurate with these costs. Then I think s3fs should make one of the header for xattr.
And I think that there is a few users who change the xattr directly from s3cmd and aws console, because xattr is for filesystem.

I wait to build v1.79 release until this issue is resolved.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jul 12, 2015): Yes, I understand advantages. But I worry about performance and complex logic and benefits commensurate with the cost. For example, clients(users) can set any extended attribute name and value. Then s3fs need to escape(or base64 encoding) those and set "x-amz-meta-xattr...". We must always encode extended attribute value(and just maybe name), because the value can be set to binary data. (Maybe the name can be set characters that are not allowed as an HTTP header too. However, I have not been confirmed completely.) Thus, if s3fs makes each header for each extended attribute, HTTP header will be "x-amz-meta-xattr-<encoded xattr name>". And s3fs is even made of each of the header, we have to encode base64(or uri encode) for the value. So I think also be divided into each of the header, but I did not think that there is an advantage commensurate with these costs. Then I think s3fs should make one of the header for xattr. And I think that there is a few users who change the xattr directly from s3cmd and aws console, because xattr is for filesystem. I wait to build v1.79 release until this issue is resolved. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@gaul commented on GitHub (Jul 12, 2015):

If we HTTP percent-encode both xattr keys and values this would be user-friendly for common uses of ASCII data while allowing binary data for other uses like encfs. Specifically I would like to interact with storage class and content type from the command line:

setfattr -n x-amz-storage-class -v REDUCED_REDUNDANCY $TEST_TEXT_FILE
setfattr -n Content-Type -v image/jpeg $TEST_TEXT_FILE

Note that this requires a little more mapping since we need to set x-amz-meta- for unrecognized keys but map to x-amz-storage-class and Content-Type for others.

@gaul commented on GitHub (Jul 12, 2015): If we HTTP percent-encode both xattr keys and values this would be user-friendly for common uses of ASCII data while allowing binary data for other uses like encfs. Specifically I would like to interact with storage class and content type from the command line: ``` setfattr -n x-amz-storage-class -v REDUCED_REDUNDANCY $TEST_TEXT_FILE setfattr -n Content-Type -v image/jpeg $TEST_TEXT_FILE ``` Note that this requires a little more mapping since we need to set `x-amz-meta-` for unrecognized keys but map to `x-amz-storage-class` and `Content-Type` for others.

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 13, 2015):

About percent encoding, it might be good.
But security.capability(and other security namespace xattrs) attribute is binary array, base64 efficiency is better than percent encoding in the case of binary data.

And importantly, extended attribute belongs to the namespace which are user, security, etc…
It means if you set xattr to a file by setfattr, xattr name should be “user.xxxxx”.
So your example xattr is probably set “user.x-amz-storage-class” name.
If I am not mistaken, s3fs does not work good when the name does not have namespace.

If the purpose for making http-header by each xattr is equivalently treats xattr name and the HTTP header name, this namespace would get in the way.
And if s3fs(and setfattr etc) do not handle them equivalently, the benefits of using percent encoding will be smaller.

For these reasons, I think that outputing to one of the HTTP header formatted base64 is better.

Thanks in advance.

@ggtakec commented on GitHub (Jul 13, 2015): About percent encoding, it might be good. But security.capability(and other security namespace xattrs) attribute is binary array, base64 efficiency is better than percent encoding in the case of binary data. And importantly, extended attribute belongs to the namespace which are user, security, etc… It means if you set xattr to a file by setfattr, xattr name should be “user.xxxxx”. So your example xattr is probably set “user.x-amz-storage-class” name. If I am not mistaken, s3fs does not work good when the name does not have namespace. If the purpose for making http-header by each xattr is equivalently treats xattr name and the HTTP header name, this namespace would get in the way. And if s3fs(and setfattr etc) do not handle them equivalently, the benefits of using percent encoding will be smaller. For these reasons, I think that outputing to one of the HTTP header formatted base64 is better. Thanks in advance.

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 18, 2015):

@andrewgaul
I would like to relase new version without correspondence this Issue, because I think that there is no benefit in support of this Issue as I have said just before.

Please let me know if you have other opinion about benefit and cost for changing xattr format.
Thanks in advance for your help.

@ggtakec commented on GitHub (Jul 18, 2015): @andrewgaul I would like to relase new version without correspondence this Issue, because I think that there is no benefit in support of this Issue as I have said just before. Please let me know if you have other opinion about benefit and cost for changing xattr format. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@gaul commented on GitHub (Jul 18, 2015):

@ggtakec Let's move forward with a release without changing the format. I would like to keep this issue open and I will work on it later.

@gaul commented on GitHub (Jul 18, 2015): @ggtakec Let's move forward with a release without changing the format. I would like to keep this issue open and I will work on it later.

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 18, 2015):

Thank you for replying quickly.

Ok, I will release new version a few days.
Regards,

@ggtakec commented on GitHub (Jul 18, 2015): Thank you for replying quickly. Ok, I will release new version a few days. Regards,

kerem commented

2026-03-04 01:42:15 +03:00

Author

Owner

@ggtakec commented on GitHub (Sep 30, 2015):

I'm closing this issue.
If you have more problem, please reopen this issue or post new issue.
Regards,

@ggtakec commented on GitHub (Sep 30, 2015): I'm closing this issue. If you have more problem, please reopen this issue or post new issue. Regards,

kerem referenced this issue

2026-03-04 01:53:30 +03:00

[PR #114] [CLOSED] Disable pages merge #1349

kerem referenced this issue

2026-03-04 01:53:32 +03:00

[PR #122] [CLOSED] Disable pages merge #1353

Rows
Columns

[GH-ISSUE #203] Change xattr from single JSON blob into per-attribute user metadata #114