[GH-ISSUE #2585] Consecutive xattr write errors on FUSE-T macos #1243

Open
opened 2026-03-04 01:52:31 +03:00 by kerem · 1 comment
Owner

Originally created by @ggtakec on GitHub (Nov 2, 2024).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2585

Relevant Issue (if applicable)

#2418 #2205 #2198 #2531 #2533

Overview

This issue has come up many times in the past, and I was able to find the situation in which test_extended_attributes was failing.

Detail

I confirmed an unknown behavior of FUSE-T in the following part of the test_extended_attributes test code(excerpt).

set_xattr key1 value0 <file>        (1)
get_xattr key1 <file>               (2)
set_xattr key1 value1 <file>        (3) THIS COMMAND
get_xattr key1 <file>               (4)

In part (3) above, there is a problem with s3fs_setxattr(setxattr hook) called by FUSE when writing to xattr(name=key1) for the second time.
Below are excerpts from the logs before and after the call when the test failed.

(1) s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x7f934e80003d][size=0][flags=0x0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d80d600][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":""}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":""}
    s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x7f934e80003d][size=6][flags=0x0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934e00e000][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getattr(981): [path=/testrun-28677/test-s3fs-12609.txt] uid=501, gid=20, mode=100644
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d013c00][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20]

(2) s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"}
    s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20]

(3) s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x2][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=][value=0x7f9350801039][size=6][flags=0x0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x7f934e80003d][size=0][flags=0x0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=3][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=3][pid=0,uid=501,gid=20]

(4) s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934c045a00][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"}
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934e00e000][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384]
    s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"}
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20]
    s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"}
    s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20]
    s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20]

There seem to be two problems.

The first is the following part in (3) log:

s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=][value=0x7f9350801039][size=6][flags=0x0][pid=0,uid=501,gid=20]

The part where name should be key1 is empty.
As a result, the value that should be set to key1 is registered as the value of the "" key.

The other is that when (3) is written, s3fs_open(open hook) is called continuously.(Actually, it may be a different thread, but this has not been confirmed.)
One of the two opens closes(flush->releases) after writing xattr, and the other closes after reading xattr.
This looks like the file is being opened twice simultaneously to set/get xattr (the commands are being called sequentially so it makes me wonder why they are being processed simultaneously)

Either way, the test fails due to the empty key call to xattr write.

I will share this situation with the fuse-t as issue.
As for s3fs, I will bypass this unstable test in the case of macos.(I will post a PR later)

Originally created by @ggtakec on GitHub (Nov 2, 2024). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2585 ### Relevant Issue (if applicable) #2418 #2205 #2198 #2531 #2533 ### Overview This issue has come up many times in the past, and I was able to find the situation in which `test_extended_attributes` was failing. ### Detail I confirmed an unknown behavior of FUSE-T in the following part of the `test_extended_attributes` test code(excerpt). ``` set_xattr key1 value0 <file> (1) get_xattr key1 <file> (2) set_xattr key1 value1 <file> (3) THIS COMMAND get_xattr key1 <file> (4) ``` In part (3) above, there is a problem with `s3fs_setxattr`(`setxattr` hook) called by FUSE when writing to `xattr`(`name=key1`) for the second time. Below are excerpts from the logs before and after the call when the test failed. ``` (1) s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x7f934e80003d][size=0][flags=0x0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d80d600][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":""} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":""} s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x7f934e80003d][size=6][flags=0x0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934e00e000][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getattr(981): [path=/testrun-28677/test-s3fs-12609.txt] uid=501, gid=20, mode=100644 s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d013c00][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20] (2) s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"key1":"dmFsdWUw"} s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20] (3) s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_open(2765): [path=/testrun-28677/test-s3fs-12609.txt][flags=0x2][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=][value=0x7f9350801039][size=6][flags=0x0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x7f934e80003d][size=0][flags=0x0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=3][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=3][pid=0,uid=501,gid=20] (4) s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934c045a00][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"} s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934e00e000][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_listxattr(3992): [path=/testrun-28677/test-s3fs-12609.txt][list=0x7f934d813800][size=16384] s3fs: [DBG] s3fs.cpp:s3fs_listxattr(4022): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"} s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3923): [path=/testrun-28677/test-s3fs-12609.txt][name=key1][value=0x0][size=0][pid=0,uid=501,gid=20] s3fs: [DBG] s3fs.cpp:s3fs_getxattr(3963): Get xattrs = {"":"dmFsdWUx","key1":"dmFsdWUw"} s3fs: [INF] s3fs.cpp:s3fs_flush(2949): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20] s3fs: [INF] s3fs.cpp:s3fs_release(3037): [path=/testrun-28677/test-s3fs-12609.txt][pseudo_fd=2][pid=0,uid=501,gid=20] ``` There seem to be two problems. The first is the following part in (3) log: ``` s3fs_setxattr(3775): [path=/testrun-28677/test-s3fs-12609.txt][name=][value=0x7f9350801039][size=6][flags=0x0][pid=0,uid=501,gid=20] ``` The part where `name` should be `key1` is `empty`. As a result, the value that should be set to `key1` is registered as the value of the `""` key. The other is that when (3) is written, `s3fs_open`(`open` hook) is called continuously.(Actually, it may be a different thread, but this has not been confirmed.) One of the two opens closes(`flush`->`releases`) after writing `xattr`, and the other closes after reading `xattr`. This looks like the file is being opened twice simultaneously to `set/get xattr` (the commands are being called sequentially so it makes me wonder why they are being processed simultaneously) Either way, the test fails due to the `empty` key call to `xattr` write. - How to reproduce You can easily inspect this example at https://github.com/ggtakec/s3fs-fuse/tree/test/bug_macos_test. (This is only test on macos, and only run `test_extended_attributes` case) I will share this situation with the fuse-t as issue. As for s3fs, I will bypass this unstable test in the case of macos.(I will post a PR later)
Author
Owner

@ggtakec commented on GitHub (Jun 28, 2025):

I've made this issue a reference for the problem with bypassing tests in #2687.
In #2687, only the delete part of xattr is bypassed, but there may still be problems with set.

And I was a bit worried; I don't think AppleDouble files were created in macos-fuse (osx-fuse), but it seems that macos-fuse-t creates them.
In currently Github Actions(CI), it seems that AppleDouble files are created.
I'm leaving this here to share information as it may be necessary to investigate this further.

<!-- gh-comment-id:3015157699 --> @ggtakec commented on GitHub (Jun 28, 2025): I've made this issue a reference for the problem with bypassing tests in #2687. In #2687, only the delete part of xattr is bypassed, but there may still be problems with set. And I was a bit worried; I don't think AppleDouble files were created in macos-fuse (osx-fuse), but it seems that macos-fuse-t creates them. In currently Github Actions(CI), it seems that AppleDouble files are created. I'm leaving this here to share information as it may be necessary to investigate this further.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1243
No description provided.