[GH-ISSUE #78] s3fs crashes on ls #44

Closed
opened 2026-03-04 01:41:29 +03:00 by kerem · 7 comments
Owner

Originally created by @sanemat on GitHub (Oct 30, 2014).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/78

I can mount the bucket which I want, but I met a crash on ls. After the crash, I see Device not configured, and s3fs unmount this automatically(?).

$ cd /mnt/s3fs-test/
$ ls
(It's ok.)
$ cd logs
(logs has many log files)
$ ls
ls: fts_read: Device not configured
$ ls
ls: .: No such file or directory

This is the log with -f -d option, actual command: s3fs BUCKET_NAME /mnt/s3fs-test/ -o uid=UID -o gid=GID -o use_cache=/tmp -d -f

(snip)
    Request(3305): [count=20]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141016-03_0.gz]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141020-11_0.gz]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141020-10_0.gz]
MultiRead(3264): failed to read(remaining: 16 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 15 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 14 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 13 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 12 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 11 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 10 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 9 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 8 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 7 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 6 code: 7  msg: Couldn't connect to server), so retry this.
    AddStat(247): add stat cache entry[path=/logs/(snip)20141019-14_0.gz]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141019-16_0.gz]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141019-13_0.gz]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141019-07_0.gz]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141019-17_0.gz]
    AddStat(247): add stat cache entry[path=/logs/(snip)20141020-07_0.gz]
MultiRead(3264): failed to read(remaining: 10 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 9 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 8 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 7 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 6 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 5 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 4 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 3 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 2 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 1 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 0 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 10 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 9 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 8 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 7 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 6 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 5 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 4 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 3 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 2 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 1 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 0 code: 7  msg: Couldn't connect to server), so retry this.
MultiRead(3264): failed to read(remaining: 10 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-04_0.gz).
MultiRead(3264): failed to read(remaining: 9 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-02_0.gz).
MultiRead(3264): failed to read(remaining: 8 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-01_0.gz).
MultiRead(3264): failed to read(remaining: 7 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141017-06_0.gz).
MultiRead(3264): failed to read(remaining: 6 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141017-02_0.gz).
MultiRead(3264): failed to read(remaining: 5 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141016-07_0.gz).
MultiRead(3264): failed to read(remaining: 4 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141016-05_0.gz).
MultiRead(3264): failed to read(remaining: 3 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-12_0.gz).
MultiRead(3264): failed to read(remaining: 2 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-19_0.gz).
MultiRead(3264): failed to read(remaining: 1 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-21_0.gz).
MultiRead(3264): failed to read(remaining: 0 code: 7  msg: Couldn't connect to server), so retry this.
multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141016-04_0.gz).
    GetStat(170): stat cache hit [path=/logs/(snip)20141016-03_0.gz][time=1414677745][hit count=0]
    readdir_multi_head(2208): Could not find /logs/(snip)20141016-04_0.gz file in stat cache.
    readdir_multi_head(2208): Could not find /logs/(snip)20141016-05_0.gz file in stat cache.
    readdir_multi_head(2208): Could not find /logs/(snip)20141016-07_0.gz file in stat cache.
    readdir_multi_head(2208): Could not find /logs/(snip)20141017-02_0.gz file in stat cache.
    readdir_multi_head(2208): Could not find /logs/(snip)20141017-06_0.gz file in stat cache.
    readdir_multi_head(2208): Could not find /logs/(snip)20141018-01_0.gz file in stat cache.
    (snip)
    GetStat(170): stat cache hit [path=/logs/(snip)20141030-07_0.gz][time=1414678058][hit count=0]
s3fs_destroy(2689): destroy

$ s3fs --version
Amazon Simple Storage Service File System V1.78 with OpenSSL

I use current(2014-10-30) s3fs-fuse master 77d4d066b5
Mac OS X 10.9.5

Related issue(?)
s3fs appears to crash on ls with permission failure

Originally created by @sanemat on GitHub (Oct 30, 2014). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/78 I can mount the bucket which I want, but I met a crash on ls. After the crash, I see `Device not configured`, and s3fs unmount this automatically(?). ``` $ cd /mnt/s3fs-test/ $ ls (It's ok.) $ cd logs (logs has many log files) $ ls ls: fts_read: Device not configured $ ls ls: .: No such file or directory ``` This is the log with `-f -d` option, actual command: `s3fs BUCKET_NAME /mnt/s3fs-test/ -o uid=UID -o gid=GID -o use_cache=/tmp -d -f` ``` (snip) Request(3305): [count=20] AddStat(247): add stat cache entry[path=/logs/(snip)20141016-03_0.gz] AddStat(247): add stat cache entry[path=/logs/(snip)20141020-11_0.gz] AddStat(247): add stat cache entry[path=/logs/(snip)20141020-10_0.gz] MultiRead(3264): failed to read(remaining: 16 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 15 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 14 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 13 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 12 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 11 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 10 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 9 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 8 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 7 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 6 code: 7 msg: Couldn't connect to server), so retry this. AddStat(247): add stat cache entry[path=/logs/(snip)20141019-14_0.gz] AddStat(247): add stat cache entry[path=/logs/(snip)20141019-16_0.gz] AddStat(247): add stat cache entry[path=/logs/(snip)20141019-13_0.gz] AddStat(247): add stat cache entry[path=/logs/(snip)20141019-07_0.gz] AddStat(247): add stat cache entry[path=/logs/(snip)20141019-17_0.gz] AddStat(247): add stat cache entry[path=/logs/(snip)20141020-07_0.gz] MultiRead(3264): failed to read(remaining: 10 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 9 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 8 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 7 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 6 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 5 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 4 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 3 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 2 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 1 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 0 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 10 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 9 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 8 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 7 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 6 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 5 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 4 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 3 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 2 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 1 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 0 code: 7 msg: Couldn't connect to server), so retry this. MultiRead(3264): failed to read(remaining: 10 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-04_0.gz). MultiRead(3264): failed to read(remaining: 9 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-02_0.gz). MultiRead(3264): failed to read(remaining: 8 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-01_0.gz). MultiRead(3264): failed to read(remaining: 7 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141017-06_0.gz). MultiRead(3264): failed to read(remaining: 6 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141017-02_0.gz). MultiRead(3264): failed to read(remaining: 5 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141016-07_0.gz). MultiRead(3264): failed to read(remaining: 4 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141016-05_0.gz). MultiRead(3264): failed to read(remaining: 3 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-12_0.gz). MultiRead(3264): failed to read(remaining: 2 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-19_0.gz). MultiRead(3264): failed to read(remaining: 1 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141018-21_0.gz). MultiRead(3264): failed to read(remaining: 0 code: 7 msg: Couldn't connect to server), so retry this. multi_head_retry_callback(2119): Over retry count(3) limit(/logs/(snip)20141016-04_0.gz). GetStat(170): stat cache hit [path=/logs/(snip)20141016-03_0.gz][time=1414677745][hit count=0] readdir_multi_head(2208): Could not find /logs/(snip)20141016-04_0.gz file in stat cache. readdir_multi_head(2208): Could not find /logs/(snip)20141016-05_0.gz file in stat cache. readdir_multi_head(2208): Could not find /logs/(snip)20141016-07_0.gz file in stat cache. readdir_multi_head(2208): Could not find /logs/(snip)20141017-02_0.gz file in stat cache. readdir_multi_head(2208): Could not find /logs/(snip)20141017-06_0.gz file in stat cache. readdir_multi_head(2208): Could not find /logs/(snip)20141018-01_0.gz file in stat cache. (snip) GetStat(170): stat cache hit [path=/logs/(snip)20141030-07_0.gz][time=1414678058][hit count=0] s3fs_destroy(2689): destroy ``` $ s3fs --version Amazon Simple Storage Service File System V1.78 with OpenSSL I use current(2014-10-30) s3fs-fuse master 77d4d066b5b1d28b1bcd3d5a5a3a946873355e43 Mac OS X 10.9.5 Related issue(?) [s3fs appears to crash on ls with permission failure](https://github.com/s3fs-fuse/s3fs-fuse/issues/25)
kerem closed this issue 2026-03-04 01:41:30 +03:00
Author
Owner

@ggtakec commented on GitHub (Nov 8, 2014):

It seems that s3fs(curl library) failed to connect s3 server.(code 7 means CURLE_COULDNT_CONNECT).
So that, s3fs exited without unmounting.

I will check codes for unmounting.

<!-- gh-comment-id:62261301 --> @ggtakec commented on GitHub (Nov 8, 2014): It seems that s3fs(curl library) failed to connect s3 server.(code 7 means CURLE_COULDNT_CONNECT). So that, s3fs exited without unmounting. I will check codes for unmounting.
Author
Owner

@ggtakec commented on GitHub (Nov 8, 2014):

If you can, please set multireq_max option as under 20( try 10 or small).

And about unmounting, I think that something wrong is occured in fuse because s3fs destroy hook is called. I will continue to check the reason about it.

<!-- gh-comment-id:62265691 --> @ggtakec commented on GitHub (Nov 8, 2014): If you can, please set multireq_max option as under 20( try 10 or small). And about unmounting, I think that something wrong is occured in fuse because s3fs destroy hook is called. I will continue to check the reason about it.
Author
Owner

@ggtakec commented on GitHub (Jan 17, 2016):

I'm sorry about this Issue had been left at a long period of time.

If you have this problem yet, please try to use latest code because some problem about multipart request etc were fixed.
("-f", "dbglevel" and "curldbg" options help to solve your problem.)
And if you found a problem, please post new issue.

Thanks in advance for your help.

<!-- gh-comment-id:172296547 --> @ggtakec commented on GitHub (Jan 17, 2016): I'm sorry about this Issue had been left at a long period of time. If you have this problem yet, please try to use latest code because some problem about multipart request etc were fixed. ("-f", "dbglevel" and "curldbg" options help to solve your problem.) And if you found a problem, please post new issue. Thanks in advance for your help.
Author
Owner

@bs-thomas commented on GitHub (Jul 4, 2017):

Hello there,

I still have the exact issue @sanemat is addressing. I have a log folder of 1000 files, and it works fine when I list folders with little or no files. But once I do "ls" on the log folder of 1000 files, it simply freezes.

root@732a6273f2c9:/mnt/s3/logs#  s3fs --version
Amazon Simple Storage Service File System V1.82(commit:620f6ec) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

I also tried to check the s3fs logs (/var/log/messages) as suggested on the wiki:
https://github.com/s3fs-fuse/s3fs-fuse/wiki/FAQ

However, no logs, and not even the file was there.

After I Ctrl+C, I cannot access any mounted files anymore (even those that do not work). It shows the error as follows:

root@732a6273f2c9:/mnt# cd s3
bash: cd: s3: Transport endpoint is not connected

Wondering if anyone else is facing this issue, or has any fixes or thoughts about it.

<!-- gh-comment-id:312906553 --> @bs-thomas commented on GitHub (Jul 4, 2017): Hello there, I still have the exact issue @sanemat is addressing. I have a log folder of 1000 files, and it works fine when I list folders with little or no files. But once I do "ls" on the log folder of 1000 files, it simply freezes. root@732a6273f2c9:/mnt/s3/logs# s3fs --version Amazon Simple Storage Service File System V1.82(commit:620f6ec) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. I also tried to check the s3fs logs (/var/log/messages) as suggested on the wiki: https://github.com/s3fs-fuse/s3fs-fuse/wiki/FAQ However, no logs, and not even the file was there. After I Ctrl+C, I cannot access any mounted files anymore (even those that do not work). It shows the error as follows: root@732a6273f2c9:/mnt# cd s3 bash: cd: s3: Transport endpoint is not connected Wondering if anyone else is facing this issue, or has any fixes or thoughts about it.
Author
Owner

@Izzette commented on GitHub (Oct 22, 2020):

I'm thinking I'm having this same issue too, despite the age, it seems not to work.

After I (finally) scan the entire directory index, s3fs does a bunch of this and says "Device not configured":

[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2545): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:insertV4Headers(2863): computing signature [HEAD] [/60007396-6089-4723-88fc-d1476e1240a0/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2545): HTTP response code 404 was returned, returning ENOENT
<!-- gh-comment-id:714562087 --> @Izzette commented on GitHub (Oct 22, 2020): I'm thinking I'm having this same issue too, despite the age, it seems not to work. After I (finally) scan the entire directory index, s3fs does a bunch of this and says "Device not configured": ``` [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2545): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:insertV4Headers(2863): computing signature [HEAD] [/60007396-6089-4723-88fc-d1476e1240a0/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2545): HTTP response code 404 was returned, returning ENOENT ```
Author
Owner

@kakadais commented on GitHub (Dec 28, 2020):

This ages problem happend to me too;;
Can't believe there's no answer still ;(

<!-- gh-comment-id:751882778 --> @kakadais commented on GitHub (Dec 28, 2020): This ages problem happend to me too;; Can't believe there's no answer still ;(
Author
Owner

@gaul commented on GitHub (Dec 29, 2020):

After I (finally) scan the entire directory index, s3fs does a bunch of this and says "Device not configured":

Please open a separate issue instead of tagging an unrelated 6-year old issue. You will likely need to collect a stack trace via gdb.

Can't believe there's no answer still ;(

Answers do not magically appear -- volunteers support s3fs with their free time. Please consider your tone when interacting with public issues which reflects on you professionally.

<!-- gh-comment-id:751912827 --> @gaul commented on GitHub (Dec 29, 2020): > After I (finally) scan the entire directory index, s3fs does a bunch of this and says "Device not configured": Please open a separate issue instead of tagging an unrelated 6-year old issue. You will likely need to collect a stack trace via gdb. > Can't believe there's no answer still ;( Answers do not magically appear -- volunteers support s3fs with their free time. Please consider your tone when interacting with public issues which reflects on you professionally.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#44
No description provided.