[GH-ISSUE #1255] S3FS fails randomly without error

kerem commented

2026-03-04 01:47:44 +03:00

Owner

Originally created by @simonlmay on GitHub (Mar 22, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1255

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

V1.86

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Version : 2.9.2
Release : 11.el7
Architecture: x86_64

Kernel information (uname -r)

3.10.0-1062.18.1.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="CentOS Linux"
VERSION="7 (Core)"

s3fs command line used, if applicable

/etc/fstab entry, if applicable

s3fs#xxxxxx.xxxxxxxx.com.au /storagemotasker fuse allow_other,use_cache="",uid=xxxx,gid=xxxx,use_path_request_style,dbglevel=info,curldbg,passwd_file=/etc/xxxxx,url=https://s3.ap-southeast-2.amazonaws.com 0 0

s3fs syslog messages (

messages_s3fs.log.zip

Details about issue

Issue has only been seen since upgrade to V1.86 on March 5th

other systems running previous versions not showing issue

Originally created by @simonlmay on GitHub (Mar 22, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1255 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) V1.86 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Version : 2.9.2 Release : 11.el7 Architecture: x86_64 #### Kernel information (uname -r) 3.10.0-1062.18.1.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="CentOS Linux" VERSION="7 (Core)" #### s3fs command line used, if applicable ``` ``` #### /etc/fstab entry, if applicable s3fs#xxxxxx.xxxxxxxx.com.au /storagemotasker fuse allow_other,use_cache="",uid=xxxx,gid=xxxx,use_path_request_style,dbglevel=info,curldbg,passwd_file=/etc/xxxxx,url=https://s3.ap-southeast-2.amazonaws.com 0 0 #### s3fs syslog messages ( [messages_s3fs.log.zip](https://github.com/s3fs-fuse/s3fs-fuse/files/4366463/messages_s3fs.log.zip) ### Details about issue Issue has only been seen since upgrade to V1.86 on March 5th other systems running previous versions not showing issue

kerem closed this issue

2026-03-04 01:47:45 +03:00

kerem commented

2026-03-04 01:47:45 +03:00

Author

Owner

@simonlmay commented on GitHub (Mar 25, 2020):

As in issue #1256
having set dbglevel=info the issue has not reoccured

@simonlmay commented on GitHub (Mar 25, 2020): As in issue #1256 having set dbglevel=info the issue has not reoccured

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@gaul commented on GitHub (Mar 25, 2020):

Could you run s3fs under gdb to see where the error is?

@gaul commented on GitHub (Mar 25, 2020): Could you run s3fs under gdb to see where the error is?

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@simonlmay commented on GitHub (Mar 26, 2020):

BTW it crashed this morning even with dbglevel=info

@simonlmay commented on GitHub (Mar 26, 2020): BTW it crashed this morning even with dbglevel=info

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@simonlmay commented on GitHub (Mar 26, 2020):

s3fs under gdb
this is not something I have do before.
been reading the docs and see I can use attach then continue
but not sure that is the best way to go forward.
happy to take some advise.

I was also thinking of removing dbglevel=info,curldbg from the stab options
hopefully it will crash sooner without it.

@simonlmay commented on GitHub (Mar 26, 2020): s3fs under gdb this is not something I have do before. been reading the docs and see I can use attach then continue but not sure that is the best way to go forward. happy to take some advise. I was also thinking of removing dbglevel=info,curldbg from the stab options hopefully it will crash sooner without it.

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@gaul commented on GitHub (Mar 26, 2020):

You can attach gdb to a running s3fs via gdb $(pidof s3fs). This will pause execution and if you type continue it will resume. If s3fs crashes, you can get the location of the crash via bt (backtrace). This will help us understand the issue.

@gaul commented on GitHub (Mar 26, 2020): You can attach gdb to a running s3fs via `gdb $(pidof s3fs)`. This will pause execution and if you type `continue` it will resume. If s3fs crashes, you can get the location of the crash via `bt` (backtrace). This will help us understand the issue.

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@simonlmay commented on GitHub (Mar 26, 2020):

I have changed the sftab
to
s3fs#storage.motasker.com.au /storagemotasker fuse allow_other,use_cache="",uid=1000,gid=1002,use_path_request_style,passwd_file=/etc/passwd-s3fs,url=https://s3.ap-southeast-2.amazonaws.com 0 0

I will see if that means it crashes more often.
I have a watchdog in place

I'll then sort out to do the gdb

Just being careful as this is on a production server

@simonlmay commented on GitHub (Mar 26, 2020): I have changed the sftab to s3fs#storage.motasker.com.au /storagemotasker fuse allow_other,use_cache="",uid=1000,gid=1002,use_path_request_style,passwd_file=/etc/passwd-s3fs,url=https://s3.ap-southeast-2.amazonaws.com 0 0 I will see if that means it crashes more often. I have a watchdog in place I'll then sort out to do the gdb Just being careful as this is on a production server

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@woodcoder commented on GitHub (Mar 28, 2020):

Possible duplicate of https://github.com/s3fs-fuse/s3fs-fuse/issues/1245#issuecomment-599047227 which is to do with stat files getting corrupted. Do you have a script or similar clearing the cache?

@woodcoder commented on GitHub (Mar 28, 2020): Possible duplicate of https://github.com/s3fs-fuse/s3fs-fuse/issues/1245#issuecomment-599047227 which is to do with stat files getting corrupted. Do you have a script or similar clearing the cache?

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@simonlmay commented on GitHub (Mar 28, 2020):

The bucket is mounted via fstab with use_cache=“”

Sent from my phone

From: Andy Wood notifications@github.com
Sent: Sunday, March 29, 2020 4:00:56 AM
To: s3fs-fuse/s3fs-fuse s3fs-fuse@noreply.github.com
Cc: simonlmay maysimon@gmail.com; Author author@noreply.github.com
Subject: Re: [s3fs-fuse/s3fs-fuse] S3FS fails randomly without error (#1255)

Possible duplicate of #1245 (comment)https://github.com/s3fs-fuse/s3fs-fuse/issues/1245#issuecomment-599047227 which is to do with stat files getting corrupted. Do you have a script or similar clearing the cache?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/s3fs-fuse/s3fs-fuse/issues/1255#issuecomment-605488871, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AO5AP5HWW5CG6TPJ6WFCSPTRJYUMRANCNFSM4LROZZIA.

@simonlmay commented on GitHub (Mar 28, 2020): The bucket is mounted via fstab with use_cache=“” Sent from my phone ________________________________ From: Andy Wood <notifications@github.com> Sent: Sunday, March 29, 2020 4:00:56 AM To: s3fs-fuse/s3fs-fuse <s3fs-fuse@noreply.github.com> Cc: simonlmay <maysimon@gmail.com>; Author <author@noreply.github.com> Subject: Re: [s3fs-fuse/s3fs-fuse] S3FS fails randomly without error (#1255) Possible duplicate of #1245 (comment)<https://github.com/s3fs-fuse/s3fs-fuse/issues/1245#issuecomment-599047227> which is to do with stat files getting corrupted. Do you have a script or similar clearing the cache? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://github.com/s3fs-fuse/s3fs-fuse/issues/1255#issuecomment-605488871>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO5AP5HWW5CG6TPJ6WFCSPTRJYUMRANCNFSM4LROZZIA>.

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@woodcoder commented on GitHub (Mar 29, 2020):

Ah sorry, hadn't appreciated that meant disabled. Probably not that then!

@woodcoder commented on GitHub (Mar 29, 2020): Ah sorry, hadn't appreciated that meant disabled. Probably not that then!

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@simonlmay commented on GitHub (Mar 30, 2020):

So currently there has not been another failure since Thu Mar 26 08:20:02 AEDT 2020
I expect this is because the usage has declined from the period before that, for obvious reasons.

If run gdb in a screen session and log the debug to a log.
attach and continue
when it crashes ( if it does ) I can send you that log.
Does that sound like a plan to works for you ?

@simonlmay commented on GitHub (Mar 30, 2020): So currently there has not been another failure since Thu Mar 26 08:20:02 AEDT 2020 I expect this is because the usage has declined from the period before that, for obvious reasons. If run gdb in a screen session and log the debug to a log. attach and continue when it crashes ( if it does ) I can send you that log. Does that sound like a plan to works for you ?

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@esteban1983cl commented on GitHub (Apr 7, 2020):

I have the same issue and I solved it deleting large files. For instance:
In a folder "x" I had 180 files aprox. with size over 170Mb .
When I list the content using linux command, it takes too much on response, so I have to cancel the command.
So when I deleted the files, s3fs doesn't crash anymore.
I expect this issue transforms in a feature to enhance the performance with linux commands like
'ls', 'find', etc.

Thanks

@esteban1983cl commented on GitHub (Apr 7, 2020): I have the same issue and I solved it deleting large files. For instance: In a folder "x" I had 180 files aprox. with size over 170Mb . When I list the content using linux command, it takes too much on response, so I have to cancel the command. So when I deleted the files, s3fs doesn't crash anymore. I expect this issue transforms in a feature to enhance the performance with linux commands like 'ls', 'find', etc. Thanks

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@simonlmay commented on GitHub (Apr 7, 2020):

I can run find /s3mountpoint
without crashing s3fs
there are 52202 files with a total of 36G of space used.
So not sure you are having the same issue.
What I have noticed is that is has not crashed in the past 14 days.
Without being able to find the trigger to reproduce the issue, it's hard to progress this issue.

@simonlmay commented on GitHub (Apr 7, 2020): I can run find /s3mountpoint without crashing s3fs there are 52202 files with a total of 36G of space used. So not sure you are having the same issue. What I have noticed is that is has not crashed in the past 14 days. Without being able to find the trigger to reproduce the issue, it's hard to progress this issue.

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@esteban1983cl commented on GitHub (Apr 7, 2020):

Hi, this issue is related to "Transport is not connected" error message.
I solved the issue changing cache configuration.

Before

s3fs#xxxxxxxxxxxxxxxxxxx    /ecommerce/prod/       fuse _netdev,allow_other,uid=510,gid=501,use_cache=/mnt/data,umask=002,iam_role=auto,endpoint=us-east-1,nonempty 0 0

After

s3fs#xxxxxxxxxxxxxxxxxxx    /ecommerce/prod/       fuse _netdev,allow_other,uid=510,gid=501,use_cache="",umask=002,iam_role=auto,endpoint=us-east-1,nonempty 0 0

s3fs-fuse version: v1.86

fuse version
Version : 2.9.2
Release : 11.amzn2
Architecture: x86_64

Kernel Information
4.14.114-105.126.amzn2.x86_64

cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"

@esteban1983cl commented on GitHub (Apr 7, 2020): Hi, this issue is related to "Transport is not connected" error message. I solved the issue changing cache configuration. Before ```bash s3fs#xxxxxxxxxxxxxxxxxxx /ecommerce/prod/ fuse _netdev,allow_other,uid=510,gid=501,use_cache=/mnt/data,umask=002,iam_role=auto,endpoint=us-east-1,nonempty 0 0 ``` After ```bash s3fs#xxxxxxxxxxxxxxxxxxx /ecommerce/prod/ fuse _netdev,allow_other,uid=510,gid=501,use_cache="",umask=002,iam_role=auto,endpoint=us-east-1,nonempty 0 0 ``` **s3fs-fuse version: v1.86** **fuse version** Version : 2.9.2 Release : 11.amzn2 Architecture: x86_64 **Kernel Information** 4.14.114-105.126.amzn2.x86_64 **cat /etc/os-release** NAME="Amazon Linux" VERSION="2"

kerem commented

2026-03-04 01:47:46 +03:00

Author

Owner

@simonlmay commented on GitHub (Apr 7, 2020):

Hi Esteban
What issue number is that?

Strange though as the cache was already set to use_cache=“” when the issue arose.
Also there is no messages in the logs to indicate that it is related to "Transport is not connected"

As the system has now been stable for over two weeks.
I will close this issue
If I'm able to get a debug log I will open a new issue

Thanks Andrew for your assistance re gdb

@simonlmay commented on GitHub (Apr 7, 2020): Hi Esteban What issue number is that? Strange though as the cache was already set to use_cache=“” when the issue arose. Also there is no messages in the logs to indicate that it is related to "Transport is not connected" As the system has now been stable for over two weeks. I will close this issue If I'm able to get a debug log I will open a new issue Thanks Andrew for your assistance re gdb

kerem referenced this issue

2026-03-04 02:01:14 +03:00

[PR #672] [CLOSED] README.md Addition #1598

kerem referenced this issue

2026-03-04 02:01:16 +03:00

[PR #679] [MERGED] README.md Addition #1601