starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #2763] fio stuck when using libaio #1291

New issue

Closed

opened 2026-03-04 01:52:52 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:52:52 +03:00

Owner

Originally created by @myeyre on GitHub (Dec 2, 2025).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2763

Additional Information

Version of s3fs being used (`s3fs --version`)

V1.96

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

fuse3_3.10.5-1build1_amd64

Provider (`AWS`, `OVH`, `Hetzner`, `iDrive E2`, ...)

JuiceFS 1.3.0 S3 Gateway

Kernel information (`uname -r`)

4.19.0-240.23.21.e18_2.bclinux.x86_64

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

I'm running s3fs inside a container, the container os is:
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

the host os is bclinux, a rhel compat distro in china:
NAME="BigCloud Enterprise Linux"
VERSION="8.2 (Core)"
ID="bclinux"
ID_LIKE="rhel fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:an8"
PRETTY_NAME="BigCloud Enterprise Linux 8.2 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:bclinux:bclinux:8"
HOME_URL="https://mirrors.bclinux.org/"
BUG_REPORT_URL="https://bugs.bclinux.org/"

BCLINUX_BUGZILLA_PRODUCT="BigCloud Enterprise Linux 8 (Core)"
BCLINUX_BUGZILLA_PRODUCT_VERSION=8.2
BCLINUX_SUPPORT_PRODUCT="BigCloud Enterprise Linux 8 (Core)"
BCLINUX_SUPPORT_PRODUCT_VERSION=8.2

How to run s3fs, if applicable

[x] command line
[] /etc/fstab

s3fs mybucket:/ /mnt/s3fs  -o url=http://xxxx:8088/ -o use_path_request_style -o parallel_count=10000 -o rw

parallel_count changed to 10000 according to:
https://github.com/s3fs-fuse/s3fs-fuse/issues/2077

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

n/a

Details about issue

I'm running s3fs inside a container, fio test stuck when using libaio. while it happened, df/ls both stuck.

df:

ls:

strace:

gdb:

Originally created by @myeyre on GitHub (Dec 2, 2025). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2763  ### Additional Information #### Version of s3fs being used (`s3fs --version`)  V1.96 #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)  fuse3_3.10.5-1build1_amd64 #### Provider (`AWS`, `OVH`, `Hetzner`, `iDrive E2`, ...)  JuiceFS 1.3.0 S3 Gateway #### Kernel information (`uname -r`)  4.19.0-240.23.21.e18_2.bclinux.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`)  I'm running s3fs inside a container, the container os is: PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy the host os is bclinux, a rhel compat distro in china: NAME="BigCloud Enterprise Linux" VERSION="8.2 (Core)" ID="bclinux" ID_LIKE="rhel fedora" VERSION_ID="8.2" PLATFORM_ID="platform:an8" PRETTY_NAME="BigCloud Enterprise Linux 8.2 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:bclinux:bclinux:8" HOME_URL="https://mirrors.bclinux.org/" BUG_REPORT_URL="https://bugs.bclinux.org/" BCLINUX_BUGZILLA_PRODUCT="BigCloud Enterprise Linux 8 (Core)" BCLINUX_BUGZILLA_PRODUCT_VERSION=8.2 BCLINUX_SUPPORT_PRODUCT="BigCloud Enterprise Linux 8 (Core)" BCLINUX_SUPPORT_PRODUCT_VERSION=8.2 #### How to run s3fs, if applicable  [x] command line [] /etc/fstab  ``` s3fs mybucket:/ /mnt/s3fs -o url=http://xxxx:8088/ -o use_path_request_style -o parallel_count=10000 -o rw ``` parallel_count changed to 10000 according to: https://github.com/s3fs-fuse/s3fs-fuse/issues/2077 #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  ``` n/a ``` ### Details about issue  I'm running s3fs inside a container, fio test stuck when using libaio. while it happened, df/ls both stuck. <img width="1434" height="186" alt="Image" src="https://github.com/user-attachments/assets/0d6dc309-aa96-4934-ad71-23d0658e8254" /> df: <img width="244" height="31" alt="Image" src="https://github.com/user-attachments/assets/1ec1a786-cfc9-4b93-a85b-e436c282e14a" /> ls: <img width="253" height="61" alt="Image" src="https://github.com/user-attachments/assets/251eb70e-49aa-4ac1-996b-b08648a506d4" /> strace: <img width="1074" height="70" alt="Image" src="https://github.com/user-attachments/assets/e6f3358d-bd34-495b-b15e-4118bb38c0c5" /> gdb: <img width="1813" height="231" alt="Image" src="https://github.com/user-attachments/assets/5d577126-0447-4df0-9af9-1bd37268e9f7" />

kerem closed this issue

2026-03-04 01:52:52 +03:00

kerem commented

2026-03-04 01:52:53 +03:00

Author

Owner

@myeyre commented on GitHub (Dec 3, 2025):

It seemed that the juicefs gateway got throttled under high load.

We'll look into the juicefs side, it's not a s3fs problem anyway. Closing this issue.

@myeyre commented on GitHub (Dec 3, 2025): It seemed that the juicefs gateway got throttled under high load. We'll look into the juicefs side, it's not a s3fs problem anyway. Closing this issue.

kerem commented

2026-03-04 01:52:53 +03:00

Author

Owner

@gaul commented on GitHub (Dec 7, 2025):

Could you share any logs from s3fs during this symptom? You should see something like:

"HTTP response code %ld was returned, slowing down"

if juicefs returns an HTTP 429, 500, or 503 error. This has exponential backoff behavior and retries up to -o retries times (default 5).

@gaul commented on GitHub (Dec 7, 2025): Could you share any logs from s3fs during this symptom? You should see something like: ``` "HTTP response code %ld was returned, slowing down" ``` if juicefs returns an HTTP 429, 500, or 503 error. This has exponential backoff behavior and retries up to `-o retries` times (default 5).

kerem referenced this issue

2026-03-04 02:02:46 +03:00

[PR #1312] [MERGED] Fixed a bug about serializing from cache file #1914

kerem referenced this issue

2026-03-04 02:02:49 +03:00

[PR #1334] [MERGED] Added SIGUSR1 option for cache file integrity test #1925