starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 05:16:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2019] When tested with fio and the -numjobs parameter is added, performance drops considerably compared to not adding it #1013

New issue
Closed
opened 2026-03-04 01:50:38 +03:00 by kerem · 4 comments
kerem commented 2026-03-04 01:50:38 +03:00
Owner
Copy link

Originally created by @zhijian-pro on GitHub (Aug 10, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2019

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.90 (commit:unknown) with GnuTLS(gcrypt)
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

root@tmp-fio-bench:~# find / -name libfuse.so*
/snap/lxd/22923/lib/x86_64-linux-gnu/libfuse.so.2
/snap/lxd/22923/lib/x86_64-linux-gnu/libfuse.so.2.9.9
/usr/lib/x86_64-linux-gnu/libfuse.so.2.9.9
/usr/lib/x86_64-linux-gnu/libfuse.so.2

Kernel information (uname -r)

_command result: 5.15.0-1015-aws

GNU/Linux Distribution, if applicable (cat /etc/os-release)

PRETTY_NAME="Ubuntu 22.04 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

s3fs command line used, if applicable

s3fs zjtest1 /tmp/s3fs1 -o endpoint=ap-southeast-1 -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel=info -f -o curldbg

Details about issue

When tested with fio and the -numjobs parameter is added, performance drops considerably compared to not adding it.
The test machine is AWS t2.small.

root@tmp-fio-bench:~# fio --name=sequential-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G
sequential-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
sequential-read: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=47.8MiB/s][r=11 IOPS][eta 00m:00s]
sequential-read: (groupid=0, jobs=1): err= 0: pid=19148: Wed Aug 10 03:23:51 2022
  read: IOPS=16, BW=67.8MiB/s (71.1MB/s)(2048MiB/30216msec)
    clat (msec): min=3, max=935, avg=58.98, stdev=189.27
     lat (msec): min=3, max=935, avg=58.99, stdev=189.27
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    5], 80.00th=[    6], 90.00th=[    7], 95.00th=[  634],
     | 99.00th=[  810], 99.50th=[  818], 99.90th=[  936], 99.95th=[  936],
     | 99.99th=[  936]
   bw (  KiB/s): min=16384, max=106496, per=100.00%, avg=95024.19, stdev=21750.68, samples=43
   iops        : min=    4, max=   26, avg=23.14, stdev= 5.30, samples=43
  lat (msec)   : 4=66.60%, 10=24.02%, 20=1.37%, 500=0.39%, 750=3.12%
  lat (msec)   : 1000=4.49%
  cpu          : usr=0.01%, sys=3.48%, ctx=16536, majf=0, minf=1035
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s (71.1MB/s-71.1MB/s), io=2048MiB (2147MB), run=30216-30216msec


root@tmp-fio-bench:~# fio --name=big-file-multi-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G --numjobs=4
big-file-multi-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
...
fio-3.28
Starting 4 processes
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
Jobs: 2 (f=2): [_(1),R(1),_(1),R(1)][100.0%][r=168MiB/s][r=41 IOPS][eta 00m:00s]
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21020: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/131104msec)
    clat (msec): min=16, max=3379, avg=256.01, stdev=641.99
     lat (msec): min=16, max=3379, avg=256.01, stdev=641.99
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[  995], 95.00th=[ 2072],
     | 99.00th=[ 3004], 99.50th=[ 3071], 99.90th=[ 3373], 99.95th=[ 3373],
     | 99.99th=[ 3373]
   bw (  KiB/s): min= 8175, max=98304, per=71.82%, avg=45595.01, stdev=37884.57, samples=90
   iops        : min=    1, max=   24, avg=11.08, stdev= 9.23, samples=90
  lat (msec)   : 20=35.55%, 50=49.80%, 250=0.20%, 750=1.76%, 1000=2.73%
  lat (msec)   : 2000=4.88%, >=2000=5.08%
  cpu          : usr=0.01%, sys=1.00%, ctx=15877, majf=0, minf=1038
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21021: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.5MiB/s (16.3MB/s)(2048MiB/132144msec)
    clat (msec): min=15, max=3377, avg=258.05, stdev=643.26
     lat (msec): min=15, max=3377, avg=258.06, stdev=643.26
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   23], 90.00th=[  877], 95.00th=[ 1770],
     | 99.00th=[ 3071], 99.50th=[ 3138], 99.90th=[ 3373], 99.95th=[ 3373],
     | 99.99th=[ 3373]
   bw (  KiB/s): min= 8192, max=106283, per=71.53%, avg=45405.58, stdev=37512.44, samples=92
   iops        : min=    2, max=   25, avg=11.04, stdev= 9.13, samples=92
  lat (msec)   : 20=33.01%, 50=51.76%, 750=2.54%, 1000=2.73%, 2000=5.27%
  lat (msec)   : >=2000=4.69%
  cpu          : usr=0.01%, sys=0.99%, ctx=15910, majf=0, minf=1038
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21022: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/130901msec)
    clat (msec): min=16, max=3103, avg=255.62, stdev=620.89
     lat (msec): min=16, max=3103, avg=255.62, stdev=620.89
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   23], 90.00th=[  927], 95.00th=[ 2072],
     | 99.00th=[ 2769], 99.50th=[ 2937], 99.90th=[ 3104], 99.95th=[ 3104],
     | 99.99th=[ 3104]
   bw (  KiB/s): min= 8175, max=98304, per=68.77%, avg=43654.97, stdev=37028.46, samples=94
   iops        : min=    1, max=   24, avg=10.61, stdev= 9.02, samples=94
  lat (msec)   : 20=34.77%, 50=49.80%, 750=1.76%, 1000=4.10%, 2000=4.49%
  lat (msec)   : >=2000=5.08%
  cpu          : usr=0.00%, sys=1.00%, ctx=15942, majf=2, minf=1036
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21023: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.6MiB/s (16.3MB/s)(2048MiB/131632msec)
    clat (msec): min=7, max=3088, avg=257.06, stdev=677.23
     lat (msec): min=7, max=3088, avg=257.06, stdev=677.23
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   22],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[  818], 95.00th=[ 2232],
     | 99.00th=[ 3037], 99.50th=[ 3071], 99.90th=[ 3104], 99.95th=[ 3104],
     | 99.99th=[ 3104]
   bw (  KiB/s): min= 8192, max=98304, per=76.52%, avg=48573.40, stdev=38062.27, samples=86
   iops        : min=    2, max=   24, avg=11.81, stdev= 9.26, samples=86
  lat (msec)   : 10=0.20%, 20=33.98%, 50=52.15%, 750=2.15%, 1000=2.34%
  lat (msec)   : 2000=3.32%, >=2000=5.86%
  cpu          : usr=0.01%, sys=1.00%, ctx=15933, majf=0, minf=1038
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=62.0MiB/s (65.0MB/s), 15.5MiB/s-15.6MiB/s (16.3MB/s-16.4MB/s), io=8192MiB (8590MB), run=130901-132144msec


root@tmp-fio-bench:~# fio --name=sequential-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G
sequential-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
sequential-read: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=47.8MiB/s][r=11 IOPS][eta 00m:00s]
sequential-read: (groupid=0, jobs=1): err= 0: pid=19148: Wed Aug 10 03:23:51 2022
  read: IOPS=16, BW=67.8MiB/s (71.1MB/s)(2048MiB/30216msec)
    clat (msec): min=3, max=935, avg=58.98, stdev=189.27
     lat (msec): min=3, max=935, avg=58.99, stdev=189.27
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    5], 80.00th=[    6], 90.00th=[    7], 95.00th=[  634],
     | 99.00th=[  810], 99.50th=[  818], 99.90th=[  936], 99.95th=[  936],
     | 99.99th=[  936]
   bw (  KiB/s): min=16384, max=106496, per=100.00%, avg=95024.19, stdev=21750.68, samples=43
   iops        : min=    4, max=   26, avg=23.14, stdev= 5.30, samples=43
  lat (msec)   : 4=66.60%, 10=24.02%, 20=1.37%, 500=0.39%, 750=3.12%
  lat (msec)   : 1000=4.49%
  cpu          : usr=0.01%, sys=3.48%, ctx=16536, majf=0, minf=1035
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s (71.1MB/s-71.1MB/s), io=2048MiB (2147MB), run=30216-30216msec
root@tmp-fio-bench:~# fio --name=big-file-multi-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G --numjobs=4
big-file-multi-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
...
fio-3.28
Starting 4 processes
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
big-file-multi-read: Laying out IO file (1 file / 2048MiB)
Jobs: 2 (f=2): [_(1),R(1),_(1),R(1)][100.0%][r=168MiB/s][r=41 IOPS][eta 00m:00s]
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21020: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/131104msec)
    clat (msec): min=16, max=3379, avg=256.01, stdev=641.99
     lat (msec): min=16, max=3379, avg=256.01, stdev=641.99
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[  995], 95.00th=[ 2072],
     | 99.00th=[ 3004], 99.50th=[ 3071], 99.90th=[ 3373], 99.95th=[ 3373],
     | 99.99th=[ 3373]
   bw (  KiB/s): min= 8175, max=98304, per=71.82%, avg=45595.01, stdev=37884.57, samples=90
   iops        : min=    1, max=   24, avg=11.08, stdev= 9.23, samples=90
  lat (msec)   : 20=35.55%, 50=49.80%, 250=0.20%, 750=1.76%, 1000=2.73%
  lat (msec)   : 2000=4.88%, >=2000=5.08%
  cpu          : usr=0.01%, sys=1.00%, ctx=15877, majf=0, minf=1038
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21021: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.5MiB/s (16.3MB/s)(2048MiB/132144msec)
    clat (msec): min=15, max=3377, avg=258.05, stdev=643.26
     lat (msec): min=15, max=3377, avg=258.06, stdev=643.26
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   23], 90.00th=[  877], 95.00th=[ 1770],
     | 99.00th=[ 3071], 99.50th=[ 3138], 99.90th=[ 3373], 99.95th=[ 3373],
     | 99.99th=[ 3373]
   bw (  KiB/s): min= 8192, max=106283, per=71.53%, avg=45405.58, stdev=37512.44, samples=92
   iops        : min=    2, max=   25, avg=11.04, stdev= 9.13, samples=92
  lat (msec)   : 20=33.01%, 50=51.76%, 750=2.54%, 1000=2.73%, 2000=5.27%
  lat (msec)   : >=2000=4.69%
  cpu          : usr=0.01%, sys=0.99%, ctx=15910, majf=0, minf=1038
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21022: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/130901msec)
    clat (msec): min=16, max=3103, avg=255.62, stdev=620.89
     lat (msec): min=16, max=3103, avg=255.62, stdev=620.89
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   23], 90.00th=[  927], 95.00th=[ 2072],
     | 99.00th=[ 2769], 99.50th=[ 2937], 99.90th=[ 3104], 99.95th=[ 3104],
     | 99.99th=[ 3104]
   bw (  KiB/s): min= 8175, max=98304, per=68.77%, avg=43654.97, stdev=37028.46, samples=94
   iops        : min=    1, max=   24, avg=10.61, stdev= 9.02, samples=94
  lat (msec)   : 20=34.77%, 50=49.80%, 750=1.76%, 1000=4.10%, 2000=4.49%
  lat (msec)   : >=2000=5.08%
  cpu          : usr=0.00%, sys=1.00%, ctx=15942, majf=2, minf=1036
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21023: Wed Aug 10 03:34:27 2022
  read: IOPS=3, BW=15.6MiB/s (16.3MB/s)(2048MiB/131632msec)
    clat (msec): min=7, max=3088, avg=257.06, stdev=677.23
     lat (msec): min=7, max=3088, avg=257.06, stdev=677.23
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   20], 40.00th=[   21], 50.00th=[   21], 60.00th=[   22],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[  818], 95.00th=[ 2232],
     | 99.00th=[ 3037], 99.50th=[ 3071], 99.90th=[ 3104], 99.95th=[ 3104],
     | 99.99th=[ 3104]
   bw (  KiB/s): min= 8192, max=98304, per=76.52%, avg=48573.40, stdev=38062.27, samples=86
   iops        : min=    2, max=   24, avg=11.81, stdev= 9.26, samples=86
  lat (msec)   : 10=0.20%, 20=33.98%, 50=52.15%, 750=2.15%, 1000=2.34%
  lat (msec)   : 2000=3.32%, >=2000=5.86%
  cpu          : usr=0.01%, sys=1.00%, ctx=15933, majf=0, minf=1038
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=62.0MiB/s (65.0MB/s), 15.5MiB/s-15.6MiB/s (16.3MB/s-16.4MB/s), io=8192MiB (8590MB), run=130901-132144msec
root@tmp-fio-bench:~#
root@tmp-fio-bench:~#
root@tmp-fio-bench:~#
root@tmp-fio-bench:~# fio --name=sequential-write --directory=/tmp/s3fs1 --rw=write --refill_buffers --bs=4M --size=2G --end_fsync=1
sequential-write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
sequential-write: Laying out IO file (1 file / 2048MiB)
Jobs: 1 (f=1): [F(1)][100.0%][eta 00m:00s]
sequential-write: (groupid=0, jobs=1): err= 0: pid=22175: Wed Aug 10 03:42:52 2022
  write: IOPS=8, BW=33.2MiB/s (34.8MB/s)(2048MiB/61621msec); 0 zone resets
    clat (msec): min=5, max=206, avg=54.90, stdev=30.54
     lat (msec): min=5, max=206, avg=54.90, stdev=30.54
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    6], 20.00th=[   33],
     | 30.00th=[   42], 40.00th=[   48], 50.00th=[   54], 60.00th=[   70],
     | 70.00th=[   75], 80.00th=[   80], 90.00th=[   86], 95.00th=[   99],
     | 99.00th=[  126], 99.50th=[  128], 99.90th=[  207], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=49152, max=540672, per=100.00%, avg=73435.96, stdev=66642.02, samples=57
   iops        : min=   12, max=  132, avg=17.89, stdev=16.28, samples=57
  lat (msec)   : 10=17.38%, 50=27.15%, 100=51.37%, 250=4.10%
  cpu          : usr=0.76%, sys=1.69%, ctx=17547, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=2048MiB (2147MB), run=61621-61621msec



root@tmp-fio-bench:~# fio --name=big-file-multi-write --directory=/tmp/s3fs1 --rw=write --refill_buffers --bs=4M --size=2G --numjobs=4 --end_fsync=1
big-file-multi-write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
...
fio-3.28
Starting 4 processes
big-file-multi-write: Laying out IO file (1 file / 2048MiB)
big-file-multi-write: Laying out IO file (1 file / 2048MiB)
big-file-multi-write: Laying out IO file (1 file / 2048MiB)
big-file-multi-write: Laying out IO file (1 file / 2048MiB)
Jobs: 3 (f=3): [_(1),F(2),f(1)][100.0%][eta 00m:00s]
big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22385: Wed Aug 10 03:47:37 2022
  write: IOPS=3, BW=12.7MiB/s (13.3MB/s)(2048MiB/160853msec); 0 zone resets
    clat (msec): min=13, max=324, avg=248.09, stdev=54.24
     lat (msec): min=13, max=324, avg=248.09, stdev=54.24
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[  159], 10.00th=[  232], 20.00th=[  243],
     | 30.00th=[  249], 40.00th=[  253], 50.00th=[  257], 60.00th=[  264],
     | 70.00th=[  271], 80.00th=[  275], 90.00th=[  284], 95.00th=[  288],
     | 99.00th=[  305], 99.50th=[  305], 99.90th=[  326], 99.95th=[  326],
     | 99.99th=[  326]
   bw (  KiB/s): min= 8175, max=155648, per=51.37%, avg=16420.09, stdev=9463.81, samples=255
   iops        : min=    1, max=   38, avg= 3.96, stdev= 2.33, samples=255
  lat (msec)   : 20=1.76%, 50=2.93%, 100=0.20%, 250=28.71%, 500=66.41%
  cpu          : usr=0.35%, sys=0.65%, ctx=21283, majf=2, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22386: Wed Aug 10 03:47:37 2022
  write: IOPS=1, BW=7992KiB/s (8184kB/s)(2048MiB/262413msec); 0 zone resets
    clat (msec): min=9, max=33348, avg=313.15, stdev=1463.77
     lat (msec): min=9, max=33348, avg=313.16, stdev=1463.77
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[  169], 10.00th=[  230], 20.00th=[  243],
     | 30.00th=[  249], 40.00th=[  253], 50.00th=[  259], 60.00th=[  264],
     | 70.00th=[  271], 80.00th=[  275], 90.00th=[  284], 95.00th=[  292],
     | 99.00th=[  300], 99.50th=[  313], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min= 8192, max=73728, per=50.85%, avg=16255.35, stdev=5664.84, samples=256
   iops        : min=    2, max=   18, avg= 3.95, stdev= 1.39, samples=256
  lat (msec)   : 10=0.20%, 20=0.20%, 50=4.10%, 100=0.20%, 250=29.30%
  lat (msec)   : 500=65.82%, >=2000=0.20%
  cpu          : usr=0.18%, sys=0.44%, ctx=21156, majf=1, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22387: Wed Aug 10 03:47:37 2022
  write: IOPS=1, BW=8001KiB/s (8193kB/s)(2048MiB/262118msec); 0 zone resets
    clat (msec): min=14, max=33349, avg=313.31, stdev=1463.82
     lat (msec): min=14, max=33349, avg=313.32, stdev=1463.82
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[  153], 10.00th=[  232], 20.00th=[  245],
     | 30.00th=[  249], 40.00th=[  253], 50.00th=[  257], 60.00th=[  264],
     | 70.00th=[  271], 80.00th=[  275], 90.00th=[  284], 95.00th=[  292],
     | 99.00th=[  305], 99.50th=[  313], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min= 8175, max=80170, per=50.92%, avg=16279.29, stdev=6017.65, samples=256
   iops        : min=    1, max=   19, avg= 3.92, stdev= 1.48, samples=256
  lat (msec)   : 20=0.59%, 50=3.91%, 100=0.20%, 250=30.27%, 500=64.84%
  lat (msec)   : >=2000=0.20%
  cpu          : usr=0.20%, sys=0.41%, ctx=20918, majf=0, minf=20
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22388: Wed Aug 10 03:47:37 2022
  write: IOPS=1, BW=8017KiB/s (8209kB/s)(2048MiB/261598msec); 0 zone resets
    clat (msec): min=9, max=33123, avg=313.15, stdev=1453.87
     lat (msec): min=9, max=33123, avg=313.15, stdev=1453.87
    clat percentiles (msec):
     |  1.00th=[   19],  5.00th=[  163], 10.00th=[  230], 20.00th=[  243],
     | 30.00th=[  249], 40.00th=[  253], 50.00th=[  259], 60.00th=[  266],
     | 70.00th=[  271], 80.00th=[  275], 90.00th=[  284], 95.00th=[  292],
     | 99.00th=[  317], 99.50th=[  334], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min= 8175, max=155648, per=51.05%, avg=16318.77, stdev=9403.45, samples=256
   iops        : min=    1, max=   38, avg= 3.94, stdev= 2.31, samples=256
  lat (msec)   : 10=0.20%, 20=1.56%, 50=2.73%, 250=28.91%, 500=66.41%
  lat (msec)   : >=2000=0.20%
  cpu          : usr=0.19%, sys=0.43%, ctx=21457, majf=5, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=31.2MiB/s (32.7MB/s), 7992KiB/s-12.7MiB/s (8184kB/s-13.3MB/s), io=8192MiB (8590MB), run=160853-262413msec
Originally created by @zhijian-pro on GitHub (Aug 10, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2019 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.90 (commit:unknown) with GnuTLS(gcrypt) Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` root@tmp-fio-bench:~# find / -name libfuse.so* /snap/lxd/22923/lib/x86_64-linux-gnu/libfuse.so.2 /snap/lxd/22923/lib/x86_64-linux-gnu/libfuse.so.2.9.9 /usr/lib/x86_64-linux-gnu/libfuse.so.2.9.9 /usr/lib/x86_64-linux-gnu/libfuse.so.2 ``` #### Kernel information (uname -r) _command result: 5.15.0-1015-aws #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` PRETTY_NAME="Ubuntu 22.04 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` #### s3fs command line used, if applicable ``` s3fs zjtest1 /tmp/s3fs1 -o endpoint=ap-southeast-1 -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel=info -f -o curldbg ``` ### Details about issue When tested with fio and the -numjobs parameter is added, performance drops considerably compared to not adding it. The test machine is AWS t2.small. ``` root@tmp-fio-bench:~# fio --name=sequential-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G sequential-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.28 Starting 1 process sequential-read: Laying out IO file (1 file / 2048MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=47.8MiB/s][r=11 IOPS][eta 00m:00s] sequential-read: (groupid=0, jobs=1): err= 0: pid=19148: Wed Aug 10 03:23:51 2022 read: IOPS=16, BW=67.8MiB/s (71.1MB/s)(2048MiB/30216msec) clat (msec): min=3, max=935, avg=58.98, stdev=189.27 lat (msec): min=3, max=935, avg=58.99, stdev=189.27 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 634], | 99.00th=[ 810], 99.50th=[ 818], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min=16384, max=106496, per=100.00%, avg=95024.19, stdev=21750.68, samples=43 iops : min= 4, max= 26, avg=23.14, stdev= 5.30, samples=43 lat (msec) : 4=66.60%, 10=24.02%, 20=1.37%, 500=0.39%, 750=3.12% lat (msec) : 1000=4.49% cpu : usr=0.01%, sys=3.48%, ctx=16536, majf=0, minf=1035 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s (71.1MB/s-71.1MB/s), io=2048MiB (2147MB), run=30216-30216msec root@tmp-fio-bench:~# fio --name=big-file-multi-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G --numjobs=4 big-file-multi-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 ... fio-3.28 Starting 4 processes big-file-multi-read: Laying out IO file (1 file / 2048MiB) big-file-multi-read: Laying out IO file (1 file / 2048MiB) big-file-multi-read: Laying out IO file (1 file / 2048MiB) big-file-multi-read: Laying out IO file (1 file / 2048MiB) Jobs: 2 (f=2): [_(1),R(1),_(1),R(1)][100.0%][r=168MiB/s][r=41 IOPS][eta 00m:00s] big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21020: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/131104msec) clat (msec): min=16, max=3379, avg=256.01, stdev=641.99 lat (msec): min=16, max=3379, avg=256.01, stdev=641.99 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 995], 95.00th=[ 2072], | 99.00th=[ 3004], 99.50th=[ 3071], 99.90th=[ 3373], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 8175, max=98304, per=71.82%, avg=45595.01, stdev=37884.57, samples=90 iops : min= 1, max= 24, avg=11.08, stdev= 9.23, samples=90 lat (msec) : 20=35.55%, 50=49.80%, 250=0.20%, 750=1.76%, 1000=2.73% lat (msec) : 2000=4.88%, >=2000=5.08% cpu : usr=0.01%, sys=1.00%, ctx=15877, majf=0, minf=1038 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21021: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.5MiB/s (16.3MB/s)(2048MiB/132144msec) clat (msec): min=15, max=3377, avg=258.05, stdev=643.26 lat (msec): min=15, max=3377, avg=258.06, stdev=643.26 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 23], 90.00th=[ 877], 95.00th=[ 1770], | 99.00th=[ 3071], 99.50th=[ 3138], 99.90th=[ 3373], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 8192, max=106283, per=71.53%, avg=45405.58, stdev=37512.44, samples=92 iops : min= 2, max= 25, avg=11.04, stdev= 9.13, samples=92 lat (msec) : 20=33.01%, 50=51.76%, 750=2.54%, 1000=2.73%, 2000=5.27% lat (msec) : >=2000=4.69% cpu : usr=0.01%, sys=0.99%, ctx=15910, majf=0, minf=1038 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21022: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/130901msec) clat (msec): min=16, max=3103, avg=255.62, stdev=620.89 lat (msec): min=16, max=3103, avg=255.62, stdev=620.89 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 23], 90.00th=[ 927], 95.00th=[ 2072], | 99.00th=[ 2769], 99.50th=[ 2937], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8175, max=98304, per=68.77%, avg=43654.97, stdev=37028.46, samples=94 iops : min= 1, max= 24, avg=10.61, stdev= 9.02, samples=94 lat (msec) : 20=34.77%, 50=49.80%, 750=1.76%, 1000=4.10%, 2000=4.49% lat (msec) : >=2000=5.08% cpu : usr=0.00%, sys=1.00%, ctx=15942, majf=2, minf=1036 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21023: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.6MiB/s (16.3MB/s)(2048MiB/131632msec) clat (msec): min=7, max=3088, avg=257.06, stdev=677.23 lat (msec): min=7, max=3088, avg=257.06, stdev=677.23 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 818], 95.00th=[ 2232], | 99.00th=[ 3037], 99.50th=[ 3071], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8192, max=98304, per=76.52%, avg=48573.40, stdev=38062.27, samples=86 iops : min= 2, max= 24, avg=11.81, stdev= 9.26, samples=86 lat (msec) : 10=0.20%, 20=33.98%, 50=52.15%, 750=2.15%, 1000=2.34% lat (msec) : 2000=3.32%, >=2000=5.86% cpu : usr=0.01%, sys=1.00%, ctx=15933, majf=0, minf=1038 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=62.0MiB/s (65.0MB/s), 15.5MiB/s-15.6MiB/s (16.3MB/s-16.4MB/s), io=8192MiB (8590MB), run=130901-132144msec root@tmp-fio-bench:~# fio --name=sequential-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G sequential-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.28 Starting 1 process sequential-read: Laying out IO file (1 file / 2048MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=47.8MiB/s][r=11 IOPS][eta 00m:00s] sequential-read: (groupid=0, jobs=1): err= 0: pid=19148: Wed Aug 10 03:23:51 2022 read: IOPS=16, BW=67.8MiB/s (71.1MB/s)(2048MiB/30216msec) clat (msec): min=3, max=935, avg=58.98, stdev=189.27 lat (msec): min=3, max=935, avg=58.99, stdev=189.27 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 5], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 634], | 99.00th=[ 810], 99.50th=[ 818], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min=16384, max=106496, per=100.00%, avg=95024.19, stdev=21750.68, samples=43 iops : min= 4, max= 26, avg=23.14, stdev= 5.30, samples=43 lat (msec) : 4=66.60%, 10=24.02%, 20=1.37%, 500=0.39%, 750=3.12% lat (msec) : 1000=4.49% cpu : usr=0.01%, sys=3.48%, ctx=16536, majf=0, minf=1035 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s (71.1MB/s-71.1MB/s), io=2048MiB (2147MB), run=30216-30216msec root@tmp-fio-bench:~# fio --name=big-file-multi-read --directory=/tmp/s3fs1 --rw=read --refill_buffers --bs=4M --size=2G --numjobs=4 big-file-multi-read: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 ... fio-3.28 Starting 4 processes big-file-multi-read: Laying out IO file (1 file / 2048MiB) big-file-multi-read: Laying out IO file (1 file / 2048MiB) big-file-multi-read: Laying out IO file (1 file / 2048MiB) big-file-multi-read: Laying out IO file (1 file / 2048MiB) Jobs: 2 (f=2): [_(1),R(1),_(1),R(1)][100.0%][r=168MiB/s][r=41 IOPS][eta 00m:00s] big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21020: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/131104msec) clat (msec): min=16, max=3379, avg=256.01, stdev=641.99 lat (msec): min=16, max=3379, avg=256.01, stdev=641.99 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 995], 95.00th=[ 2072], | 99.00th=[ 3004], 99.50th=[ 3071], 99.90th=[ 3373], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 8175, max=98304, per=71.82%, avg=45595.01, stdev=37884.57, samples=90 iops : min= 1, max= 24, avg=11.08, stdev= 9.23, samples=90 lat (msec) : 20=35.55%, 50=49.80%, 250=0.20%, 750=1.76%, 1000=2.73% lat (msec) : 2000=4.88%, >=2000=5.08% cpu : usr=0.01%, sys=1.00%, ctx=15877, majf=0, minf=1038 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21021: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.5MiB/s (16.3MB/s)(2048MiB/132144msec) clat (msec): min=15, max=3377, avg=258.05, stdev=643.26 lat (msec): min=15, max=3377, avg=258.06, stdev=643.26 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 23], 90.00th=[ 877], 95.00th=[ 1770], | 99.00th=[ 3071], 99.50th=[ 3138], 99.90th=[ 3373], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 8192, max=106283, per=71.53%, avg=45405.58, stdev=37512.44, samples=92 iops : min= 2, max= 25, avg=11.04, stdev= 9.13, samples=92 lat (msec) : 20=33.01%, 50=51.76%, 750=2.54%, 1000=2.73%, 2000=5.27% lat (msec) : >=2000=4.69% cpu : usr=0.01%, sys=0.99%, ctx=15910, majf=0, minf=1038 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21022: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.6MiB/s (16.4MB/s)(2048MiB/130901msec) clat (msec): min=16, max=3103, avg=255.62, stdev=620.89 lat (msec): min=16, max=3103, avg=255.62, stdev=620.89 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 23], 90.00th=[ 927], 95.00th=[ 2072], | 99.00th=[ 2769], 99.50th=[ 2937], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8175, max=98304, per=68.77%, avg=43654.97, stdev=37028.46, samples=94 iops : min= 1, max= 24, avg=10.61, stdev= 9.02, samples=94 lat (msec) : 20=34.77%, 50=49.80%, 750=1.76%, 1000=4.10%, 2000=4.49% lat (msec) : >=2000=5.08% cpu : usr=0.00%, sys=1.00%, ctx=15942, majf=2, minf=1036 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-read: (groupid=0, jobs=1): err= 0: pid=21023: Wed Aug 10 03:34:27 2022 read: IOPS=3, BW=15.6MiB/s (16.3MB/s)(2048MiB/131632msec) clat (msec): min=7, max=3088, avg=257.06, stdev=677.23 lat (msec): min=7, max=3088, avg=257.06, stdev=677.23 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 22], 80.00th=[ 22], 90.00th=[ 818], 95.00th=[ 2232], | 99.00th=[ 3037], 99.50th=[ 3071], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8192, max=98304, per=76.52%, avg=48573.40, stdev=38062.27, samples=86 iops : min= 2, max= 24, avg=11.81, stdev= 9.26, samples=86 lat (msec) : 10=0.20%, 20=33.98%, 50=52.15%, 750=2.15%, 1000=2.34% lat (msec) : 2000=3.32%, >=2000=5.86% cpu : usr=0.01%, sys=1.00%, ctx=15933, majf=0, minf=1038 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=512,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=62.0MiB/s (65.0MB/s), 15.5MiB/s-15.6MiB/s (16.3MB/s-16.4MB/s), io=8192MiB (8590MB), run=130901-132144msec root@tmp-fio-bench:~# root@tmp-fio-bench:~# root@tmp-fio-bench:~# root@tmp-fio-bench:~# fio --name=sequential-write --directory=/tmp/s3fs1 --rw=write --refill_buffers --bs=4M --size=2G --end_fsync=1 sequential-write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.28 Starting 1 process sequential-write: Laying out IO file (1 file / 2048MiB) Jobs: 1 (f=1): [F(1)][100.0%][eta 00m:00s] sequential-write: (groupid=0, jobs=1): err= 0: pid=22175: Wed Aug 10 03:42:52 2022 write: IOPS=8, BW=33.2MiB/s (34.8MB/s)(2048MiB/61621msec); 0 zone resets clat (msec): min=5, max=206, avg=54.90, stdev=30.54 lat (msec): min=5, max=206, avg=54.90, stdev=30.54 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 33], | 30.00th=[ 42], 40.00th=[ 48], 50.00th=[ 54], 60.00th=[ 70], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 86], 95.00th=[ 99], | 99.00th=[ 126], 99.50th=[ 128], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min=49152, max=540672, per=100.00%, avg=73435.96, stdev=66642.02, samples=57 iops : min= 12, max= 132, avg=17.89, stdev=16.28, samples=57 lat (msec) : 10=17.38%, 50=27.15%, 100=51.37%, 250=4.10% cpu : usr=0.76%, sys=1.69%, ctx=17547, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=2048MiB (2147MB), run=61621-61621msec root@tmp-fio-bench:~# fio --name=big-file-multi-write --directory=/tmp/s3fs1 --rw=write --refill_buffers --bs=4M --size=2G --numjobs=4 --end_fsync=1 big-file-multi-write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 ... fio-3.28 Starting 4 processes big-file-multi-write: Laying out IO file (1 file / 2048MiB) big-file-multi-write: Laying out IO file (1 file / 2048MiB) big-file-multi-write: Laying out IO file (1 file / 2048MiB) big-file-multi-write: Laying out IO file (1 file / 2048MiB) Jobs: 3 (f=3): [_(1),F(2),f(1)][100.0%][eta 00m:00s] big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22385: Wed Aug 10 03:47:37 2022 write: IOPS=3, BW=12.7MiB/s (13.3MB/s)(2048MiB/160853msec); 0 zone resets clat (msec): min=13, max=324, avg=248.09, stdev=54.24 lat (msec): min=13, max=324, avg=248.09, stdev=54.24 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 159], 10.00th=[ 232], 20.00th=[ 243], | 30.00th=[ 249], 40.00th=[ 253], 50.00th=[ 257], 60.00th=[ 264], | 70.00th=[ 271], 80.00th=[ 275], 90.00th=[ 284], 95.00th=[ 288], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8175, max=155648, per=51.37%, avg=16420.09, stdev=9463.81, samples=255 iops : min= 1, max= 38, avg= 3.96, stdev= 2.33, samples=255 lat (msec) : 20=1.76%, 50=2.93%, 100=0.20%, 250=28.71%, 500=66.41% cpu : usr=0.35%, sys=0.65%, ctx=21283, majf=2, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22386: Wed Aug 10 03:47:37 2022 write: IOPS=1, BW=7992KiB/s (8184kB/s)(2048MiB/262413msec); 0 zone resets clat (msec): min=9, max=33348, avg=313.15, stdev=1463.77 lat (msec): min=9, max=33348, avg=313.16, stdev=1463.77 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 169], 10.00th=[ 230], 20.00th=[ 243], | 30.00th=[ 249], 40.00th=[ 253], 50.00th=[ 259], 60.00th=[ 264], | 70.00th=[ 271], 80.00th=[ 275], 90.00th=[ 284], 95.00th=[ 292], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8192, max=73728, per=50.85%, avg=16255.35, stdev=5664.84, samples=256 iops : min= 2, max= 18, avg= 3.95, stdev= 1.39, samples=256 lat (msec) : 10=0.20%, 20=0.20%, 50=4.10%, 100=0.20%, 250=29.30% lat (msec) : 500=65.82%, >=2000=0.20% cpu : usr=0.18%, sys=0.44%, ctx=21156, majf=1, minf=18 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22387: Wed Aug 10 03:47:37 2022 write: IOPS=1, BW=8001KiB/s (8193kB/s)(2048MiB/262118msec); 0 zone resets clat (msec): min=14, max=33349, avg=313.31, stdev=1463.82 lat (msec): min=14, max=33349, avg=313.32, stdev=1463.82 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 153], 10.00th=[ 232], 20.00th=[ 245], | 30.00th=[ 249], 40.00th=[ 253], 50.00th=[ 257], 60.00th=[ 264], | 70.00th=[ 271], 80.00th=[ 275], 90.00th=[ 284], 95.00th=[ 292], | 99.00th=[ 305], 99.50th=[ 313], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8175, max=80170, per=50.92%, avg=16279.29, stdev=6017.65, samples=256 iops : min= 1, max= 19, avg= 3.92, stdev= 1.48, samples=256 lat (msec) : 20=0.59%, 50=3.91%, 100=0.20%, 250=30.27%, 500=64.84% lat (msec) : >=2000=0.20% cpu : usr=0.20%, sys=0.41%, ctx=20918, majf=0, minf=20 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 big-file-multi-write: (groupid=0, jobs=1): err= 0: pid=22388: Wed Aug 10 03:47:37 2022 write: IOPS=1, BW=8017KiB/s (8209kB/s)(2048MiB/261598msec); 0 zone resets clat (msec): min=9, max=33123, avg=313.15, stdev=1453.87 lat (msec): min=9, max=33123, avg=313.15, stdev=1453.87 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 163], 10.00th=[ 230], 20.00th=[ 243], | 30.00th=[ 249], 40.00th=[ 253], 50.00th=[ 259], 60.00th=[ 266], | 70.00th=[ 271], 80.00th=[ 275], 90.00th=[ 284], 95.00th=[ 292], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8175, max=155648, per=51.05%, avg=16318.77, stdev=9403.45, samples=256 iops : min= 1, max= 38, avg= 3.94, stdev= 2.31, samples=256 lat (msec) : 10=0.20%, 20=1.56%, 50=2.73%, 250=28.91%, 500=66.41% lat (msec) : >=2000=0.20% cpu : usr=0.19%, sys=0.43%, ctx=21457, majf=5, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,512,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=31.2MiB/s (32.7MB/s), 7992KiB/s-12.7MiB/s (8184kB/s-13.3MB/s), io=8192MiB (8590MB), run=160853-262413msec ```
kerem closed this issue 2026-03-04 01:50:39 +03:00
kerem commented 2026-03-04 01:50:39 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Aug 27, 2022):

Thanks for the report.
We also try to look for causes and bottlenecks.
Thanks in advance for your kindness.

<!-- gh-comment-id:1229168071 --> @ggtakec commented on GitHub (Aug 27, 2022): Thanks for the report. We also try to look for causes and bottlenecks. Thanks in advance for your kindness.
kerem commented 2026-03-04 01:50:39 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Sep 4, 2022):

The reason for this seems to be the poor performance of s3fs's internal processing when some number of jobs for fio.
Currently, I found a problem function and reviewing it.
This issue is likely the same as #2027.

<!-- gh-comment-id:1236288692 --> @ggtakec commented on GitHub (Sep 4, 2022): The reason for this seems to be the poor performance of s3fs's internal processing when some number of jobs for fio. Currently, I found a problem function and reviewing it. This issue is likely the same as #2027.
kerem commented 2026-03-04 01:50:39 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Sep 4, 2022):

@zhijian-pro
I have posted PR #2028 for this issue.
If you can build and test it, try https://github.com/ggtakec/s3fs-fuse/tree/fix_fdcache_page.
Thanks in advance for your help.

<!-- gh-comment-id:1236362985 --> @ggtakec commented on GitHub (Sep 4, 2022): @zhijian-pro I have posted PR #2028 for this issue. If you can build and test it, try https://github.com/ggtakec/s3fs-fuse/tree/fix_fdcache_page. Thanks in advance for your help.
kerem commented 2026-03-04 01:50:40 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Sep 25, 2022):

PR #2028 was merged.
If you can try it with master's code, please try it.
We will close this issue, but if the problem still exists, please reopen it.

<!-- gh-comment-id:1257162261 --> @ggtakec commented on GitHub (Sep 25, 2022): PR #2028 was merged. If you can try it with master's code, please try it. We will close this issue, but if the problem still exists, please reopen it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1013
No description provided.