starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #829] S3FS Fio benchmark with cache vs no cache #481

New issue
Open
opened 2026-03-04 01:46:00 +03:00 by kerem · 3 comments
kerem commented 2026-03-04 01:46:00 +03:00
Owner
Copy link

Originally created by @CMCDragonkai on GitHub (Sep 25, 2018).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/829

I reported strange performance issues on fio: https://github.com/axboe/fio/issues/692 when using cache vs not using cache. The main issue is that although with cache was certainly faster, it reported worse IOPS and bandwidth rates.

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.84(commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse)

fusermount version: 2.9.7

System information (uname -r)

4.17.6

Distro (cat /etc/issue)

NixOS

Originally created by @CMCDragonkai on GitHub (Sep 25, 2018). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/829 I reported strange performance issues on fio: https://github.com/axboe/fio/issues/692 when using cache vs not using cache. The main issue is that although with cache was certainly faster, it reported worse IOPS and bandwidth rates. #### Version of s3fs being used (s3fs --version) ``` Amazon Simple Storage Service File System V1.84(commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Version of fuse being used (pkg-config --modversion fuse) ``` fusermount version: 2.9.7 ``` #### System information (uname -r) ``` 4.17.6 ``` #### Distro (cat /etc/issue) NixOS
kerem commented 2026-03-04 01:46:01 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Oct 1, 2018):

Hi, @CMCDragonkai

I think that your result in https://github.com/axboe/fio/issues/692 had a error.
It seems fio process gets a EIO error from s3fs.

readfiles: (groupid=0, jobs=1): err= 5 (file:filesetup.c:703, func=open(readfiles.0.984), error=Input/output error): pid=11542: Mon Sep 24 17:35:34 201

I think that you shoud check following at first:

  • "use_cache" option value is set on the local file system.
  • The target file(directory) is allowed read/write parmission

Below is the result I tested. It seems to work especially without problems.

fio

version

$ fio --version
  fio-2.1.5

fio configuration file

$ cat config.fio
  [readfiles]
  nrfiles=100
  filesize=2MiB
  bs=4k
  rw=read
  directory=/mnt/s3/fio

Using cache directory

Command(sample)

$ src/s3fs <mybucket> /mnt/s3 -o allow_other,nonempty,enable_noobj_cache,enable_content_md5,multireq_max=50,use_cache=/tmp

Result

readfiles: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.1.5
Starting 1 process
readfiles: Laying out IO file(s) (6 file(s) / 11MB)
Jobs: 1 (f=100): [R] [80.0% done] [80672KB/0KB/0KB /s] [20.2K/0/0 iops] [eta 00m:01s]
readfiles: (groupid=0, jobs=1): err= 0: pid=27650: Mon Oct  1 12:17:37 2018
  read : io=195200KB, bw=56432KB/s, iops=14108, runt=  3459msec
    clat (usec): min=5, max=146708, avg=28.69, stdev=916.65
     lat (usec): min=7, max=146710, avg=30.77, stdev=916.67
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    6], 80.00th=[    6], 90.00th=[    7], 95.00th=[    7],
     | 99.00th=[  314], 99.50th=[  596], 99.90th=[ 3216], 99.95th=[ 6368],
     | 99.99th=[20864]
    bw (KB  /s): min=  235, max=186389, per=100.00%, avg=61214.83, stdev=86751.12
    lat (usec) : 10=97.14%, 20=0.76%, 50=0.24%, 100=0.15%, 250=0.52%
    lat (usec) : 500=0.64%, 750=0.24%, 1000=0.07%
    lat (msec) : 2=0.09%, 4=0.05%, 10=0.06%, 20=0.01%, 50=0.02%
    lat (msec) : 250=0.01%
  cpu          : usr=3.24%, sys=17.00%, ctx=1232, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=48800/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=195200KB, aggrb=56432KB/s, minb=56432KB/s, maxb=56432KB/s, mint=3459msec, maxt=3459msec

No cache directory

Command(sample)

src/s3fs <mybucket> /mnt/s3 -o allow_other,nonempty,enable_noobj_cache,enable_content_md5,multireq_max=50

Result

readfiles: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.1.5
Starting 1 process
Jobs: 1 (f=100): [R] [82.5% done] [23804KB/0KB/0KB /s] [5951/0/0 iops] [eta 00m:07s]
readfiles: (groupid=0, jobs=1): err= 0: pid=27676: Mon Oct  1 12:19:22 2018
  read : io=195200KB, bw=5986.7KB/s, iops=1496, runt= 32606msec
    clat (usec): min=5, max=372507, avg=521.09, stdev=8659.92
     lat (usec): min=7, max=372511, avg=523.34, stdev=8659.98
    clat percentiles (usec):
     |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    6], 80.00th=[    6], 90.00th=[    7], 95.00th=[    8],
     | 99.00th=[  564], 99.50th=[10048], 99.90th=[171008], 99.95th=[177152],
     | 99.99th=[197632]
    bw (KB  /s): min=    5, max=54578, per=90.45%, avg=5414.20, stdev=13632.30
    lat (usec) : 10=96.56%, 20=0.75%, 50=0.56%, 100=0.36%, 250=0.38%
    lat (usec) : 500=0.35%, 750=0.12%, 1000=0.07%
    lat (msec) : 2=0.14%, 4=0.06%, 10=0.14%, 20=0.12%, 50=0.07%
    lat (msec) : 100=0.07%, 250=0.24%, 500=0.01%
  cpu          : usr=0.32%, sys=1.93%, ctx=1782, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=48800/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=195200KB, aggrb=5986KB/s, minb=5986KB/s, maxb=5986KB/s, mint=32606msec, maxt=32606msec

Thanks in advance for your assistance.

<!-- gh-comment-id:425891516 --> @ggtakec commented on GitHub (Oct 1, 2018): Hi, @CMCDragonkai I think that your result in https://github.com/axboe/fio/issues/692 had a error. It seems fio process gets a EIO error from s3fs. ``` readfiles: (groupid=0, jobs=1): err= 5 (file:filesetup.c:703, func=open(readfiles.0.984), error=Input/output error): pid=11542: Mon Sep 24 17:35:34 201 ``` I think that you shoud check following at first: - "use_cache" option value is set on the local file system. - The target file(directory) is allowed read/write parmission Below is the result I tested. It seems to work especially without problems. ## fio ### version ``` $ fio --version fio-2.1.5 ``` ### fio configuration file ``` $ cat config.fio [readfiles] nrfiles=100 filesize=2MiB bs=4k rw=read directory=/mnt/s3/fio ``` ## Using cache directory ### Command(sample) ``` $ src/s3fs <mybucket> /mnt/s3 -o allow_other,nonempty,enable_noobj_cache,enable_content_md5,multireq_max=50,use_cache=/tmp ``` ### Result ``` readfiles: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.5 Starting 1 process readfiles: Laying out IO file(s) (6 file(s) / 11MB) Jobs: 1 (f=100): [R] [80.0% done] [80672KB/0KB/0KB /s] [20.2K/0/0 iops] [eta 00m:01s] readfiles: (groupid=0, jobs=1): err= 0: pid=27650: Mon Oct 1 12:17:37 2018 read : io=195200KB, bw=56432KB/s, iops=14108, runt= 3459msec clat (usec): min=5, max=146708, avg=28.69, stdev=916.65 lat (usec): min=7, max=146710, avg=30.77, stdev=916.67 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 7], | 99.00th=[ 314], 99.50th=[ 596], 99.90th=[ 3216], 99.95th=[ 6368], | 99.99th=[20864] bw (KB /s): min= 235, max=186389, per=100.00%, avg=61214.83, stdev=86751.12 lat (usec) : 10=97.14%, 20=0.76%, 50=0.24%, 100=0.15%, 250=0.52% lat (usec) : 500=0.64%, 750=0.24%, 1000=0.07% lat (msec) : 2=0.09%, 4=0.05%, 10=0.06%, 20=0.01%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=3.24%, sys=17.00%, ctx=1232, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=48800/w=0/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: io=195200KB, aggrb=56432KB/s, minb=56432KB/s, maxb=56432KB/s, mint=3459msec, maxt=3459msec ``` ## No cache directory ### Command(sample) ``` src/s3fs <mybucket> /mnt/s3 -o allow_other,nonempty,enable_noobj_cache,enable_content_md5,multireq_max=50 ``` ### Result ``` readfiles: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.5 Starting 1 process Jobs: 1 (f=100): [R] [82.5% done] [23804KB/0KB/0KB /s] [5951/0/0 iops] [eta 00m:07s] readfiles: (groupid=0, jobs=1): err= 0: pid=27676: Mon Oct 1 12:19:22 2018 read : io=195200KB, bw=5986.7KB/s, iops=1496, runt= 32606msec clat (usec): min=5, max=372507, avg=521.09, stdev=8659.92 lat (usec): min=7, max=372511, avg=523.34, stdev=8659.98 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 6], 80.00th=[ 6], 90.00th=[ 7], 95.00th=[ 8], | 99.00th=[ 564], 99.50th=[10048], 99.90th=[171008], 99.95th=[177152], | 99.99th=[197632] bw (KB /s): min= 5, max=54578, per=90.45%, avg=5414.20, stdev=13632.30 lat (usec) : 10=96.56%, 20=0.75%, 50=0.56%, 100=0.36%, 250=0.38% lat (usec) : 500=0.35%, 750=0.12%, 1000=0.07% lat (msec) : 2=0.14%, 4=0.06%, 10=0.14%, 20=0.12%, 50=0.07% lat (msec) : 100=0.07%, 250=0.24%, 500=0.01% cpu : usr=0.32%, sys=1.93%, ctx=1782, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=48800/w=0/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: io=195200KB, aggrb=5986KB/s, minb=5986KB/s, maxb=5986KB/s, mint=32606msec, maxt=32606msec ``` Thanks in advance for your assistance.
kerem commented 2026-03-04 01:46:01 +03:00
Author
Owner
Copy link

@CMCDragonkai commented on GitHub (Oct 2, 2018):

My cache directory was set to the local filesystem backed by ZFS, my umask was 027 for the creation of the cache directory.

No idea why you're not getting the error, but I suppose I can try dumping all the logs on s3fs to you for the test, or even stracing the s3fs program. I don't know if I have the time to focus on this.

<!-- gh-comment-id:426102400 --> @CMCDragonkai commented on GitHub (Oct 2, 2018): My cache directory was set to the local filesystem backed by ZFS, my umask was 027 for the creation of the cache directory. No idea why you're not getting the error, but I suppose I can try dumping all the logs on s3fs to you for the test, or even stracing the s3fs program. I don't know if I have the time to focus on this.
kerem commented 2026-03-04 01:46:01 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Oct 8, 2018):

There are several reasons why s3fs returned an EIO error.
(For example, it can not communicate.)
If the s3fs error is reproduced, you can obtain the log using the s3fs command parameter "-o dbglevel=dbg".
If you get something s3fs error, we may find a solution.
Thanks,

<!-- gh-comment-id:427883887 --> @ggtakec commented on GitHub (Oct 8, 2018): There are several reasons why s3fs returned an EIO error. (For example, it can not communicate.) If the s3fs error is reproduced, you can obtain the log using the s3fs command parameter "-o dbglevel=dbg". If you get something s3fs error, we may find a solution. Thanks,
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#481
No description provided.