[GH-ISSUE #1791] Make tests more independent of each other #920

New issue

Open

opened 2026-03-04 01:49:56 +03:00 by kerem · 3 comments

kerem commented

2026-03-04 01:49:56 +03:00

Owner

Originally created by @gaul on GitHub (Oct 26, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1791

#1784 proposes to use different file names to work around a stat cache issue in s3fs. But I think we can make tests more independent of each other by clearing the bucket between tests (rm -rf) and maybe restarting s3fs as well. This may have some issues with some Linux distributions scanning the new mount point for .Trash and other files (actually annoying behavior itself). It is not a good idea to restart S3Proxy every time since it takes 2.5 seconds of CPU time to start up.

Originally created by @gaul on GitHub (Oct 26, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1791 #1784 proposes to use different file names to work around a stat cache issue in s3fs. But I think we can make tests more independent of each other by clearing the bucket between tests (`rm -rf`) and maybe restarting s3fs as well. This may have some issues with some Linux distributions scanning the new mount point for `.Trash` and other files (actually annoying behavior itself). It is not a good idea to restart S3Proxy every time since it takes 2.5 seconds of CPU time to start up.

kerem commented

2026-03-04 01:49:57 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 26, 2023):

@gaul
I would like to make the tests independent of each other in the next code fix(not soon).
For that reason, if only one s3proxy process is run, could s3proxy operate sufficiently even if multiple s3fs processes access it at the same time?
It would expect the buckets, files, etc. to be non-overlapping, but I would expect access to them to occur at the same time.

@ggtakec commented on GitHub (Mar 26, 2023): @gaul I would like to make the tests independent of each other in the next code fix(not soon). For that reason, if only one s3proxy process is run, could s3proxy operate sufficiently even if multiple s3fs processes access it at the same time? It would expect the buckets, files, etc. to be non-overlapping, but I would expect access to them to occur at the same time.

kerem commented

2026-03-04 01:49:57 +03:00

Author

Owner

@gaul commented on GitHub (Mar 28, 2023):

S3Proxy should support many concurrent s3fs processes if you use different buckets. One limitation is that s3fs tests currently use the transient (memory) storage backend and not the filesystem (on-disk). The former may use more memory if we have many concurrent tests.

I think using one S3Proxy process with many buckets is the right approach. One thing to consider is that the tests should still allow testing against real AWS and there are some limitations on creating buckets there.

@gaul commented on GitHub (Mar 28, 2023): S3Proxy should support many concurrent s3fs processes if you use different buckets. One limitation is that s3fs tests currently use the transient (memory) storage backend and not the filesystem (on-disk). The former may use more memory if we have many concurrent tests. I think using one S3Proxy process with many buckets is the right approach. One thing to consider is that the tests should still allow testing against real AWS and there are some limitations on creating buckets there.

kerem commented

2026-03-04 01:49:57 +03:00

Author

Owner

@ggtakec commented on GitHub (Apr 8, 2023):

@gaul Thanks.
I think that being able to mount a single bucket in multiple s3fs processes at the same time might also help split tests.
Since the current test is sequentially starting and stopping s3fs multiple times, I think that it would be nice to be able to run this in parallel.
Parallel execution of tests will continue to be considered.

@ggtakec commented on GitHub (Apr 8, 2023): @gaul Thanks. I think that being able to mount a single bucket in multiple s3fs processes at the same time might also help split tests. Since the current test is sequentially starting and stopping s3fs multiple times, I think that it would be nice to be able to run this in parallel. Parallel execution of tests will continue to be considered.