[GH-ISSUE #1543] Run integration tests with larger files #811

New issue

Open

opened 2026-03-04 01:48:59 +03:00 by kerem · 4 comments

kerem commented

2026-03-04 01:48:59 +03:00

Owner

Originally created by @gaul on GitHub (Feb 2, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1543

s3fs sets BIG_FILE_LENGTH to 25 MB which is large enough to trigger multipart upload but lower than other limits like the single file upload size. This limit also makes sense when using S3Proxy's transient backend which store objects in-memory. When apache/jclouds#96 merges, s3fs can instead use the filesystem backend which stores objects on-disk. This will allow the integration tests to use larger objects, e.g., > 5 GB single-part objects. Related to #1542.

Originally created by @gaul on GitHub (Feb 2, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1543 s3fs sets `BIG_FILE_LENGTH` to 25 MB which is large enough to trigger multipart upload but lower than other limits like the single file upload size. This limit also makes sense when using S3Proxy's transient backend which store objects in-memory. When apache/jclouds#96 merges, s3fs can instead use the filesystem backend which stores objects on-disk. This will allow the integration tests to use larger objects, e.g., > 5 GB single-part objects. Related to #1542.

kerem commented

2026-03-04 01:49:00 +03:00

Author

Owner

@gaul commented on GitHub (Feb 4, 2021):

With larger files reading from /dev/urandom becomes a bottleneck. We may want to create our own helper which makes pseudo-random data faster.

@gaul commented on GitHub (Feb 4, 2021): With larger files reading from `/dev/urandom` becomes a bottleneck. We may want to create our own helper which makes pseudo-random data faster.

kerem commented

2026-03-04 01:49:00 +03:00

Author

Owner

@gaul commented on GitHub (Feb 4, 2021):

To use the filesystem blobstore to test larger files we will need a new release of S3Proxy to include the jclouds fix.

@gaul commented on GitHub (Feb 4, 2021): To use the filesystem blobstore to test larger files we will need a new release of S3Proxy to include the jclouds fix.

kerem commented

2026-03-04 01:49:00 +03:00

Author

Owner

@gaul commented on GitHub (Feb 7, 2021):

Testing with larger files is proving to be a rich source of bugs. Some tests require files as large as 11 GB to trigger various conditions which takes up to 15 minutes to run on my desktop. We could add some flags to s3fs and S3Proxy to use smaller thresholds to trigger conditions like copy part range more easily.

@gaul commented on GitHub (Feb 7, 2021): Testing with larger files is proving to be a rich source of bugs. Some tests require files as large as 11 GB to trigger various conditions which takes up to 15 minutes to run on my desktop. We could add some flags to s3fs and S3Proxy to use smaller thresholds to trigger conditions like copy part range more easily.

kerem commented

2026-03-04 01:49:00 +03:00

Author

Owner

@gaul commented on GitHub (May 30, 2021):

Related to #1665, currently blocked on macOS Java xattr compatibility.

@gaul commented on GitHub (May 30, 2021): Related to #1665, currently blocked on macOS Java xattr compatibility.