[GH-ISSUE #89] checksum of file uploaded via s3fs to S3 is affected when using a file parallel upload #53

New issue

Closed

opened 2026-03-04 01:41:35 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:41:35 +03:00

Owner

Originally created by @ghost on GitHub (Nov 24, 2014).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/89

Hello,

I am using gridftp as a data transmission protocol to upload data to a bucket in AWS S3 mounted via S3FS.
The calculated checksum (adler32 or md5) of the file is uploaded to S3 differs from the one calculated at source if I use multiple gridftp streams to upload the file. On contrary if I use a single or no parallelism at all the calculated checksum at source and destination is the same.
Note that this issue is only observed when using the s3fs directory as a destination. Standard local file system directory the checksums are the same.

Is this issue known to the community? Is there any special setup to be done while mounting the bucket such this issue could be prevented?

I am using
s3fs 1.78
fuse-2.9.3-1.el6

Mount point
s3fs#s3-fuse-test /mnt/fuse fuse allow_other,max_stat_cache_size=0,use_cache=/s3cache_ssd,parallel_count=16,connect_timeout=20,retries=1,readwrite_timeout=60,fd_page_size=1677721600,multireq_max=200 0 0

All the best,
Carlos

Originally created by @ghost on GitHub (Nov 24, 2014). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/89 Hello, I am using gridftp as a data transmission protocol to upload data to a bucket in AWS S3 mounted via S3FS. The calculated checksum (adler32 or md5) of the file is uploaded to S3 differs from the one calculated at source if I use multiple gridftp streams to upload the file. On contrary if I use a single or no parallelism at all the calculated checksum at source and destination is the same. Note that this issue is only observed when using the s3fs directory as a destination. Standard local file system directory the checksums are the same. Is this issue known to the community? Is there any special setup to be done while mounting the bucket such this issue could be prevented? I am using s3fs 1.78 fuse-2.9.3-1.el6 Mount point s3fs#s3-fuse-test /mnt/fuse fuse allow_other,max_stat_cache_size=0,use_cache=/s3cache_ssd,parallel_count=16,connect_timeout=20,retries=1,readwrite_timeout=60,fd_page_size=1677721600,multireq_max=200 0 0 All the best, Carlos

kerem closed this issue

2026-03-04 01:41:35 +03:00

kerem commented

2026-03-04 01:41:36 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 13, 2015):

I'm sorry for replying late.
I did not have knew about this case issue(wrong checksum) ever.
Do you solve this problem?

Regards,

@ggtakec commented on GitHub (Jan 13, 2015): I'm sorry for replying late. I did not have knew about this case issue(wrong checksum) ever. Do you solve this problem? Regards,

kerem commented

2026-03-04 01:41:36 +03:00

Author

Owner

@ghost commented on GitHub (Jan 28, 2015):

I have tuned the S3fs parameters ( parallel_count, multireq_max, multipart_size among others) to mount the directories. This reduce the number of corrupted transfers.
All the best.
Carlos

On Jan 13, 2015, at 9:53 AM, Takeshi Nakatani notifications@github.com wrote:

I'm sorry for replying late.
I did not have knew about this case issue(wrong checksum) ever.
Do you solve this problem?

Regards,

—
Reply to this email directly or view it on GitHub https://github.com/s3fs-fuse/s3fs-fuse/issues/89#issuecomment-69755364.

@ghost commented on GitHub (Jan 28, 2015): I have tuned the S3fs parameters ( parallel_count, multireq_max, multipart_size among others) to mount the directories. This reduce the number of corrupted transfers. All the best. Carlos > On Jan 13, 2015, at 9:53 AM, Takeshi Nakatani notifications@github.com wrote: > > I'm sorry for replying late. > I did not have knew about this case issue(wrong checksum) ever. > Do you solve this problem? > > Regards, > > — > Reply to this email directly or view it on GitHub https://github.com/s3fs-fuse/s3fs-fuse/issues/89#issuecomment-69755364.

kerem referenced this issue

2026-03-04 01:53:24 +03:00

[PR #53] [MERGED] Support for SSE-C #39 #1328