[GH-ISSUE #11] 0 byte files

kerem commented

2026-03-04 01:41:04 +03:00

Owner

Originally created by @jonbev on GitHub (Jan 27, 2014).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/11

We have a problem where we are getting 0 byte files written back to s3. I can seem to find a way to replicate as it seems to be quite random. We cant tell if it is a s3fs issue or something on our side.

I was wondering if there is a way of disallowing 0 byte files from being written?

Thanks.

Originally created by @jonbev on GitHub (Jan 27, 2014). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/11 We have a problem where we are getting 0 byte files written back to s3. I can seem to find a way to replicate as it seems to be quite random. We cant tell if it is a s3fs issue or something on our side. I was wondering if there is a way of disallowing 0 byte files from being written? Thanks.

kerem closed this issue

2026-03-04 01:41:05 +03:00

kerem commented

2026-03-04 01:41:05 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 27, 2014):

The 0 byte file is made by something error occurred dialing uploading and changing file attributes etc.
(These action are sent from fuse to s3fs.)

I think it is difficult to remove 0 byte file by s3fs after something error is occurred.
But maybe we think that it is best way.

If you can, please run s3fs with -d option, it makes s3fs write messages(/var/log/) for debug.
Probably this messages helps us to know why this error occurred.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 27, 2014): The 0 byte file is made by something error occurred dialing uploading and changing file attributes etc. (These action are sent from fuse to s3fs.) I think it is difficult to remove 0 byte file by s3fs after something error is occurred. But maybe we think that it is best way. If you can, please run s3fs with -d option, it makes s3fs write messages(/var/log/) for debug. Probably this messages helps us to know why this error occurred. Thanks in advance for your help.

kerem commented

2026-03-04 01:41:05 +03:00

Author

Owner

@geekpete commented on GitHub (Sep 5, 2014):

I see the same behaviour, but it seems to be creating these zero byte files at the start of a transfer and once the transfer is completed, the file size becomes correct.

This is really annoying, as the usual behaviour for an s3 upload tool is to only reveal the file once the full multipart upload is entirely completed.

Is there any way to disable this "poor man's locking" mechanism and make it so that the empty file is not written at the start of the transfer?

@geekpete commented on GitHub (Sep 5, 2014): I see the same behaviour, but it seems to be creating these zero byte files at the start of a transfer and once the transfer is completed, the file size becomes correct. This is really annoying, as the usual behaviour for an s3 upload tool is to only reveal the file once the full multipart upload is entirely completed. Is there any way to disable this "poor man's locking" mechanism and make it so that the empty file is not written at the start of the transfer?

kerem commented

2026-03-04 01:41:06 +03:00

Author

Owner

@geekpete commented on GitHub (Sep 9, 2014):

Is the creation of the zero byte file absolutely necessary for the operation of s3fs-fuse or can the zero byte file creation be skipped/optional?

Is there any way to just start the multipart upload and close it off when it completes, only then revealing the file to s3 clients?

From https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/curl.cpp:

int S3fsCurl::PutRequest(const char* tpath, headers_t& meta, int fd)
{
struct stat st;
FILE* file = NULL;
int fd2;
FPRNNN("[tpath=%s]", SAFESTRPTR(tpath));
if(!tpath){
return -1;
}
if(-1 != fd){
// duplicate fd
if(-1 == (fd2 = dup(fd)) || -1 == fstat(fd2, &st) || 0 != lseek(fd2, 0, SEEK_SET) || NULL == (file = fdopen(fd2, "rb"))){
DPRN("Could not duplicate file discriptor(errno=%d)", errno);
return -errno;
}
b_infile = file;
}else{
// This case is creating zero byte obejct.(calling by create_file_object())
DPRNNN("create zero byte file object.");
}
if(!CreateCurlHandle(true)){
if(file){
fclose(file);
}
return -1;
}

@geekpete commented on GitHub (Sep 9, 2014): Is the creation of the zero byte file absolutely necessary for the operation of s3fs-fuse or can the zero byte file creation be skipped/optional? Is there any way to just start the multipart upload and close it off when it completes, only then revealing the file to s3 clients? From https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/curl.cpp: ``` c++ int S3fsCurl::PutRequest(const char* tpath, headers_t& meta, int fd) { struct stat st; FILE* file = NULL; int fd2; FPRNNN("[tpath=%s]", SAFESTRPTR(tpath)); if(!tpath){ return -1; } if(-1 != fd){ // duplicate fd if(-1 == (fd2 = dup(fd)) || -1 == fstat(fd2, &st) || 0 != lseek(fd2, 0, SEEK_SET) || NULL == (file = fdopen(fd2, "rb"))){ DPRN("Could not duplicate file discriptor(errno=%d)", errno); return -errno; } b_infile = file; }else{ // This case is creating zero byte obejct.(calling by create_file_object()) DPRNNN("create zero byte file object."); } if(!CreateCurlHandle(true)){ if(file){ fclose(file); } return -1; } ```

kerem commented

2026-03-04 01:41:06 +03:00

Author

Owner

@ggtakec commented on GitHub (Sep 9, 2014):

On FUSE, s3fs works following:

s3fs_create()
create 0 byte file
and returns file descriptor to FUSE.
s3fs_getattr()
returns file status to FUSE.
s3fs_write
write a part of data to file.
repeat this calling.
s3fs_flush()
flush file discriptor
s3fs_release()
close file descriptor

At (1), s3fs makes temporary file, and open it, returns this local file descriptor to FUSE.
Then at this time, s3fs uploads 0 byte file to S3.(*)
At (3), s3fs writes data to local file descriptor which is returned at (1) to FUSE.
At (4), s3fs uploads(multipart upload) local file data to S3.

All these operation are given an order to start by FUSE, then s3fs do working.
I think it is not a wrong thing that s3fs makes a file of 0 bytes, because a file was made in the same way as other file system.
In the case of normal file system, it does not take time to file closing from file creating.
However, in s3fs, it takes time. So I think that this becomes the problem.

Maybe I can change a cord for not uploading 0 byte file on S3.
However, It probabry become the problem when other systems which mounts same bucket accesses that object.

So that, I think that this change should not do.
And I should know detail of your problem, please let me know detail, it helps us for solving this issue.

Thanks in advance for your help.

@ggtakec commented on GitHub (Sep 9, 2014): On FUSE, s3fs works following: 1) s3fs_create() create 0 byte file and returns file descriptor to FUSE. 2) s3fs_getattr() returns file status to FUSE. 3) s3fs_write write a part of data to file. repeat this calling. 4) s3fs_flush() flush file discriptor 5) s3fs_release() close file descriptor At (1), s3fs makes temporary file, and open it, returns this local file descriptor to FUSE. Then at this time, s3fs uploads 0 byte file to S3.(*) At (3), s3fs writes data to local file descriptor which is returned at (1) to FUSE. At (4), s3fs uploads(multipart upload) local file data to S3. All these operation are given an order to start by FUSE, then s3fs do working. I think it is not a wrong thing that s3fs makes a file of 0 bytes, because a file was made in the same way as other file system. In the case of normal file system, it does not take time to file closing from file creating. However, in s3fs, it takes time. So I think that this becomes the problem. Maybe I can change a cord for not uploading 0 byte file on S3. However, It probabry become the problem when other systems which mounts same bucket accesses that object. So that, I think that this change should not do. And I should know detail of your problem, please let me know detail, it helps us for solving this issue. Thanks in advance for your help.

kerem commented

2026-03-04 01:41:06 +03:00

Author

Owner

@geekpete commented on GitHub (Sep 10, 2014):

If this is a limitation (feature?) of fuse filesystems then I'll close this ticket.

My use case is a bit odd, I need an SFTP to S3 service, but for reasons I cannot go into, I am stuck building the solution this way.

So I have proftpd dropping files right onto an s3fs-fuse mount to get files up to S3.

I then have another process doing things to new files that appear in the s3 bucket from the other end.

My problem is that if I have a temporary file appear while the transfer is still in progress, then the process at the other end will start trying to process it while it is still zero bytes and fail. Also trying to avoid altering code of this process as it is legacy.

BUT!

With proftpd, I am now using HiddenStores and DeleteAbortedStore to drop the zero byte file as a temporary filename, once the sftp upload completes successfully the file is renamed.
If the upload fails, the temp file is deleted.

This is working well for me currently.

@geekpete commented on GitHub (Sep 10, 2014): If this is a limitation (feature?) of fuse filesystems then I'll close this ticket. My use case is a bit odd, I need an SFTP to S3 service, but for reasons I cannot go into, I am stuck building the solution this way. So I have proftpd dropping files right onto an s3fs-fuse mount to get files up to S3. I then have another process doing things to new files that appear in the s3 bucket from the other end. My problem is that if I have a temporary file appear while the transfer is still in progress, then the process at the other end will start trying to process it while it is still zero bytes and fail. Also trying to avoid altering code of this process as it is legacy. BUT! With proftpd, I am now using HiddenStores and DeleteAbortedStore to drop the zero byte file as a temporary filename, once the sftp upload completes successfully the file is renamed. If the upload fails, the temp file is deleted. This is working well for me currently.

kerem commented

2026-03-04 01:41:06 +03:00

Author

Owner

@ggtakec commented on GitHub (Sep 15, 2014):

Hi, geekpete

Making 0 byte file that is limitation on fuse.(I think this is normally when local disk.)

About 0 byte file uploaded and completely file after that, so these object have original ETag, does proftpd can know it(ETag)?
If it can, ETag header is used for checking objects.

Thanks.

@ggtakec commented on GitHub (Sep 15, 2014): Hi, geekpete Making 0 byte file that is limitation on fuse.(I think this is normally when local disk.) About 0 byte file uploaded and completely file after that, so these object have original ETag, does proftpd can know it(ETag)? If it can, ETag header is used for checking objects. Thanks.

kerem commented

2026-03-04 01:41:06 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 17, 2016):

I'm sorry about this Issue had been left at a long period of time.

s3fs makes 0 byte file before uploading new file(object), normally the object size is changed after uploading the object.
But that the file size is still 0 bytes means that it has failed to upload.

Latest codes are fixed about multipart uploading etc, please test latest codes.
Anf if you have a problem yet, please post new issue about latest codes.

I'm closing this issue now, thanks in advance for your help.

@ggtakec commented on GitHub (Jan 17, 2016): I'm sorry about this Issue had been left at a long period of time. s3fs makes 0 byte file before uploading new file(object), normally the object size is changed after uploading the object. But that the file size is still 0 bytes means that it has failed to upload. Latest codes are fixed about multipart uploading etc, please test latest codes. Anf if you have a problem yet, please post new issue about latest codes. I'm closing this issue now, thanks in advance for your help.

kerem commented

2026-03-04 01:41:06 +03:00

Author

Owner

@KrzysztofMadejski commented on GitHub (Dec 28, 2016):

Just noting as I have problem related to that: My provider of S3 don't like 0-byte files (soft refuses to upload it) and that stops me from being able to write to S3. I'll come back if I find any solution.

Dec 28 13:57:19 dp s3fs[23671]: curl.cpp:RequestPerform(1946): Body Text: <?xml version="1.0" ?>#012<Error>#012    <Code>EntityTooSmall</Code>#012    <Message>Your proposed upload is smaller than the minimum allowed object size.</Message>#012    <RequestId>78AA0895</RequestId>#012    <BucketName>resources</BucketName>#012    <KeyName>8f-d0cc-45bc-8a1f-6e1204793bb4</KeyName>#012    <HostId>f2:0</HostId>#012    <Date>Wed, 28 Dec 2016 12:57:18 GMT</Date>#012</Error>

@KrzysztofMadejski commented on GitHub (Dec 28, 2016): Just noting as I have problem related to that: My provider of S3 don't like 0-byte files (soft refuses to upload it) and that stops me from being able to write to S3. I'll come back if I find any solution. ``` Dec 28 13:57:19 dp s3fs[23671]: curl.cpp:RequestPerform(1946): Body Text: <?xml version="1.0" ?>#012<Error>#012 <Code>EntityTooSmall</Code>#012 <Message>Your proposed upload is smaller than the minimum allowed object size.</Message>#012 <RequestId>78AA0895</RequestId>#012 <BucketName>resources</BucketName>#012 <KeyName>8f-d0cc-45bc-8a1f-6e1204793bb4</KeyName>#012 <HostId>f2:0</HostId>#012 <Date>Wed, 28 Dec 2016 12:57:18 GMT</Date>#012</Error> ```

kerem commented

2026-03-04 01:41:06 +03:00

Author

Owner

@alperen66 commented on GitHub (Feb 18, 2021):

Hi Im really new to github just made a account to write here
Recently fuse is not working ı cant export anything it says 0kb everytime
ı also try to reinstall but never worked, even try to download it from other websites.
and today ı found the adobe fuse cc version and tried to export again but it didnt worked again
please ı really need to use this software if you know the solve of this problem what can ı do ?

thanks

@alperen66 commented on GitHub (Feb 18, 2021): Hi Im really new to github just made a account to write here Recently fuse is not working ı cant export anything it says 0kb everytime ı also try to reinstall but never worked, even try to download it from other websites. and today ı found the adobe fuse cc version and tried to export again but it didnt worked again please ı really need to use this software if you know the solve of this problem what can ı do ? thanks

kerem referenced this issue

2026-03-04 01:47:21 +03:00

[GH-ISSUE #1191] s3fs crash during cache read #630

kerem referenced this issue

2026-03-04 02:01:18 +03:00

[PR #697] [MERGED] Fixing race condition in FdEntity::GetStats #1610

Rows
Columns

[GH-ISSUE #11] 0 byte files #5