mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 21:35:58 +03:00
[GH-ISSUE #1217] Everything Uploaded To S3 Is Content-Type application/octet-stream #647
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#647
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @mydigia on GitHub (Dec 18, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1217
My environment info:
$ s3fs --version
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)
$ pkg-config --modversion fuse
2.9.2$ rpm -qi fuse, dpkg -s fuse
Kernel information (uname -r)
GNU/Linux Distribution, if applicable (cat /etc/os-release)
s3fs command line used, if applicable
Details about issue
When I mount my s3 bucket and then use rsync to sync my data to the S3 bucket, then every single object on the bucket has the following metadata:
It looks like "Content-Type | application/octet-stream" stop my S3 static website hosting from working, because it downloads the files instead of serving them.
If I upload a single object using Cyberduck to S3 the metadata is correctly picked and served on the public URL of the bucket.
@gaul commented on GitHub (Feb 2, 2020):
@mydigia I wrote a test for this behavior and s3fs correctly detects text/plain and image/jpeg. Could you be more specific about which file types are not detected?
@NickMillerUK commented on GitHub (Mar 12, 2020):
I am experiencing exactly the same issue. Here are the details of my installation:
$ s3fs --version
$ rpm -qi fuse
$ uname -r
3.10.0-1062.12.1.el7.x86_64$ cat /etc/os-release
`NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"`
I am mounting my S3 bucket as follows:
s3fs#s3.bucket.name /path/to/mount/to _netdev,allow_other,mp_umask=022,use_path_request_style,url=https://s3.us-east-1.amazonaws.com 0 0All files uploaded have ContentType set to: application/octet-stream
I have tested with images (.jpg), html files (.html) and text files (.txt).
What additional information do you need to be able to debug this issue?
Thanks
Nick
@tryggve commented on GitHub (Mar 17, 2020):
We had the same issue that all files uploaded got application/octet-stream. It turns out that the ec2-instance running our ftp-server did not have mimetypes configured on the OS level.
https://wiki.debian.org/MIME/etc/mime.types
Once Linux knew how to recognize different data types the ContentType header in s3 looked as we wanted.
@NickMillerUK commented on GitHub (Mar 18, 2020):
@tryggve thank you for this. This was exactly the issue.
This was a new install of Centos 7, and there was no /etc/mime.types file.
The fix was simple - obtain a valid mime.types file (in my case I took one from an existing Centos 7 install) and copy it into place in /etc. Make sure that the file is owned by root (chown root:root /etc/mime.types) and that all users can read it (chmod a+r /etc/mime.types).
Best to reboot the system to ensure that any software that relies on this file notices it is present.
Thanks again for the pointer.
@juliogonzalez commented on GitHub (Mar 18, 2020):
Is this happening with s3fs-fuse compiled from source, or also with the s3fuse-packages from EPEL?
Looks to me that maybe you are missing the package that provides it.
This is from a Docker container:
If it is required for s3fs-fuse to work, maybe we should make sure the file exists, and require mailcap for the RPMs/DEBs, etc.
Could be that for most user this works because the mailcap is installed by default on standard setups, but maybe on minimal installations this is not the case and it should be required.
@juliogonzalez commented on GitHub (Mar 18, 2020):
Well, in fact we already have mailcap at the CentOS instructions:
https://github.com/s3fs-fuse/s3fs-fuse/wiki/Installation-Notes#fedora--centos--rhel
But no mention to the package or
/etc/mime.typeson the generic instructions.And it seems I removed it from the spec at 1.84:
https://github.com/juliogonzalez/s3fs-fuse-rpm/blob/master/SPECS/s3fs-fuse.spec#L89
Because it is NOT a build requirement, but a runtime requirement.
So three things need a fix:
a) The
COMPILE.mdfile to mention this somehow (so it's generic enough)b) The SPEC for EPEL/Fedora (link above, will take care of it)
c) The SPEC for SLE/openSUSE (https://build.opensuse.org/package/view_file/filesystems/s3fs/s3fs.spec?expand=1 can handle it as well).
@juliogonzalez commented on GitHub (Mar 18, 2020):
For my own reference,
cis NOT needed. On SLE/openSUSE/etc/mime.typesis part ofaaa_basethat is always installed even of the most minimal setups.On Debian/Ubuntu is part of
mime-supportwhich is not installed on the Docker images, and I guess maybe it's absent on minimal installations, so yet another reason to mention it atCOMPILE.md.However the Debian package is right, as it requires
mime-support.@juliogonzalez commented on GitHub (Mar 18, 2020):
Bodhi requests with the fix for:
Fedora30: https://bodhi.fedoraproject.org/updates/FEDORA-2020-951f18a723
Fedora31: https://bodhi.fedoraproject.org/updates/FEDORA-2020-92af351313
Fedora32: https://bodhi.fedoraproject.org/updates/FEDORA-2020-3aaa1f559c
EPEL7: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-e6bada7ce3
EPEL8: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-932011492d
Rawhide: No bodhi enablement, it will be released automatically from the koji build (https://koji.fedoraproject.org/koji/taskinfo?taskID=42608070 https://bodhi.fedoraproject.org/updates/FEDORA-2020-baff878a5f)
@NickMillerUK commented on GitHub (Mar 19, 2020):
@juliogonzalez thank you for following up on this.
I can confirm the package was installed from EPEL, not source. Also, mailcap is indeed not installed, and this was a minimal install of Centos.
I does look like this should be a required dependency for s3fs-fuse.
It is great to see such a rapid and complete response to an issue like this from an open-source project. Thank you.
Nick
@ggtakec commented on GitHub (Mar 19, 2020):
I looked at this issue and comment of #1253 and I created #1254.
It is that Linux and macOS have a default mime.types path, and this file path can be specified by "mime" option.
And if mime.types cannot be loaded correctly, s3fs will fail to start.
Please let me know if you find any problems, I think the PR will be merged soon.
@ggtakec commented on GitHub (Apr 11, 2020):
I closed this issue because #1254 had been merged into the current master.
If you still have problems, please reopen.
Thanks,