starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1588] 1.88 release appears to have broken in Docker #831

New issue
Closed
opened 2026-03-04 01:49:09 +03:00 by kerem · 10 comments
kerem commented 2026-03-04 01:49:09 +03:00
Owner
Copy link

Originally created by @dannysauer on GitHub (Feb 23, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1588

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

root@51d2dd529caa:/# dpkg -s fuse
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 113
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Architecture: amd64
Version: 2.9.9-3
Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-3), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
/etc/fuse.conf 298587592c8444196833f317def414f2
Description: Filesystem in Userspace
Filesystem in Userspace (FUSE) is a simple interface for userspace programs to
export a virtual filesystem to the Linux kernel. It also aims to provide a
secure method for non privileged users to create and mount their own filesystem
implementations.
Original-Maintainer: Laszlo Boszormenyi (GCS) gcs@debian.org
Homepage: https://github.com/libfuse/libfuse/wiki

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

example: 2.9.4

Kernel information (uname -r)

4.15.0-128-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

root@51d2dd529caa:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="21.04 (Hirsute Hippo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu Hirsute Hippo (development branch)"
VERSION_ID="21.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=hirsute
UBUNTU_CODENAME=hirsute

s3fs command line used, if applicable

root@51d2dd529caa:/# s3fs --version
Amazon Simple Storage Service File System V1.88 (commit:unknown) with GnuTLS(gcrypt)
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

/etc/fstab entry, if applicable

N/A

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

Ubuntu 21.04 updated s3fs from 1.87 to1.88 yesterday afternoon. This immediately broke the ability to mount within a container. So I think this tracks back to upstream here.

The docker command I run is:

docker run --rm -ti --env-file=dannysauer-kong-packages.creds --env-file=bucket.defs --device /dev/fuse --cap-add SYS_ADMIN pkgtst /bin/bash

Inside the container, I run this script:

root@f499a24e89b9:/# cat /usr/local/bin/mounter.sh 
#!/bin/bash
################################################################################
# Mount incoming and outgoing buckets
################################################################################
set -o errexit

mkdir -p /opt/pkgs/{incoming,outgoing}
s3fs "$BUCKET_INCOMING" /opt/pkgs/incoming/
s3fs "$BUCKET_OUTGOING" /opt/pkgs/outgoing/ -o nomultipart

This should all work, and was working prior to the version bump. Here's the output on the command line:

root@f499a24e89b9:/# s3fs "$BUCKET_INCOMING" /opt/pkgs/incoming/ -dd
fuse: bad mount point `/opt/pkgs/incoming/': Operation not permitted
root@f499a24e89b9:/# ls -ld / /opt /opt/pkgs/ /opt/pkgs/incoming/
drwxr-xr-x 1 root root 4096 Feb 23 19:12 /
drwxr-xr-x 1 root root 4096 Feb 23 19:12 /opt
drwxr-xr-x 4 root root 4096 Feb 23 19:12 /opt/pkgs/
drwxr-xr-x 2 root root 4096 Feb 23 19:12 /opt/pkgs/incoming/

I don't have syslog in the container, but I do have strace:

root@f499a24e89b9:/# strace -f -o /tmp/s3fs.trace s3fs "$BUCKET_INCOMING" /opt/pkgs/incoming/ -dd
fuse: bad mount point `/opt/pkgs/incoming/': Operation not permitted
root@f499a24e89b9:/# tail -10 /tmp/s3fs.trace 
58    close(4)                          = 0
58    close(3)                          = 0
58    statfs("/tmp/.", {f_type=OVERLAYFS_SUPER_MAGIC, f_bsize=4096, f_blocks=65793553, f_bfree=63532184, f_bavail=60172645, f_files=16777216, f_ffree=16508989, f_fsid={val=[4275870371, 3177561175]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
58    readlink("/opt", 0x7ffd9c74f020, 1023) = -1 EINVAL (Invalid argument)
58    readlink("/opt/pkgs", 0x7ffd9c74f020, 1023) = -1 EINVAL (Invalid argument)
58    readlink("/opt/pkgs/incoming", 0x7ffd9c74f020, 1023) = -1 EINVAL (Invalid argument)
58    faccessat2(AT_FDCWD, "/opt/pkgs/incoming/", F_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
58    write(2, "fuse: bad mount point `/opt/pkgs"..., 69) = 69
58    exit_group(1)                     = ?
58    +++ exited with 1 +++

For some reason, the faccessat2 syscall is failing. This specific example is happening on both docker on Windows called from OpenSUSE Leap 15.2 in WSL2, and in older and newer Docker on bare metal Ubuntu & OpenSUSE Leap - all of which worked before. So I've ruled Docker itself out.

To replicate, the Dockerfile is pretty simple:

FROM ubuntu:hirsute

RUN \
  apt update \
  && \
  apt -y dist-upgrade \
  && \
  DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \
    awscli \
    jq \
    s3fs \
    createrepo-c \
    aptly \
  && \
  apt clean

COPY support_files/bin/* /usr/local/bin/

# drop into an interactive shell by default
CMD ["/bin/bash", "--login"]

And then the mounter.sh from above is in support_files/bin/.

Originally created by @dannysauer on GitHub (Feb 23, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1588 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) root@51d2dd529caa:/# dpkg -s fuse Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 113 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Architecture: amd64 Version: 2.9.9-3 Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-3), adduser, mount (>= 2.19.1), sed (>= 4) Conffiles: /etc/fuse.conf 298587592c8444196833f317def414f2 Description: Filesystem in Userspace Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual filesystem to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own filesystem implementations. Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> Homepage: https://github.com/libfuse/libfuse/wiki #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) _example: 2.9.4_ #### Kernel information (uname -r) 4.15.0-128-generic #### GNU/Linux Distribution, if applicable (cat /etc/os-release) root@51d2dd529caa:/# cat /etc/os-release NAME="Ubuntu" VERSION="21.04 (Hirsute Hippo)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu Hirsute Hippo (development branch)" VERSION_ID="21.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=hirsute UBUNTU_CODENAME=hirsute #### s3fs command line used, if applicable root@51d2dd529caa:/# s3fs --version Amazon Simple Storage Service File System V1.88 (commit:unknown) with GnuTLS(gcrypt) Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. #### /etc/fstab entry, if applicable N/A #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` ``` ### Details about issue Ubuntu 21.04 updated s3fs from 1.87 to1.88 yesterday afternoon. This immediately broke the ability to mount within a container. So I _think_ this tracks back to upstream here. The docker command I run is: ``` docker run --rm -ti --env-file=dannysauer-kong-packages.creds --env-file=bucket.defs --device /dev/fuse --cap-add SYS_ADMIN pkgtst /bin/bash ``` Inside the container, I run this script: ```bash root@f499a24e89b9:/# cat /usr/local/bin/mounter.sh #!/bin/bash ################################################################################ # Mount incoming and outgoing buckets ################################################################################ set -o errexit mkdir -p /opt/pkgs/{incoming,outgoing} s3fs "$BUCKET_INCOMING" /opt/pkgs/incoming/ s3fs "$BUCKET_OUTGOING" /opt/pkgs/outgoing/ -o nomultipart ``` This should all work, and was working prior to the version bump. Here's the output on the command line: ``` root@f499a24e89b9:/# s3fs "$BUCKET_INCOMING" /opt/pkgs/incoming/ -dd fuse: bad mount point `/opt/pkgs/incoming/': Operation not permitted root@f499a24e89b9:/# ls -ld / /opt /opt/pkgs/ /opt/pkgs/incoming/ drwxr-xr-x 1 root root 4096 Feb 23 19:12 / drwxr-xr-x 1 root root 4096 Feb 23 19:12 /opt drwxr-xr-x 4 root root 4096 Feb 23 19:12 /opt/pkgs/ drwxr-xr-x 2 root root 4096 Feb 23 19:12 /opt/pkgs/incoming/ ``` I don't have syslog in the container, but I do have strace: ``` root@f499a24e89b9:/# strace -f -o /tmp/s3fs.trace s3fs "$BUCKET_INCOMING" /opt/pkgs/incoming/ -dd fuse: bad mount point `/opt/pkgs/incoming/': Operation not permitted root@f499a24e89b9:/# tail -10 /tmp/s3fs.trace 58 close(4) = 0 58 close(3) = 0 58 statfs("/tmp/.", {f_type=OVERLAYFS_SUPER_MAGIC, f_bsize=4096, f_blocks=65793553, f_bfree=63532184, f_bavail=60172645, f_files=16777216, f_ffree=16508989, f_fsid={val=[4275870371, 3177561175]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0 58 readlink("/opt", 0x7ffd9c74f020, 1023) = -1 EINVAL (Invalid argument) 58 readlink("/opt/pkgs", 0x7ffd9c74f020, 1023) = -1 EINVAL (Invalid argument) 58 readlink("/opt/pkgs/incoming", 0x7ffd9c74f020, 1023) = -1 EINVAL (Invalid argument) 58 faccessat2(AT_FDCWD, "/opt/pkgs/incoming/", F_OK, AT_EACCESS) = -1 EPERM (Operation not permitted) 58 write(2, "fuse: bad mount point `/opt/pkgs"..., 69) = 69 58 exit_group(1) = ? 58 +++ exited with 1 +++ ``` For some reason, the faccessat2 syscall is failing. This specific example is happening on both docker on Windows called from OpenSUSE Leap 15.2 in WSL2, and in older and newer Docker on bare metal Ubuntu & OpenSUSE Leap - all of which worked before. So I've ruled Docker itself out. To replicate, the Dockerfile is pretty simple: ``` FROM ubuntu:hirsute RUN \ apt update \ && \ apt -y dist-upgrade \ && \ DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \ awscli \ jq \ s3fs \ createrepo-c \ aptly \ && \ apt clean COPY support_files/bin/* /usr/local/bin/ # drop into an interactive shell by default CMD ["/bin/bash", "--login"] ``` And then the `mounter.sh` from above is in `support_files/bin/`.
kerem 2026-03-04 01:49:09 +03:00
  • closed this issue
  • added the
    need info
         label
kerem commented 2026-03-04 01:49:10 +03:00
Author
Owner
Copy link

@dannysauer commented on GitHub (Feb 23, 2021):

It's worth noting that adding --privileged to the docker command does make this work. But I'm more interested in tracking down what it is that changed to require additional privileges, as the prior version worked with just the sys_admin capability added to enable mounting. Maybe the description should be "requires more permissions" rather than "broken". 🤷

Also possibly related to #1246.

<!-- gh-comment-id:784459320 --> @dannysauer commented on GitHub (Feb 23, 2021): It's worth noting that adding `--privileged` to the docker command does make this work. But I'm more interested in tracking down what it is that changed to require additional privileges, as the prior version worked with just the sys_admin capability added to enable mounting. Maybe the description should be "requires more permissions" rather than "broken". :shrug: Also possibly related to #1246.
kerem commented 2026-03-04 01:49:10 +03:00
Author
Owner
Copy link

@dannysauer commented on GitHub (Feb 23, 2021):

https://launchpad.net/ubuntu/+source/s3fs-fuse/+changelog might be useful. Specifically the diff at http://launchpadlibrarian.net/517283969/s3fs-fuse_1.87-1_1.88-1.diff.gz

<!-- gh-comment-id:784464499 --> @dannysauer commented on GitHub (Feb 23, 2021): https://launchpad.net/ubuntu/+source/s3fs-fuse/+changelog might be useful. Specifically the diff at http://launchpadlibrarian.net/517283969/s3fs-fuse_1.87-1_1.88-1.diff.gz
kerem commented 2026-03-04 01:49:10 +03:00
Author
Owner
Copy link

@dannysauer commented on GitHub (Feb 24, 2021):

I think you meant to close #1558, @gaul :)

<!-- gh-comment-id:784623842 --> @dannysauer commented on GitHub (Feb 24, 2021): I think you meant to close #1558, @gaul :)
kerem commented 2026-03-04 01:49:11 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Feb 24, 2021):

Sorry about the typo! Unfortunately we don't have much Docker expertise. Could you try git bisect to find which commit introduced this regression?

<!-- gh-comment-id:784649253 --> @gaul commented on GitHub (Feb 24, 2021): Sorry about the typo! Unfortunately we don't have much Docker expertise. Could you try `git bisect` to find which commit introduced this regression?
kerem commented 2026-03-04 01:49:11 +03:00
Author
Owner
Copy link

@dannysauer commented on GitHub (Feb 24, 2021):

If no one knows off the top of their head, I'll spend some time on it in a few days and try to track down exactly what caused it. I'm in the middle of a pretty high criticality issue until then. :)

Also, I was impressed that the ticket was closed several hours before it was opened. That's some amazing efficiency. 😂

<!-- gh-comment-id:784656710 --> @dannysauer commented on GitHub (Feb 24, 2021): If no one knows off the top of their head, I'll spend some time on it in a few days and try to track down exactly what caused it. I'm in the middle of a pretty high criticality issue until then. :) Also, I was impressed that the ticket was closed several hours before it was opened. That's some amazing efficiency. 😂
kerem commented 2026-03-04 01:49:11 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Apr 25, 2021):

@dannysauer do you have any updates on this issue?

<!-- gh-comment-id:826248238 --> @gaul commented on GitHub (Apr 25, 2021): @dannysauer do you have any updates on this issue?
kerem commented 2026-03-04 01:49:11 +03:00
Author
Owner
Copy link

@dannysauer commented on GitHub (Apr 25, 2021):

I forgot all about this (ended up solving the problem differently with boto/python). Thanks for the reminder.

<!-- gh-comment-id:826272358 --> @dannysauer commented on GitHub (Apr 25, 2021): I forgot all about this (ended up solving the problem differently with boto/python). Thanks for the reminder.
kerem commented 2026-03-04 01:49:11 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jul 25, 2021):

@dannysauer do you have any updates on this issue? Otherwise let's close out this issue.

<!-- gh-comment-id:886176302 --> @gaul commented on GitHub (Jul 25, 2021): @dannysauer do you have any updates on this issue? Otherwise let's close out this issue.
kerem commented 2026-03-04 01:49:11 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Sep 8, 2023):

Closing due to inactivity.

<!-- gh-comment-id:1710960959 --> @gaul commented on GitHub (Sep 8, 2023): Closing due to inactivity.
kerem commented 2026-03-04 01:49:11 +03:00
Author
Owner
Copy link

@dannysauer commented on GitHub (Sep 9, 2023):

That's fair. If nobody has replicated it in two years, it must not be important. :)

<!-- gh-comment-id:1712386515 --> @dannysauer commented on GitHub (Sep 9, 2023): That's fair. If nobody has replicated it in two years, it must not be important. :)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#831
No description provided.