[GH-ISSUE #1557] Verify s3fs behaviour with S3 VPC Endpoints and regional configuration in fstab #819

New issue

Open

opened 2026-03-04 01:49:03 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:49:03 +03:00

Owner

Originally created by @kwerey on GitHub (Feb 8, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1557

Hi s3fs team - thank you for this useful tool!

I've been debugging some instability in mounting a bucket with s3fs in fstab - I keep logging in to find the message

ls: cannot access /outliers: Transport endpoint is not connected

I'm trying to confirm that

s3fs is connecting to the right region
s3fs is making use of the VPC endpoint

My fstab configuration is this:

$ cat /etc/fstab
s3fs#BUCKET-1 /s3mnt fuse allow_other,use_rrs,uid=hadean,gid=hadean,use_cache=/storage/s3-cache-shared,iam_role=IAM_ROLE,_netdev,curldbg,-d,endpoint=eu-west-2 0 0
s3fs#BUCKET-1 /outliers fuse allow_other,uid=hadean,gid=hadean,iam_role=IAM_ROLE,nonempty,_netdev,curldbg,-d,endpoint=eu-west-2 0 0

I was surprised after adding the endpoint=eu-west-2 property to to see /var/log/messages refer to BUCKET-1.s3.amazonaws.com. The VPC Endpoint docs say you must use a region-specific HTTP request to ensure S3 requests go via the endpoint:

Endpoints currently do not support cross-Region requests—ensure that you create your endpoint in the same Region as your bucket. You can find the location of your bucket by using the Amazon S3 console, or by using the get-bucket-location command. Use a Region-specific Amazon S3 endpoint to access your bucket; for example, mybucket.s3.us-west-2.amazonaws.com

By adding -d,-d,-d into fstab I get a much more verbose log that includes x-amz-bucket-region: eu-west-2 and that gives me an IP address

* Closing connection 0
*   Trying 52.95.150.9...
* TCP_NODELAY set
* Connected to BUCKET-1.s3.amazonaws.com (52.95.150.9) port 80 (#0)
> GET / HTTP/1.1
Host: BUCKET-1.s3.amazonaws.com
Accept: */*
Authorization: AWS4-HMAC-SHA256 [etc etc]

< HTTP/1.1 200 OK
< x-amz-id-2: [etc etc]
< Date: Mon, 08 Feb 2021 15:10:31 GMT
< x-amz-bucket-region: eu-west-2
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Server: AmazonS3
<

That IP address matches the prefix lists for the VPC endpoint (from VPC -> Route Tables): com.amazonaws.eu-west-2.s3, 52.95.150.0/24, 3.5.244.0/22, 52.95.148.0/23

My host is connecting to 52.95.150.9, which is within that 52.95.150.0/24 CIDR block - so the route table seems to be sending it to the VPC endpoint and hitting eu-west-2 cleanly.

If anyone can confirm whether seeing BUCKET-1.s3.amazonaws.com in logs instead of BUCKET-1.REGION.s3.amazonaws.com is expected behaviour, and not the cause of my buckets-unmounting-themselves issues, I'd much appreciate it!

Additional Information

Version of s3fs being used (s3fs --version)

$ s3fs --version
Amazon Simple Storage Service File System V1.79 with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

$ rpm -qi fuse
Name        : fuse
Version     : 2.9.4
Release     : 1.17.amzn1
Architecture: x86_64
Install Date: Tue 02 Feb 2021 11:58:02 UTC
Group       : System Environment/Base
Size        : 213757
License     : GPL+
Signature   : RSA/SHA256, Tue 07 Jul 2015 19:28:26 UTC, Key ID bcb4a85b21c0f39f
Source RPM  : fuse-2.9.4-1.17.amzn1.src.rpm
Build Date  : Tue 07 Jul 2015 18:22:41 UTC
Build Host  : build-60009.build
Relocations : (not relocatable)
Packager    : Amazon.com, Inc. <http://aws.amazon.com>
Vendor      : Amazon.com
URL         : http://fuse.sf.net
Summary     : File System in Userspace (FUSE) utilities
Description :
With FUSE it is possible to implement a fully functional filesystem in a
userspace program. This package contains the FUSE userspace tools to
mount a FUSE filesystem.

Kernel information (uname -r)

$ uname -r
4.14.214-118.339.amzn1.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

$ cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

/etc/fstab entry, if applicable

$ cat /etc/fstab
s3fs#BUCKET /s3mnt fuse allow_other,use_rrs,uid=hadean,gid=hadean,use_cache=/storage/s3-cache-shared,iam_role=IAM_ROLE,_netdev,curldbg,-d,endpoint=eu-west-2 0 0
s3fs#BUCKET /outliers fuse allow_other,uid=hadean,gid=hadean,iam_role=IAM_ROLE,nonempty,_netdev,curldbg,-d,endpoint=eu-west-2 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Noted above

Details about issue

Originally created by @kwerey on GitHub (Feb 8, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1557 Hi s3fs team - thank you for this useful tool! I've been debugging some instability in mounting a bucket with s3fs in fstab - I keep logging in to find the message ``` ls: cannot access /outliers: Transport endpoint is not connected ``` I'm trying to confirm that 1. s3fs is connecting to the right region 2. s3fs is making use of the VPC endpoint My fstab configuration is this: ``` $ cat /etc/fstab s3fs#BUCKET-1 /s3mnt fuse allow_other,use_rrs,uid=hadean,gid=hadean,use_cache=/storage/s3-cache-shared,iam_role=IAM_ROLE,_netdev,curldbg,-d,endpoint=eu-west-2 0 0 s3fs#BUCKET-1 /outliers fuse allow_other,uid=hadean,gid=hadean,iam_role=IAM_ROLE,nonempty,_netdev,curldbg,-d,endpoint=eu-west-2 0 0 ``` I was surprised after adding the `endpoint=eu-west-2` property to to see /var/log/messages refer to `BUCKET-1.s3.amazonaws.com`. The VPC Endpoint docs say you must use a region-specific HTTP request to ensure S3 requests go via the endpoint: > Endpoints currently do not support cross-Region requests—ensure that you create your endpoint in the same Region as your bucket. You can find the location of your bucket by using the Amazon S3 console, or by using the get-bucket-location command. Use a Region-specific Amazon S3 endpoint to access your bucket; for example, mybucket.s3.us-west-2.amazonaws.com By adding `-d,-d,-d` into fstab I get a much more verbose log that includes `x-amz-bucket-region: eu-west-2` and that gives me an IP address ``` * Closing connection 0 * Trying 52.95.150.9... * TCP_NODELAY set * Connected to BUCKET-1.s3.amazonaws.com (52.95.150.9) port 80 (#0) > GET / HTTP/1.1 Host: BUCKET-1.s3.amazonaws.com Accept: */* Authorization: AWS4-HMAC-SHA256 [etc etc] < HTTP/1.1 200 OK < x-amz-id-2: [etc etc] < Date: Mon, 08 Feb 2021 15:10:31 GMT < x-amz-bucket-region: eu-west-2 < Content-Type: application/xml < Transfer-Encoding: chunked < Server: AmazonS3 < ``` That IP address matches the prefix lists for the VPC endpoint (from VPC -> Route Tables): `com.amazonaws.eu-west-2.s3, 52.95.150.0/24, 3.5.244.0/22, 52.95.148.0/23` My host is connecting to 52.95.150.9, which is within that 52.95.150.0/24 CIDR block - so the route table seems to be sending it to the VPC endpoint and hitting eu-west-2 cleanly. If anyone can confirm whether seeing `BUCKET-1.s3.amazonaws.com` in logs instead of `BUCKET-1.REGION.s3.amazonaws.com` is expected behaviour, and not the cause of my buckets-unmounting-themselves issues, I'd much appreciate it! ### Additional Information #### Version of s3fs being used (s3fs --version) ``` $ s3fs --version Amazon Simple Storage Service File System V1.79 with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` $ rpm -qi fuse Name : fuse Version : 2.9.4 Release : 1.17.amzn1 Architecture: x86_64 Install Date: Tue 02 Feb 2021 11:58:02 UTC Group : System Environment/Base Size : 213757 License : GPL+ Signature : RSA/SHA256, Tue 07 Jul 2015 19:28:26 UTC, Key ID bcb4a85b21c0f39f Source RPM : fuse-2.9.4-1.17.amzn1.src.rpm Build Date : Tue 07 Jul 2015 18:22:41 UTC Build Host : build-60009.build Relocations : (not relocatable) Packager : Amazon.com, Inc. <http://aws.amazon.com> Vendor : Amazon.com URL : http://fuse.sf.net Summary : File System in Userspace (FUSE) utilities Description : With FUSE it is possible to implement a fully functional filesystem in a userspace program. This package contains the FUSE userspace tools to mount a FUSE filesystem. ``` #### Kernel information (uname -r) ``` $ uname -r 4.14.214-118.339.amzn1.x86_64 ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` $ cat /etc/os-release NAME="Amazon Linux AMI" VERSION="2018.03" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2018.03" PRETTY_NAME="Amazon Linux AMI 2018.03" ANSI_COLOR="0;33" CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" HOME_URL="http://aws.amazon.com/amazon-linux-ami/" ``` #### /etc/fstab entry, if applicable ``` $ cat /etc/fstab s3fs#BUCKET /s3mnt fuse allow_other,use_rrs,uid=hadean,gid=hadean,use_cache=/storage/s3-cache-shared,iam_role=IAM_ROLE,_netdev,curldbg,-d,endpoint=eu-west-2 0 0 s3fs#BUCKET /outliers fuse allow_other,uid=hadean,gid=hadean,iam_role=IAM_ROLE,nonempty,_netdev,curldbg,-d,endpoint=eu-west-2 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _Noted above_ ### Details about issue

kerem commented

2026-03-04 01:49:04 +03:00

Author

Owner

@syu-lk4b commented on GitHub (Nov 10, 2021):

I am facing the same issue now, this happens when we start the machine learning job, when the job start, it will read a lot of data from the s3mount, then after a while running.

from the mount commands, you can still see it.
s3fs on /home/jovyan/s3mount type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=1000,group_id=100)

however from the df command, the s3mount is gone

#df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 7436684528 2143823288 5292844856 29% /
tmpfs 65536 0 65536 0% /dev
tmpfs 395629536 0 395629536 0% /sys/fs/cgroup
tmpfs 395629536 1308 395628228 1% /dev/shm
udev 395597684 0 395597684 0% /dev/fuse
/dev/md1 7436684528 2143823288 5292844856 29% /etc/hosts
/dev/rbd0 10190100 42516 10131200 1% /home/jovyan
tmpfs 395629536 12 395629524 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 395629536 12 395629524 1% /proc/driver/nvidia

when I try to mount it again, the "Transport endpoint is not connected" error message pops up

@syu-lk4b commented on GitHub (Nov 10, 2021): I am facing the same issue now, this happens when we start the machine learning job, when the job start, it will read a lot of data from the s3mount, then after a while running. from the mount commands, you can still see it. s3fs on /home/jovyan/s3mount type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=1000,group_id=100) however from the df command, the s3mount is gone #df Filesystem 1K-blocks Used Available Use% Mounted on overlay 7436684528 2143823288 5292844856 29% / tmpfs 65536 0 65536 0% /dev tmpfs 395629536 0 395629536 0% /sys/fs/cgroup tmpfs 395629536 1308 395628228 1% /dev/shm udev 395597684 0 395597684 0% /dev/fuse /dev/md1 7436684528 2143823288 5292844856 29% /etc/hosts /dev/rbd0 10190100 42516 10131200 1% /home/jovyan tmpfs 395629536 12 395629524 1% /run/secrets/kubernetes.io/serviceaccount tmpfs 395629536 12 395629524 1% /proc/driver/nvidia when I try to mount it again, the "Transport endpoint is not connected" error message pops up