starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1510] Container writing into subdirectory of s3fs mounted director fails with Input/output error first, then succeeds #791

New issue
Open
opened 2026-03-04 01:48:50 +03:00 by kerem · 2 comments
kerem commented 2026-03-04 01:48:50 +03:00
Owner
Copy link

Originally created by @muzzah on GitHub (Jan 1, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1510

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.86 (commit:unknown) with GnuTLS(gcrypt)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.9-3

Kernel information (uname -r)

5.4.0-54-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

s3fs command line used, if applicable

sudo s3fs mahbucket /mnt/mahbucket -f -d -d  -o use_path_request_style -oallow_other,uid=999 -o multipart_size=16 -o "url=https://sos-ch-dk-2.exo.io/"

/etc/fstab entry, if applicable

Not applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

I have a directory /mnt/mahbucket which is mounted using the following command

sudo s3fs mahbucket /mnt/mahbucket -f -d -d  -o use_path_request_style -oallow_other,uid=999 -o multipart_size=16 -o "url=https://sos-ch-dk-2.exo.io/"

user id 999 is the userID the container itself creates internally to run mariadb.

When I start up the contIner for the first time it fails

universe@universe:~$ docker-compose up
Starting nc-db ... error

ERROR: for nc-db  Cannot start service nc-db: error while creating mount source path '/mnt/mahbucket/nc/db': chown /mnt/mahbucket/nc/db: input/output error

ERROR: for nc-db  Cannot start service nc-db: error while creating mount source path '/mnt/mahbucket/nc/db': chown /mnt/mahbucket/nc/db: input/output error
ERROR: Encountered errors while bringing up the project.

And the s3fs log is as follows, not the last output which is an error for a chown command that the container tries to do.

[Snipped out some other logging]

mkdir /nc 0755 umask=0022
[INF] s3fs.cpp:s3fs_mkdir(1128): [path=/nc][mode=0755]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/nc]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/nc]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket?delimiter=/&max-keys=2&prefix=nc/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/?delimiter=/&max-keys=2&prefix=nc/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=nc/] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]   s3fs.cpp:create_directory_object(1101): [path=/nc][mode=0755][time=1609523812][uid=0][gid=0]
[INF]       curl.cpp:PutRequest(3218): [tpath=/nc/]
[INF]       curl.cpp:PutRequest(3236): create zero byte file object.
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/
[INF]       curl.cpp:PutRequest(3319): uploading... [path=/nc/][fd=-1][size=0]
[INF]       curl.cpp:insertV4Headers(2753): computing signature [PUT] [/nc/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       cache.cpp:DelStat(579): delete stat cache entry[path=/nc]
getattr /nc
[INF] s3fs.cpp:s3fs_getattr(876): [path=/nc]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/nc/]
   NODEID: 2
   unique: 16, success, outsize: 144
unique: 18, opcode: LOOKUP (1), nodeid: 2, insize: 43, pid: 3578
LOOKUP /nc/db
getattr /nc/db
[INF] s3fs.cpp:s3fs_getattr(876): [path=/nc/db]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/nc/db]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/nc/db]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket?delimiter=/&max-keys=2&prefix=nc/db/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/?delimiter=/&max-keys=2&prefix=nc/db/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=nc/db/] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
   unique: 18, error: -2 (No such file or directory), outsize: 16
unique: 20, opcode: MKDIR (9), nodeid: 2, insize: 51, pid: 3578
mkdir /nc/db 0755 umask=0022
[INF] s3fs.cpp:s3fs_mkdir(1128): [path=/nc/db][mode=0755]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/nc/db]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/nc/db]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket?delimiter=/&max-keys=2&prefix=nc/db/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/?delimiter=/&max-keys=2&prefix=nc/db/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=nc/db/] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]   s3fs.cpp:create_directory_object(1101): [path=/nc/db][mode=0755][time=1609523812][uid=0][gid=0]
[INF]       curl.cpp:PutRequest(3218): [tpath=/nc/db/]
[INF]       curl.cpp:PutRequest(3236): create zero byte file object.
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:PutRequest(3319): uploading... [path=/nc/db/][fd=-1][size=0]
[INF]       curl.cpp:insertV4Headers(2753): computing signature [PUT] [/nc/db/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       cache.cpp:DelStat(579): delete stat cache entry[path=/nc/db]
getattr /nc/db
[INF] s3fs.cpp:s3fs_getattr(876): [path=/nc/db]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/nc/db/]
   NODEID: 3
   unique: 20, success, outsize: 144
unique: 22, opcode: SETATTR (4), nodeid: 3, insize: 128, pid: 3578
chown /nc/db 0 0
[INF] s3fs.cpp:s3fs_chown(1798): [path=/nc/db][uid=0][gid=0]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/nc/db_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/nc/db_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]     s3fs.cpp:put_headers(828): [path=/nc/db/]
[INF]       curl.cpp:PutHeadRequest(3098): [tpath=/nc/db/]
[INF]       curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/
[INF]       curl.cpp:PutHeadRequest(3185): copying... [path=/nc/db/]
[INF]       curl.cpp:insertV4Headers(2753): computing signature [PUT] [/nc/db/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
   unique: 22, error: -5 (Input/output error), outsize: 16

The second time I start the container with the same command, everything seems to work. The first few lines I get as output from s3fs is that it checks the existing directors it created with the first run and the container continues on with creating init files with no issues

unique: 36044, opcode: LOOKUP (1), nodeid: 1, insize: 43, pid: 3577
LOOKUP /nc
getattr /nc
[INF] s3fs.cpp:s3fs_getattr(876): [path=/nc]
   NODEID: 15
   unique: 36044, success, outsize: 144
unique: 36046, opcode: LOOKUP (1), nodeid: 15, insize: 43, pid: 3577
LOOKUP /nc/db
getattr /nc/db
[INF] s3fs.cpp:s3fs_getattr(876): [path=/nc/db]
   NODEID: 16
   unique: 36046, success, outsize: 144
unique: 36048, opcode: LOOKUP (1), nodeid: 16, insize: 64, pid: 22067
LOOKUP /nc/db/9ab2aa32296e.LOWER-TEST
getattr /nc/db/9ab2aa32296e.LOWER-TEST

[Logs continue]

Im having a hard time understanding why the first run fails and the second run succeeds. Any idea?

Originally created by @muzzah on GitHub (Jan 1, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1510 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.86 (commit:unknown) with GnuTLS(gcrypt) #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.9-3 #### Kernel information (uname -r) 5.4.0-54-generic #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.1 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal #### s3fs command line used, if applicable ``` sudo s3fs mahbucket /mnt/mahbucket -f -d -d -o use_path_request_style -oallow_other,uid=999 -o multipart_size=16 -o "url=https://sos-ch-dk-2.exo.io/" ``` #### /etc/fstab entry, if applicable Not applicable #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` ``` ### Details about issue I have a directory `/mnt/mahbucket` which is mounted using the following command ``` sudo s3fs mahbucket /mnt/mahbucket -f -d -d -o use_path_request_style -oallow_other,uid=999 -o multipart_size=16 -o "url=https://sos-ch-dk-2.exo.io/" ``` user id 999 is the userID the container itself creates internally to run mariadb. When I start up the contIner for the first time it fails ``` universe@universe:~$ docker-compose up Starting nc-db ... error ERROR: for nc-db Cannot start service nc-db: error while creating mount source path '/mnt/mahbucket/nc/db': chown /mnt/mahbucket/nc/db: input/output error ERROR: for nc-db Cannot start service nc-db: error while creating mount source path '/mnt/mahbucket/nc/db': chown /mnt/mahbucket/nc/db: input/output error ERROR: Encountered errors while bringing up the project. ``` And the s3fs log is as follows, not the last output which is an error for a chown command that the container tries to do. ``` [Snipped out some other logging] mkdir /nc 0755 umask=0022 [INF] s3fs.cpp:s3fs_mkdir(1128): [path=/nc][mode=0755] [INF] curl.cpp:HeadRequest(3049): [tpath=/nc] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/nc] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/nc] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket?delimiter=/&max-keys=2&prefix=nc/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/?delimiter=/&max-keys=2&prefix=nc/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=nc/] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:create_directory_object(1101): [path=/nc][mode=0755][time=1609523812][uid=0][gid=0] [INF] curl.cpp:PutRequest(3218): [tpath=/nc/] [INF] curl.cpp:PutRequest(3236): create zero byte file object. [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/ [INF] curl.cpp:PutRequest(3319): uploading... [path=/nc/][fd=-1][size=0] [INF] curl.cpp:insertV4Headers(2753): computing signature [PUT] [/nc/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] cache.cpp:DelStat(579): delete stat cache entry[path=/nc] getattr /nc [INF] s3fs.cpp:s3fs_getattr(876): [path=/nc] [INF] curl.cpp:HeadRequest(3049): [tpath=/nc] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] cache.cpp:AddStat(371): add stat cache entry[path=/nc/] NODEID: 2 unique: 16, success, outsize: 144 unique: 18, opcode: LOOKUP (1), nodeid: 2, insize: 43, pid: 3578 LOOKUP /nc/db getattr /nc/db [INF] s3fs.cpp:s3fs_getattr(876): [path=/nc/db] [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/nc/db] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/nc/db] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket?delimiter=/&max-keys=2&prefix=nc/db/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/?delimiter=/&max-keys=2&prefix=nc/db/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=nc/db/] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 unique: 18, error: -2 (No such file or directory), outsize: 16 unique: 20, opcode: MKDIR (9), nodeid: 2, insize: 51, pid: 3578 mkdir /nc/db 0755 umask=0022 [INF] s3fs.cpp:s3fs_mkdir(1128): [path=/nc/db][mode=0755] [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/nc/db] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/nc/db] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket?delimiter=/&max-keys=2&prefix=nc/db/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/?delimiter=/&max-keys=2&prefix=nc/db/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=nc/db/] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:create_directory_object(1101): [path=/nc/db][mode=0755][time=1609523812][uid=0][gid=0] [INF] curl.cpp:PutRequest(3218): [tpath=/nc/db/] [INF] curl.cpp:PutRequest(3236): create zero byte file object. [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:PutRequest(3319): uploading... [path=/nc/db/][fd=-1][size=0] [INF] curl.cpp:insertV4Headers(2753): computing signature [PUT] [/nc/db/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] cache.cpp:DelStat(579): delete stat cache entry[path=/nc/db] getattr /nc/db [INF] s3fs.cpp:s3fs_getattr(876): [path=/nc/db] [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] cache.cpp:AddStat(371): add stat cache entry[path=/nc/db/] NODEID: 3 unique: 20, success, outsize: 144 unique: 22, opcode: SETATTR (4), nodeid: 3, insize: 128, pid: 3578 chown /nc/db 0 0 [INF] s3fs.cpp:s3fs_chown(1798): [path=/nc/db][uid=0][gid=0] [INF] curl.cpp:HeadRequest(3049): [tpath=/nc/db_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/nc/db_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/nc/db_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:put_headers(828): [path=/nc/db/] [INF] curl.cpp:PutHeadRequest(3098): [tpath=/nc/db/] [INF] curl.cpp:prepare_url(4703): URL is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:prepare_url(4736): URL changed is https://sos-ch-dk-2.exo.io/mahbucket/nc/db/ [INF] curl.cpp:PutHeadRequest(3185): copying... [path=/nc/db/] [INF] curl.cpp:insertV4Headers(2753): computing signature [PUT] [/nc/db/] [] [] [INF] curl.cpp:url_to_host(99): url is https://sos-ch-dk-2.exo.io [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT unique: 22, error: -5 (Input/output error), outsize: 16 ``` The second time I start the container with the same command, everything seems to work. The first few lines I get as output from s3fs is that it checks the existing directors it created with the first run and the container continues on with creating init files with no issues ``` unique: 36044, opcode: LOOKUP (1), nodeid: 1, insize: 43, pid: 3577 LOOKUP /nc getattr /nc [INF] s3fs.cpp:s3fs_getattr(876): [path=/nc] NODEID: 15 unique: 36044, success, outsize: 144 unique: 36046, opcode: LOOKUP (1), nodeid: 15, insize: 43, pid: 3577 LOOKUP /nc/db getattr /nc/db [INF] s3fs.cpp:s3fs_getattr(876): [path=/nc/db] NODEID: 16 unique: 36046, success, outsize: 144 unique: 36048, opcode: LOOKUP (1), nodeid: 16, insize: 64, pid: 22067 LOOKUP /nc/db/9ab2aa32296e.LOWER-TEST getattr /nc/db/9ab2aa32296e.LOWER-TEST [Logs continue] ``` Im having a hard time understanding why the first run fails and the second run succeeds. Any idea?
kerem commented 2026-03-04 01:48:51 +03:00
Author
Owner
Copy link

@muzzah commented on GitHub (Jan 1, 2021):

Also, here is the docker file

version: '3' 

services:


  nc-db:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    container_name: nc-db
    networks:
      - nc_network
    volumes:
      - /mnt/mahbucket/nc/db:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    environment:
      - MYSQL_ROOT_PASSWORD=root
      - MYSQL_PASSWORD=blahblah
      - MYSQL_DATABASE=blahblah
      - MYSQL_USER=blahblah

Note that the volume being mounted is the path in the mounted s3fs directory that is causing the problem. So docker-compose could be trying to create that as root based on the output I see in the logs

chown /nc/db 0 0
[INF] s3fs.cpp:s3fs_chown(1798): [path=/nc/db][uid=0][gid=0]

Then, when the second run kicks off, docker compose sees the directory already there (since creation succeeded but failed when chowning in the first run) and must not try to chown it again, thus the second run continues.

This is all guesses but any advice on how to have the first run succeed would be very helpful.

<!-- gh-comment-id:753361695 --> @muzzah commented on GitHub (Jan 1, 2021): Also, here is the docker file ``` version: '3' services: nc-db: image: mariadb command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW container_name: nc-db networks: - nc_network volumes: - /mnt/mahbucket/nc/db:/var/lib/mysql - /etc/localtime:/etc/localtime:ro environment: - MYSQL_ROOT_PASSWORD=root - MYSQL_PASSWORD=blahblah - MYSQL_DATABASE=blahblah - MYSQL_USER=blahblah ``` Note that the volume being mounted is the path in the mounted s3fs directory that is causing the problem. So docker-compose could be trying to create that as root based on the output I see in the logs ``` chown /nc/db 0 0 [INF] s3fs.cpp:s3fs_chown(1798): [path=/nc/db][uid=0][gid=0] ``` Then, when the second run kicks off, docker compose sees the directory already there (since creation succeeded but failed when chowning in the first run) and must not try to chown it again, thus the second run continues. This is all guesses but any advice on how to have the first run succeed would be very helpful.
kerem commented 2026-03-04 01:48:51 +03:00
Author
Owner
Copy link

@muzzah commented on GitHub (Jan 1, 2021):

Ok so after some more debugging, if I change the bind mount target for the localhost from /mnt/mahbucket/nc/db/ to the root of the mounted folder /mnt/mahbucket then it seems we skip the chown operation. So my suspicions on docker-compose trying to create and then chown the directory seem to be correct.

What I dont understand is, why does the chown fail on a subdirectory of the mounted s3 bucket that has the allow_other option enabled? Docker compose seems to create the directory fine but when it tries to change the ownership s3fs barfs. Anyway to get around this? Is this a bug since docker-compose is running as root?

<!-- gh-comment-id:753374715 --> @muzzah commented on GitHub (Jan 1, 2021): Ok so after some more debugging, if I change the bind mount target for the localhost from `/mnt/mahbucket/nc/db/` to the root of the mounted folder `/mnt/mahbucket` then it seems we skip the chown operation. So my suspicions on docker-compose trying to create and then chown the directory seem to be correct. What I dont understand is, why does the chown fail on a subdirectory of the mounted s3 bucket that has the allow_other option enabled? Docker compose seems to create the directory fine but when it tries to change the ownership s3fs barfs. Anyway to get around this? Is this a bug since docker-compose is running as root?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#791
No description provided.