starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 05:16:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1409] failed to try resolving symlinks #745

New issue
Closed
opened 2026-03-04 01:48:25 +03:00 by kerem · 5 comments
kerem commented 2026-03-04 01:48:25 +03:00
Owner
Copy link

Originally created by @guillelucero on GitHub (Sep 18, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1409

Additional Information

We are buling docker image based in ubuntu:14.04. This docker image runs in kubernetes cluster (EKS).
Client Version: v1.17.9-eks-4c6976
Server Version: v1.17.9-eks-4c6976

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.87 (commit:b5ffd41) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Compiled from github

Kernel information (uname -r)

4.14.193-149.317.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

s3fs command line used, if applicable

/usr/local/bin/s3fs $S3_BUCKET $MNT_POINT -f -o endpoint=${S3_REGION},allow_other,use_cache=/tmp,max_stat_cache_size=1000,stat_cache_expire=900,retries=5,connect_timeout=10,nonempty

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

failed to try resolving symlinks in path "/var/log/pods/default_lb-shop-s3fs-6q4cm_afb6c2c5-c11d-4e67-a586-89b511344534/lb-shop-s3fs/20.log": lstat /var/log/pods/default_lb-shop-s3fs-6q4cm_

Details about issue

Containers is running as kubernetes daemonset. It works fine.
During stress test, containers crash showing error shows above.
After crash no new containers lunched by the daemonset could be run.
Only way to recover our kubernetes node is delete them from the cluster. in this case autoscaling add a new one.

Originally created by @guillelucero on GitHub (Sep 18, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1409 ### Additional Information We are buling docker image based in ubuntu:14.04. This docker image runs in kubernetes cluster (EKS). Client Version: v1.17.9-eks-4c6976 Server Version: v1.17.9-eks-4c6976 #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.87 (commit:b5ffd41) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Compiled from github #### Kernel information (uname -r) 4.14.193-149.317.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Ubuntu" VERSION="14.04.6 LTS, Trusty Tahr" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 14.04.6 LTS" VERSION_ID="14.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" #### s3fs command line used, if applicable ``` /usr/local/bin/s3fs $S3_BUCKET $MNT_POINT -f -o endpoint=${S3_REGION},allow_other,use_cache=/tmp,max_stat_cache_size=1000,stat_cache_expire=900,retries=5,connect_timeout=10,nonempty ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` failed to try resolving symlinks in path "/var/log/pods/default_lb-shop-s3fs-6q4cm_afb6c2c5-c11d-4e67-a586-89b511344534/lb-shop-s3fs/20.log": lstat /var/log/pods/default_lb-shop-s3fs-6q4cm_ ``` ### Details about issue Containers is running as kubernetes daemonset. It works fine. During stress test, containers crash showing error shows above. After crash no new containers lunched by the daemonset could be run. Only way to recover our kubernetes node is delete them from the cluster. in this case autoscaling add a new one.
kerem 2026-03-04 01:48:25 +03:00
  • closed this issue
  • added the
    need info
         label
kerem commented 2026-03-04 01:48:26 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Oct 10, 2020):

Could you attach gdb so we can determine the backtrace of the crashing s3fs process? Running with debug logging -d might help as well. Finally try testing with master which includes a few concurrency fixes.

<!-- gh-comment-id:706567431 --> @gaul commented on GitHub (Oct 10, 2020): Could you attach gdb so we can determine the backtrace of the crashing s3fs process? Running with debug logging `-d` might help as well. Finally try testing with master which includes a few concurrency fixes.
kerem commented 2026-03-04 01:48:26 +03:00
Author
Owner
Copy link

@skolenkin commented on GitHub (Oct 20, 2020):

Hello,

I have the same issue.

My environment:

s3fs --version
Amazon Simple Storage Service File System V1.86 (commit:bb20fc3) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>

EKS 1.17
EKS worjker node:

cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Docker version:

docker version
Client:
 Version:           19.03.6-ce
 API version:       1.40
 Go version:        go1.13.4
 Git commit:        369ce74
 Built:             Fri May 29 04:01:26 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.6-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       369ce74
  Built:            Fri May 29 04:01:57 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.2
  GitCommit:        ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Containers are running as Kubernetes DaemonSet but after a few redeploys I have the following error:
ailed to try resolving symlinks in path

kubectl get pods
NAME                READY   STATUS              RESTARTS   AGE
s3-provider-whlrn   0/1     RunContainerError   4          113s
s3-provider-x28v8   0/1     RunContainerError   4          113s

kubectl logs -f s3-provider-whlrn
failed to try resolving symlinks in path "/var/log/pods/default_s3-provider-whlrn_8f7d4207-d170-49d4-b21c-c4722a6ba881/s3fuse/4.log": lstat /var/log/pods/default_s3-provider-whlrn_8f7d4207-d170-49d4-b21c-c4722a6ba881/s3fuse/4.log: no such file or directory

I was able to run container without shared volume:

        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        #- name: mntdatas3fs
        #mountPath: /var/s3fs:shared

Also, I have tried configure shared mount on the k8s nodes but these steps didn’t help:

mkdir /mnt/data-s3fs
mount --bind /mnt/data-s3fs /mnt/data-s3fs
mount --make-shared /mnt/data-s3fs

findmnt -o TARGET,PROPAGATION /mnt/data-s3fs
TARGET         PROPAGATION
/mnt/data-s3fs shared

With the following DaemonSet config and run.sh script works but not always:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: s3-provider
  name: s3-provider
spec:
  selector:
    matchLabels:
      app: s3-provider
  template:
    metadata:
      labels:
        app: s3-provider
    spec:
      containers:
      - name: s3fuse
        image: ECR/s3fs:latest
        imagePullPolicy: Always
        securityContext:
          privileged: true
          capabilities:
            add:
            - SYS_ADMIN
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh","-c","umount -f /var/s3fs"]
        envFrom:
        - configMapRef:
            name: s3-config
        - secretRef:
            name: s3-secret
        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        - name: mntdatas3fs
          mountPath: /var/s3fs:shared
      volumes:
      - name: devfuse
        hostPath:
          path: /dev/fuse
      - name: mntdatas3fs
        hostPath:
          path: /mnt/data-s3fs

Below run.sh startup script:

#!/bin/sh
set -e
echo "$AWS_KEY:$AWS_SECRET_KEY" > passwd && chmod 600 passwd
s3fs "$S3_BUCKET" "$MNT_POINT" -o passwd_file=passwd -o nonempty -d -f -o f2 -o curldbg
#s3fs "$S3_BUCKET" "$MNT_POINT" -o passwd_file=passwd  && tail -f /dev/null

Dockerfile

FROM alpine:latest

ENV MNT_POINT /var/s3fs

ARG S3FS_VERSION=v1.87

RUN apk --update --no-cache add fuse alpine-sdk automake autoconf libxml2-dev fuse-dev curl-dev git bash; \
    git clone https://github.com/s3fs-fuse/s3fs-fuse.git; \
    cd s3fs-fuse; \
    git checkout tags/${S3FS_VERSION}; \
    ./autogen.sh; \
    ./configure --prefix=/usr; \
    make; \
    make install; \
    make clean; \
    rm -rf /var/cache/apk/*; \
    apk del git automake autoconf;

RUN mkdir -p "$MNT_POINT"

COPY run.sh run.sh
CMD ./run.sh
<!-- gh-comment-id:712753853 --> @skolenkin commented on GitHub (Oct 20, 2020): Hello, I have the same issue. My environment: ``` s3fs --version Amazon Simple Storage Service File System V1.86 (commit:bb20fc3) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> ``` EKS 1.17 EKS worjker node: ``` cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" ``` Docker version: ``` docker version Client: Version: 19.03.6-ce API version: 1.40 Go version: go1.13.4 Git commit: 369ce74 Built: Fri May 29 04:01:26 2020 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 19.03.6-ce API version: 1.40 (minimum version 1.12) Go version: go1.13.4 Git commit: 369ce74 Built: Fri May 29 04:01:57 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.3.2 GitCommit: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683 ``` ``` kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} ``` Containers are running as Kubernetes DaemonSet but after a few redeploys I have the following error: ailed to try resolving symlinks in path ``` kubectl get pods NAME READY STATUS RESTARTS AGE s3-provider-whlrn 0/1 RunContainerError 4 113s s3-provider-x28v8 0/1 RunContainerError 4 113s ``` ``` kubectl logs -f s3-provider-whlrn failed to try resolving symlinks in path "/var/log/pods/default_s3-provider-whlrn_8f7d4207-d170-49d4-b21c-c4722a6ba881/s3fuse/4.log": lstat /var/log/pods/default_s3-provider-whlrn_8f7d4207-d170-49d4-b21c-c4722a6ba881/s3fuse/4.log: no such file or directory ``` I was able to run container without shared volume: ``` volumeMounts: - name: devfuse mountPath: /dev/fuse #- name: mntdatas3fs #mountPath: /var/s3fs:shared ``` Also, I have tried configure shared mount on the k8s nodes but these steps didn’t help: ``` mkdir /mnt/data-s3fs mount --bind /mnt/data-s3fs /mnt/data-s3fs mount --make-shared /mnt/data-s3fs findmnt -o TARGET,PROPAGATION /mnt/data-s3fs TARGET PROPAGATION /mnt/data-s3fs shared ``` With the following DaemonSet config and run.sh script works but not always: ``` apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: s3-provider name: s3-provider spec: selector: matchLabels: app: s3-provider template: metadata: labels: app: s3-provider spec: containers: - name: s3fuse image: ECR/s3fs:latest imagePullPolicy: Always securityContext: privileged: true capabilities: add: - SYS_ADMIN lifecycle: preStop: exec: command: ["/bin/sh","-c","umount -f /var/s3fs"] envFrom: - configMapRef: name: s3-config - secretRef: name: s3-secret volumeMounts: - name: devfuse mountPath: /dev/fuse - name: mntdatas3fs mountPath: /var/s3fs:shared volumes: - name: devfuse hostPath: path: /dev/fuse - name: mntdatas3fs hostPath: path: /mnt/data-s3fs ``` Below run.sh startup script: ``` #!/bin/sh set -e echo "$AWS_KEY:$AWS_SECRET_KEY" > passwd && chmod 600 passwd s3fs "$S3_BUCKET" "$MNT_POINT" -o passwd_file=passwd -o nonempty -d -f -o f2 -o curldbg #s3fs "$S3_BUCKET" "$MNT_POINT" -o passwd_file=passwd && tail -f /dev/null ``` Dockerfile ``` FROM alpine:latest ENV MNT_POINT /var/s3fs ARG S3FS_VERSION=v1.87 RUN apk --update --no-cache add fuse alpine-sdk automake autoconf libxml2-dev fuse-dev curl-dev git bash; \ git clone https://github.com/s3fs-fuse/s3fs-fuse.git; \ cd s3fs-fuse; \ git checkout tags/${S3FS_VERSION}; \ ./autogen.sh; \ ./configure --prefix=/usr; \ make; \ make install; \ make clean; \ rm -rf /var/cache/apk/*; \ apk del git automake autoconf; RUN mkdir -p "$MNT_POINT" COPY run.sh run.sh CMD ./run.sh ```
kerem commented 2026-03-04 01:48:26 +03:00
Author
Owner
Copy link

@jimdumont commented on GitHub (Nov 23, 2020):

Any progress on this issue? It is preventing upgrade to s3fs V1.87 which has some interesting updates. Thanks!

<!-- gh-comment-id:732363660 --> @jimdumont commented on GitHub (Nov 23, 2020): Any progress on this issue? It is preventing upgrade to s3fs V1.87 which has some interesting updates. Thanks!
kerem commented 2026-03-04 01:48:26 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Feb 19, 2021):

I'm not sure what the issue is here. Does s3fs crash or do symlinks fail to resolve? You might also try testing with the latest version. I will close out this issue unless there some steps to reproduce the symptoms.

<!-- gh-comment-id:781822760 --> @gaul commented on GitHub (Feb 19, 2021): I'm not sure what the issue is here. Does s3fs crash or do symlinks fail to resolve? You might also try testing with the latest version. I will close out this issue unless there some steps to reproduce the symptoms.
kerem commented 2026-03-04 01:48:26 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Apr 21, 2021):

Please reopen if symptoms persist.

<!-- gh-comment-id:824111598 --> @gaul commented on GitHub (Apr 21, 2021): Please reopen if symptoms persist.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#745
No description provided.