[GH-ISSUE #1383] Erroneous "file not found" issues after reboot or remount #740

Open
opened 2026-03-04 01:48:21 +03:00 by kerem · 4 comments
Owner

Originally created by @ferricoxide on GitHub (Sep 2, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1383

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

Started with: 1.85 (locally staged version)
Updated to: 1.86 (from EPEL)
Finally: 1.87 (compiled from source retrieved from GitHub)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

fuse-2.9.2-11.el7.x86_64

Kernel information (uname -r)

3.10.0-1127.18.2.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"

s3fs command line used, if applicable

Not directly applicable: issues experienced with OS commands

/etc/fstab entry, if applicable

Not applicable: using mount from /etc/rc.d/rc.local:

/usr/local/bin/s3fs -o allow_other -o umask=0000 -o nonempty  dev-build-tools:/dsw/1 /dsw/tmp/installer

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Sep 02 09:43:21 build-test.lab s3fs[15003]: sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [INF]
Sep 02 09:43:21 build-test.lab s3fs[15003]:     PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777)
Sep 02 09:43:21 build-test.lab s3fs[15003]: Loaded mime information from /etc/mime.types
Sep 02 09:43:21 build-test.lab s3fs[15003]: The path to cache top dir is empty, thus not need to check permission.
Sep 02 09:43:21 build-test.lab s3fs[15005]: init v1.87(commit:unknown) with OpenSSL
Sep 02 09:43:21 build-test.lab s3fs[15005]: check services.
Sep 02 09:43:21 build-test.lab s3fs[15005]:       check a bucket.
Sep 02 09:43:21 build-test.lab s3fs[15005]: curl.cpp:ResetHandle(2150): The CURLOPT_SSL_ENABLE_ALPN option could not be unset. S3 server does not support ALPN, then this option should be disabled to maximize performance. you need to use libcurl 7.36.0 or later.
Sep 02 09:43:21 build-test.lab s3fs[15005]: curl.cpp:ResetHandle(2153): The S3FS_CURLOPT_KEEP_SENDING_ON_ERROR option could not be set. For maximize performance you need to enable this option and you should use libcurl 7.51.0 or later.
Sep 02 09:43:21 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/
Sep 02 09:43:21 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/
Sep 02 09:43:21 build-test.lab s3fs[15005]:       computing signature [GET] [/dsw/1/] [] []
Sep 02 09:43:21 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:43:21 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:43:21 build-test.lab s3fs[15005]: Pool full: destroy the oldest handler
Sep 02 09:43:21 build-test.lab s3fs[15005]:   [path=/dsw/1]
Sep 02 09:43:21 build-test.lab s3fs[15005]: s3fs.cpp:s3fs_init(3505): Failed to initialize signal object, but continue...
Sep 02 09:43:35 build-test.lab s3fs[15005]: [path=/packages]
Sep 02 09:43:35 build-test.lab s3fs[15005]:       [tpath=/packages]
Sep 02 09:43:35 build-test.lab s3fs[15005]:       [tpath=/packages][bpath=][save=][sseckeypos=-1]
Sep 02 09:43:35 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages
Sep 02 09:43:35 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages
Sep 02 09:43:35 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages] [] []
Sep 02 09:43:35 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:43:35 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:43:35 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages]
Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/]
Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/][flags=0x18800]
Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/]
Sep 02 09:43:50 build-test.lab s3fs[15005]:   [path=/]
Sep 02 09:43:50 build-test.lab s3fs[15005]:       [tpath=/]
Sep 02 09:43:50 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools?delimiter=/&max-keys=1000&prefix=dsw/1/
Sep 02 09:43:50 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dsw/1/
Sep 02 09:43:50 build-test.lab s3fs[15005]:       computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dsw/1/] []
Sep 02 09:43:50 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:43:50 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:43:50 build-test.lab s3fs[15005]:   [path=/][list=0]
Sep 02 09:43:50 build-test.lab s3fs[15005]:       [tpath=/packages/][bpath=packages/][save=/packages/][sseckeypos=-1]
Sep 02 09:43:50 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/
Sep 02 09:43:50 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/
Sep 02 09:43:50 build-test.lab s3fs[15005]:       [count=1]
Sep 02 09:43:50 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/] [] []
Sep 02 09:43:50 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:43:50 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:43:50 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/]
Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/packages]
Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/packages]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia][bpath=][save=][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 404 was returned, returning ENOENT
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/][bpath=][save=][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia][flags=0x18800]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia]
Sep 02 09:44:01 build-test.lab s3fs[15005]:   [path=/packages/nvidia]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools?delimiter=/&max-keys=1000&prefix=dsw/1/packages/nvidia/
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dsw/1/packages/nvidia/
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dsw/1/packages/nvidia/] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:   [path=/packages/nvidia/][list=0]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/CUDA/][bpath=CUDA/][save=/packages/nvidia/CUDA/][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/CUDA/
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/CUDA/
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run][bpath=NVIDIA-Linux-x86_64-375.51.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run][bpath=NVIDIA-Linux-x86_64-396.24.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run][bpath=NVIDIA-Linux-x86_64-396.45.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run][bpath=NVIDIA-Linux-x86_64-418.43.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run][bpath=NVIDIA-Linux-x86_64-418.67.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run][bpath=NVIDIA-Linux-x86_64-440.33.01.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm][bpath=cuda-10.1.243-1.x86_64.rpm][save=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-10.1.243-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-10.1.243-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-8-0.conf][bpath=cuda-8-0.conf][save=/packages/nvidia/cuda-8-0.conf][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-8-0.conf
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-8-0.conf
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm][bpath=cuda-repo-rhel7-10.2.89-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm][bpath=cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm][bpath=cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cuda_9.2.148_396.37_linux.run][bpath=cuda_9.2.148_396.37_linux.run][save=/packages/nvidia/cuda_9.2.148_396.37_linux.run][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda_9.2.148_396.37_linux.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda_9.2.148_396.37_linux.run
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz][bpath=cudnn-10.0-linux-x64-v7.5.0.56.tgz][save=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz][bpath=cudnn-10.1-linux-x64-v7.5.1.10.tgz][save=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz][bpath=cudnn-10.2-linux-x64-v7.6.5.32.tgz][save=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz][bpath=cudnn-8.0-linux-x64-v6.0.tgz][save=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz][bpath=cudnn-9.0-linux-x64-v7.1.tgz][save=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz][bpath=cudnn-9.1-linux-x64-v7.1.tgz][save=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz][bpath=cudnn-9.2-linux-x64-v7.1.tgz][save=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm][bpath=nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm][save=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm][bpath=nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm][save=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm][bpath=nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm][save=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [tpath=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm][bpath=nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm][save=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm][sseckeypos=-1]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm
Sep 02 09:44:01 build-test.lab s3fs[15005]:       [count=32]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/CUDA/] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-8-0.conf] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-10.1.243-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cuda_9.2.148_396.37_linux.run] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       computing signature [HEAD] [/dsw/1/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm] [] []
Sep 02 09:44:01 build-test.lab s3fs[15005]:       url is https://s3.amazonaws.com
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       HTTP response code 200
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/CUDA/]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-8-0.conf]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cuda_9.2.148_396.37_linux.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]:       add stat cache entry[path=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/CUDA]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-8-0.conf]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda_9.2.148_396.37_linux.run]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm]
Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm]
Sep 02 09:44:17 build-test.lab s3fs[15005]: [path=/]
Sep 02 09:44:17 build-test.lab s3fs[15005]: destroy
Sep 02 09:44:17 build-test.lab s3fs[15005]: s3fs.cpp:s3fs_destroy(3517): Failed to clean up signal object.

Details about issue

After a system reboot (also reproducible by umounting and remounting the bucket/folder), cannot directly access files hosted within an s3fs-based mount. Have to "prime" the mount, first (e.g., executing an ls /<S3FS>/<MOUNT>/<POINT> prior to trying to directly-access desired file(s)). The above logging output was produced having done:

# umount /dsw/tmp/installer/
# /usr/local/bin/s3fs -o allow_other -o umask=0000 -o nonempty -o dbglevel=info dev-build-tools:/dsw/1 /dsw/tmp/installer
# ls /dsw/tmp/installer/packages/nvidia/
ls: cannot access /dsw/tmp/installer/packages/nvidia/: Not a directory
# ls /dsw/tmp/installer/
packages
# ls /dsw/tmp/installer/packages/nvidia/
CUDA                                                                     cudnn-10.1-linux-x64-v7.5.1.10.tgz
cuda-10.1.243-1.x86_64.rpm                                               cudnn-10.2-linux-x64-v7.6.5.32.tgz
cuda-8-0.conf                                                            cudnn-8.0-linux-x64-v6.0.tgz
cuda_9.2.148_396.37_linux.run                                            cudnn-9.0-linux-x64-v7.1.tgz
cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm              cudnn-9.1-linux-x64-v7.1.tgz
cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm              cudnn-9.2-linux-x64-v7.1.tgz
cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm              nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm
cuda-repo-rhel7-10.2.89-1.x86_64.rpm                                     nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm
cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm            nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm
cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm  nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm
cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm                        NVIDIA-Linux-x86_64-375.51.run
cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm                            NVIDIA-Linux-x86_64-396.24.run
cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm             NVIDIA-Linux-x86_64-396.45.run
cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm   NVIDIA-Linux-x86_64-418.43.run
cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm   NVIDIA-Linux-x86_64-418.67.run
cudnn-10.0-linux-x64-v7.5.0.56.tgz                                       NVIDIA-Linux-x86_64-440.33.01.run
Originally created by @ferricoxide on GitHub (Sep 2, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1383 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) Started with: 1.85 (locally staged version) Updated to: 1.86 (from EPEL) Finally: 1.87 (compiled from source retrieved from GitHub) #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) fuse-2.9.2-11.el7.x86_64 #### Kernel information (uname -r) 3.10.0-1127.18.2.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Red Hat Enterprise Linux Server" VERSION="7.8 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.8" PRETTY_NAME="Red Hat Enterprise Linux" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.8" #### s3fs command line used, if applicable Not directly applicable: issues experienced with OS commands #### /etc/fstab entry, if applicable Not applicable: using mount from `/etc/rc.d/rc.local`: ~~~ /usr/local/bin/s3fs -o allow_other -o umask=0000 -o nonempty dev-build-tools:/dsw/1 /dsw/tmp/installer ~~~ #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ~~~ Sep 02 09:43:21 build-test.lab s3fs[15003]: sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [INF] Sep 02 09:43:21 build-test.lab s3fs[15003]: PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777) Sep 02 09:43:21 build-test.lab s3fs[15003]: Loaded mime information from /etc/mime.types Sep 02 09:43:21 build-test.lab s3fs[15003]: The path to cache top dir is empty, thus not need to check permission. Sep 02 09:43:21 build-test.lab s3fs[15005]: init v1.87(commit:unknown) with OpenSSL Sep 02 09:43:21 build-test.lab s3fs[15005]: check services. Sep 02 09:43:21 build-test.lab s3fs[15005]: check a bucket. Sep 02 09:43:21 build-test.lab s3fs[15005]: curl.cpp:ResetHandle(2150): The CURLOPT_SSL_ENABLE_ALPN option could not be unset. S3 server does not support ALPN, then this option should be disabled to maximize performance. you need to use libcurl 7.36.0 or later. Sep 02 09:43:21 build-test.lab s3fs[15005]: curl.cpp:ResetHandle(2153): The S3FS_CURLOPT_KEEP_SENDING_ON_ERROR option could not be set. For maximize performance you need to enable this option and you should use libcurl 7.51.0 or later. Sep 02 09:43:21 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/ Sep 02 09:43:21 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/ Sep 02 09:43:21 build-test.lab s3fs[15005]: computing signature [GET] [/dsw/1/] [] [] Sep 02 09:43:21 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:43:21 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:43:21 build-test.lab s3fs[15005]: Pool full: destroy the oldest handler Sep 02 09:43:21 build-test.lab s3fs[15005]: [path=/dsw/1] Sep 02 09:43:21 build-test.lab s3fs[15005]: s3fs.cpp:s3fs_init(3505): Failed to initialize signal object, but continue... Sep 02 09:43:35 build-test.lab s3fs[15005]: [path=/packages] Sep 02 09:43:35 build-test.lab s3fs[15005]: [tpath=/packages] Sep 02 09:43:35 build-test.lab s3fs[15005]: [tpath=/packages][bpath=][save=][sseckeypos=-1] Sep 02 09:43:35 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages Sep 02 09:43:35 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages Sep 02 09:43:35 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages] [] [] Sep 02 09:43:35 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:43:35 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:43:35 build-test.lab s3fs[15005]: add stat cache entry[path=/packages] Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/] Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/][flags=0x18800] Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/] Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/] Sep 02 09:43:50 build-test.lab s3fs[15005]: [tpath=/] Sep 02 09:43:50 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools?delimiter=/&max-keys=1000&prefix=dsw/1/ Sep 02 09:43:50 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dsw/1/ Sep 02 09:43:50 build-test.lab s3fs[15005]: computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dsw/1/] [] Sep 02 09:43:50 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:43:50 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/][list=0] Sep 02 09:43:50 build-test.lab s3fs[15005]: [tpath=/packages/][bpath=packages/][save=/packages/][sseckeypos=-1] Sep 02 09:43:50 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/ Sep 02 09:43:50 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/ Sep 02 09:43:50 build-test.lab s3fs[15005]: [count=1] Sep 02 09:43:50 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/] [] [] Sep 02 09:43:50 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:43:50 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:43:50 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/] Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/packages] Sep 02 09:43:50 build-test.lab s3fs[15005]: [path=/packages] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia] Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia] Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia][bpath=][save=][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 404 was returned, returning ENOENT Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/] Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/][bpath=][save=][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/ Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/ Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia][flags=0x18800] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia] Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools?delimiter=/&max-keys=1000&prefix=dsw/1/packages/nvidia/ Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dsw/1/packages/nvidia/ Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dsw/1/packages/nvidia/] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/][list=0] Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/CUDA/][bpath=CUDA/][save=/packages/nvidia/CUDA/][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/CUDA/ Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/CUDA/ Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run][bpath=NVIDIA-Linux-x86_64-375.51.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run][bpath=NVIDIA-Linux-x86_64-396.24.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run][bpath=NVIDIA-Linux-x86_64-396.45.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run][bpath=NVIDIA-Linux-x86_64-418.43.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run][bpath=NVIDIA-Linux-x86_64-418.67.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run][bpath=NVIDIA-Linux-x86_64-440.33.01.run][save=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm][bpath=cuda-10.1.243-1.x86_64.rpm][save=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-10.1.243-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-10.1.243-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-8-0.conf][bpath=cuda-8-0.conf][save=/packages/nvidia/cuda-8-0.conf][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-8-0.conf Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-8-0.conf Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm][bpath=cuda-repo-rhel7-10.2.89-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm][bpath=cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm][bpath=cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm][bpath=cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm][save=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cuda_9.2.148_396.37_linux.run][bpath=cuda_9.2.148_396.37_linux.run][save=/packages/nvidia/cuda_9.2.148_396.37_linux.run][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cuda_9.2.148_396.37_linux.run Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cuda_9.2.148_396.37_linux.run Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz][bpath=cudnn-10.0-linux-x64-v7.5.0.56.tgz][save=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz][bpath=cudnn-10.1-linux-x64-v7.5.1.10.tgz][save=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz][bpath=cudnn-10.2-linux-x64-v7.6.5.32.tgz][save=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz][bpath=cudnn-8.0-linux-x64-v6.0.tgz][save=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz][bpath=cudnn-9.0-linux-x64-v7.1.tgz][save=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz][bpath=cudnn-9.1-linux-x64-v7.1.tgz][save=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz][bpath=cudnn-9.2-linux-x64-v7.1.tgz][save=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm][bpath=nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm][save=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm][bpath=nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm][save=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm][bpath=nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm][save=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [tpath=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm][bpath=nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm][save=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm][sseckeypos=-1] Sep 02 09:44:01 build-test.lab s3fs[15005]: URL is https://s3.amazonaws.com/dev-build-tools/dsw/1/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: URL changed is https://dev-build-tools.s3.amazonaws.com/dsw/1/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm Sep 02 09:44:01 build-test.lab s3fs[15005]: [count=32] Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/CUDA/] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-8-0.conf] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-10.1.243-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cuda_9.2.148_396.37_linux.run] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: computing signature [HEAD] [/dsw/1/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm] [] [] Sep 02 09:44:01 build-test.lab s3fs[15005]: url is https://s3.amazonaws.com Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: HTTP response code 200 Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/CUDA/] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-8-0.conf] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cuda_9.2.148_396.37_linux.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: add stat cache entry[path=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/CUDA] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-375.51.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-396.24.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-396.45.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-418.43.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-418.67.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/NVIDIA-Linux-x86_64-440.33.01.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-10.1.243-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-8-0.conf] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-10.2.89-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cuda_9.2.148_396.37_linux.run] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-10.0-linux-x64-v7.5.0.56.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-10.1-linux-x64-v7.5.1.10.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-10.2-linux-x64-v7.6.5.32.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-8.0-linux-x64-v6.0.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-9.0-linux-x64-v7.1.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-9.1-linux-x64-v7.1.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/cudnn-9.2-linux-x64-v7.1.tgz] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm] Sep 02 09:44:01 build-test.lab s3fs[15005]: [path=/packages/nvidia/nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm] Sep 02 09:44:17 build-test.lab s3fs[15005]: [path=/] Sep 02 09:44:17 build-test.lab s3fs[15005]: destroy Sep 02 09:44:17 build-test.lab s3fs[15005]: s3fs.cpp:s3fs_destroy(3517): Failed to clean up signal object. ~~~ ### Details about issue After a system reboot (also reproducible by `umount`ing and re`mount`ing the bucket/folder), cannot directly access files hosted within an s3fs-based mount. Have to "prime" the mount, first (e.g., executing an `ls /<S3FS>/<MOUNT>/<POINT>` prior to trying to directly-access desired file(s)). The above logging output was produced having done: ~~~ # umount /dsw/tmp/installer/ # /usr/local/bin/s3fs -o allow_other -o umask=0000 -o nonempty -o dbglevel=info dev-build-tools:/dsw/1 /dsw/tmp/installer # ls /dsw/tmp/installer/packages/nvidia/ ls: cannot access /dsw/tmp/installer/packages/nvidia/: Not a directory # ls /dsw/tmp/installer/ packages # ls /dsw/tmp/installer/packages/nvidia/ CUDA cudnn-10.1-linux-x64-v7.5.1.10.tgz cuda-10.1.243-1.x86_64.rpm cudnn-10.2-linux-x64-v7.6.5.32.tgz cuda-8-0.conf cudnn-8.0-linux-x64-v6.0.tgz cuda_9.2.148_396.37_linux.run cudnn-9.0-linux-x64-v7.1.tgz cuda-repo-rhel7-10-0-local-10.0.130-410.48-1.0-1.x86_64.rpm cudnn-9.1-linux-x64-v7.1.tgz cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm cudnn-9.2-linux-x64-v7.1.tgz cuda-repo-rhel7-10-1-local-10.1.168-418.67-1.0-1.x86_64.rpm nccl-repo-rhel7-2.4.2-ga-cuda10.0-1-1.x86_64.rpm cuda-repo-rhel7-10.2.89-1.x86_64.rpm nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm cuda-repo-rhel7-10-2-local-10.2.89-440.33.01-1.0-1.x86_64.rpm nvidia-diag-driver-local-repo-rhel7-390.46-1.0-1.x86_64.rpm cuda-repo-rhel7-8-0-local-cublas-performance-update-8.0.61-1.x86_64.rpm nvidia-driver-local-repo-rhel7-375.51-1.x86_64.rpm cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm NVIDIA-Linux-x86_64-375.51.run cuda-repo-rhel7-9-1-local-9.1.85-1.x86_64.rpm NVIDIA-Linux-x86_64-396.24.run cuda-repo-rhel7-9-1-local-compiler-update-1-1.0-1.x86_64.rpm NVIDIA-Linux-x86_64-396.45.run cuda-repo-rhel7-9-1-local-cublas-performance-update-1-1.0-1.x86_64.rpm NVIDIA-Linux-x86_64-418.43.run cuda-repo-rhel7-9-1-local-cublas-performance-update-3-1.0-1.x86_64.rpm NVIDIA-Linux-x86_64-418.67.run cudnn-10.0-linux-x64-v7.5.0.56.tgz NVIDIA-Linux-x86_64-440.33.01.run ~~~
Author
Owner

@ferricoxide commented on GitHub (Sep 9, 2020):

In further messing around with the problem, I found:

Reset s3fs mount to “starting” state:

# umount /dsw/tmp/installer

# /usr/local/bin/s3fs  -o allow_other -o umask=0000 -o nonempty  dev-build-tools:/dsw/1 /dsw/tmp/installer

Check if can immediately reference target file:

# ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm
ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory

Try to use file command against “/dsw/tmp/installer/packages” to prime the mount:

# ( cd /dsw/tmp/installer && file /dsw/tmp/installer/packages )
/dsw/tmp/installer/packages: Zip archive data, at least v2.0 to extract

Check if can now directly-reference target file:

# ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm
ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory

Try to use ls command against “/dsw/tmp/installer/packages” (fully-qualified path) to prime the mount:

# ( cd /dsw/tmp/installer && ls /dsw/tmp/installer/packages )
/dsw/tmp/installer/packages

Check if can now directly-reference target file:

# ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm
ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory

Try to use ls command against “packages” (relative path) to prime the mount:

# ( cd /dsw/tmp/installer && ls packages )
packages

Check if can now directly-reference target file:

# ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm
ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory

Try to use ls command against directory’s root-inode (relative path) to prime the mount:

# ( cd /dsw/tmp/installer && ls . )
packages

Check if can now directly-reference target file:

# ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm
/dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm

Check to see what OS thinks the “/dsw/tmp/installer/packages” directory-object is:

# ( cd /dsw/tmp/installer && file /dsw/tmp/installer/packages )
/dsw/tmp/installer/packages: directory

Basically, before successfully "priming" the s3fs mount, the /dsw/tmp/installer/packages file-object gets misidentified as a ZIP file rather than a directory-inode.

<!-- gh-comment-id:689563747 --> @ferricoxide commented on GitHub (Sep 9, 2020): In further messing around with the problem, I found: **Reset s3fs mount to “starting” state:** ~~~ # umount /dsw/tmp/installer # /usr/local/bin/s3fs -o allow_other -o umask=0000 -o nonempty dev-build-tools:/dsw/1 /dsw/tmp/installer ~~~ **Check if can immediately reference target file:** ~~~ # ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory ~~~ **Try to use `file` command against “/dsw/tmp/installer/packages” to prime the mount:** ~~~ # ( cd /dsw/tmp/installer && file /dsw/tmp/installer/packages ) /dsw/tmp/installer/packages: Zip archive data, at least v2.0 to extract ~~~ **Check if can now directly-reference target file:** ~~~ # ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory ~~~ **Try to use `ls` command against “/dsw/tmp/installer/packages” (fully-qualified path) to prime the mount:** ~~~ # ( cd /dsw/tmp/installer && ls /dsw/tmp/installer/packages ) /dsw/tmp/installer/packages ~~~ **Check if can now directly-reference target file:** ~~~ # ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory ~~~ **Try to use `ls` command against “packages” (relative path) to prime the mount:** ~~~ # ( cd /dsw/tmp/installer && ls packages ) packages ~~~ **Check if can now directly-reference target file:** ~~~ # ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm ls: cannot access /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm: Not a directory ~~~ **Try to use `ls` command against directory’s root-inode (relative path) to prime the mount:** ~~~ # ( cd /dsw/tmp/installer && ls . ) packages ~~~ **Check if can now directly-reference target file:** ~~~ # ls /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm /dsw/tmp/installer/packages/rstudio/rstudio-1.2.5042-x86_64.rpm ~~~ **Check to see what OS thinks the “/dsw/tmp/installer/packages” directory-object is:** ~~~ # ( cd /dsw/tmp/installer && file /dsw/tmp/installer/packages ) /dsw/tmp/installer/packages: directory ~~~ Basically, before successfully "priming" the s3fs mount, the `/dsw/tmp/installer/packages` file-object gets misidentified as a ZIP file rather than a directory-inode.
Author
Owner

@WencongXiao commented on GitHub (Sep 18, 2020):

any update about this issue? Do you solve it? I also meet a similar problem.

<!-- gh-comment-id:694748484 --> @WencongXiao commented on GitHub (Sep 18, 2020): any update about this issue? Do you solve it? I also meet a similar problem.
Author
Owner

@gaul commented on GitHub (Oct 10, 2020):

I wonder if this is a timing issue, e.g., it takes s3fs a while to mount the path? If you wait 60 seconds can you see the files that you expect?

<!-- gh-comment-id:706512748 --> @gaul commented on GitHub (Oct 10, 2020): I wonder if this is a timing issue, e.g., it takes s3fs a while to mount the path? If you wait 60 seconds can you see the files that you expect?
Author
Owner

@ferricoxide commented on GitHub (Oct 12, 2020):

@WencongXiao

any update about this issue? Do you solve it? I also meet a similar problem.

No. And looks like no one on the project has processed this issue, yet.

@gaul

I wonder if this is a timing issue, e.g., it takes s3fs a while to mount the path? If you wait 60 seconds can you see the files that you expect?

Negative. The only way to get the files to show up is to traverse into the directory (as previously illustrated).

<!-- gh-comment-id:707228210 --> @ferricoxide commented on GitHub (Oct 12, 2020): @WencongXiao >any update about this issue? Do you solve it? I also meet a similar problem. No. And looks like no one on the project has processed this issue, yet. @gaul > I wonder if this is a timing issue, e.g., it takes s3fs a while to mount the path? If you wait 60 seconds can you see the files that you expect? Negative. The only way to get the files to show up is to traverse into the directory (as previously illustrated).
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#740
No description provided.