starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1890] s3fs 1.90 terminates with segfault #960

New issue
Closed
opened 2026-03-04 01:50:14 +03:00 by kerem · 8 comments
kerem commented 2026-03-04 01:50:14 +03:00
Owner
Copy link

Originally created by @nkkashyap on GitHub (Feb 15, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1890

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

Version: V1.90 (commit:cd466eb)

# s3fs --version
Amazon Simple Storage Service File System V1.90 (commit:cd466eb) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Fuse Version: 2.9.7-1ubuntu1

# dpkg -s fuse
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 104
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 2.9.7-1ubuntu1

Kernel information (uname -r)

Kernel: 4.15.0-163-generic

# uname -r
4.15.0-163-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

Distro: Ubuntu 18.04.6 LTS

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"

s3fs command line used, if applicable

s3fs mybucket /var/data/kubelet/pods/a8fe5d8c-5cd6-4e2b-b956-b59d7b563ded/volumes/ibm~ibmc-s3fs/pvc-7e92b5f4-dee4-4f1e-8423-5283afc80bce -o multireq_max=20 -o use_path_request_style -o passwd_file=/var/lib/ibmc-s3fs/43b90a2450d9588e731283cd4784e4b8e867727ac188367b9ac3930d56d0f58b/passwd -o url=https://s3.private.us-south.cloud-object-storage.appdomain.cloud -o endpoint=us-south-standard -o parallel_count=20 -o multipart_size=52 -o dbglevel=debug -o max_stat_cache_size=100000 -o allow_other -o max_background=1000 -o mp_umask=002 -o cipher_suites=AESGCM -o retries=5 -o curldbg -o kernel_cache -o ibm_iam_auth -o ibm_iam_endpoint=https://iam.cloud.ibm.com

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Feb 10 07:22:10 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 30
Feb 10 07:22:10 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png
Feb 10 07:22:10 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa80401a8a0 [can pipeline]
Feb 10 07:22:10 00000c4a s3fs[214750]: * Re-using existing connection! (#1) with host s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:10 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#1)
Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png HTTP/1.1
Feb 10 07:22:10 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:10 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL)
Feb 10 07:22:10 00000c4a s3fs[214750]: > Accept: */*
Feb 10 07:22:10 00000c4a s3fs[214750]: > Authorization: Bearer xxxx
Feb 10 07:22:10 00000c4a s3fs[214750]: >
Feb 10 07:22:10 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails]
Feb 10 07:22:10 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=100]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:10 00000c4a s3fs[214750]:       delete stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:10 00000c4a s3fs[214750]:       [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:10 00000c4a s3fs[214750]:       [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][bpath=][save=][sseckeypos=18446744073709551615]
Feb 10 07:22:10 00000c4a s3fs[214750]:       URL is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:10 00000c4a s3fs[214750]:       URL changed is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:10 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 29
Feb 10 07:22:10 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:10 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa7b8001bf0 [can pipeline]
Feb 10 07:22:10 00000c4a s3fs[214750]: * Re-using existing connection! (#0) with host s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:10 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#0)
Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1
Feb 10 07:22:10 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:10 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL)
Feb 10 07:22:10 00000c4a s3fs[214750]: > Accept: */*
Feb 10 07:22:10 00000c4a s3fs[214750]: > Authorization: Bearer xxxx
Feb 10 07:22:10 00000c4a s3fs[214750]: >
Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:11 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:11 GMT
Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-Request-Id: cfb63826-7137-4341-b366-f087a6e48011
Feb 10 07:22:11 00000c4a s3fs[214750]: < Server: Cleversafe
Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5
Feb 10 07:22:11 00000c4a s3fs[214750]: < Accept-Ranges: bytes
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-request-id: cfb63826-7137-4341-b366-f087a6e48011
Feb 10 07:22:11 00000c4a s3fs[214750]: < ETag: "ef00bd95c441461ee53d26978b200b8a"
Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Type: image/png
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mode: 33188
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-ctime: 1625547380
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-gid: 0
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-uid: 0
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-atime: 1625547378
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mtime: 1625547378
Feb 10 07:22:11 00000c4a s3fs[214750]: < Last-Modified: Tue, 06 Jul 2021 04:56:20 GMT
Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Length: 5738
Feb 10 07:22:11 00000c4a s3fs[214750]: <
Feb 10 07:22:11 00000c4a s3fs[214750]: * Connection #0 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact
Feb 10 07:22:11 00000c4a s3fs[214750]:       HTTP response code 200
Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
Feb 10 07:22:11 00000c4a s3fs[214750]:       add stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.247051549][hit count=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:OpenExistFdEntity(628): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][flags=0x0]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Open(526): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=-1][time=-1][flags=0x0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:s3fs_getattr(801): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] uid=0, gid=0, mode=100644
Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][mask=F_OK]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.247051549][hit count=1]
Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][flags=0x8000]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.247051549][hit count=2]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=-1]
Feb 10 07:22:11 00000c4a s3fs[214750]:       delete stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=101]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][bpath=][save=][sseckeypos=18446744073709551615]
Feb 10 07:22:11 00000c4a s3fs[214750]:       URL is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:11 00000c4a s3fs[214750]:       URL changed is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 29
Feb 10 07:22:11 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:11 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa7b8001bf0 [can pipeline]
Feb 10 07:22:11 00000c4a s3fs[214750]: * Re-using existing connection! (#0) with host s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:11 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#0)
Feb 10 07:22:11 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1
Feb 10 07:22:11 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:11 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL)
Feb 10 07:22:11 00000c4a s3fs[214750]: > Accept: */*
Feb 10 07:22:11 00000c4a s3fs[214750]: > Authorization: Bearer xxxx
Feb 10 07:22:11 00000c4a s3fs[214750]: >
Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:11 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:11 GMT
Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-Request-Id: 06381d7c-0d13-4827-b06f-31634e74ebbe
Feb 10 07:22:11 00000c4a s3fs[214750]: < Server: Cleversafe
Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5
Feb 10 07:22:11 00000c4a s3fs[214750]: < Accept-Ranges: bytes
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-request-id: 06381d7c-0d13-4827-b06f-31634e74ebbe
Feb 10 07:22:11 00000c4a s3fs[214750]: < ETag: "ef00bd95c441461ee53d26978b200b8a"
Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Type: image/png
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mode: 33188
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-ctime: 1625547380
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-gid: 0
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-uid: 0
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-atime: 1625547378
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mtime: 1625547378
Feb 10 07:22:11 00000c4a s3fs[214750]: < Last-Modified: Tue, 06 Jul 2021 04:56:20 GMT
Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Length: 5738
Feb 10 07:22:11 00000c4a s3fs[214750]: <
Feb 10 07:22:11 00000c4a s3fs[214750]: * Connection #0 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact
Feb 10 07:22:11 00000c4a s3fs[214750]:       HTTP response code 200
Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
Feb 10 07:22:11 00000c4a s3fs[214750]:       add stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=1]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Open(526): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=5738][time=1625547378][flags=0x8000]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Open(411): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=-1][size=5738][time=1625547378][flags=0x8000]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=17][mtime=1625547378][ctime=1625547378]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:s3fs_read(2325): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=8192][offset=0][pseudo_fd=2]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:GetExistFdEntity(611): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Read(1698): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2][physical_fd=17][offset=0][size=8192]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Load(1001): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=17][offset=0][size=8192]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][start=0][size=5738]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=2]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][start=0][size=5738]
Feb 10 07:22:11 00000c4a s3fs[214750]:       URL is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:11 00000c4a s3fs[214750]:       URL changed is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 29
Feb 10 07:22:11 00000c4a s3fs[214750]:       downloading... [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][fd=17]
Feb 10 07:22:11 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png
Feb 10 07:22:11 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa7b8001bf0 [can pipeline]
Feb 10 07:22:11 00000c4a s3fs[214750]: * Re-using existing connection! (#0) with host s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:11 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#0)
Feb 10 07:22:11 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1
Feb 10 07:22:11 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud
Feb 10 07:22:11 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL)
Feb 10 07:22:11 00000c4a s3fs[214750]: > Accept: */*
Feb 10 07:22:11 00000c4a s3fs[214750]: > Authorization: Bearer xxxx
Feb 10 07:22:11 00000c4a s3fs[214750]: > Range: bytes=0-5737
Feb 10 07:22:11 00000c4a s3fs[214750]: >
Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content
Feb 10 07:22:11 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:11 GMT
Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-Request-Id: 2366fc68-3a4a-47e2-a7ee-dcacad56fcb1
Feb 10 07:22:11 00000c4a s3fs[214750]: < Server: Cleversafe
Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5
Feb 10 07:22:11 00000c4a s3fs[214750]: < Accept-Ranges: bytes
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-request-id: 2366fc68-3a4a-47e2-a7ee-dcacad56fcb1
Feb 10 07:22:11 00000c4a s3fs[214750]: < ETag: "ef00bd95c441461ee53d26978b200b8a"
Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Type: image/png
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mode: 33188
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-ctime: 1625547380
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-gid: 0
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-uid: 0
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-atime: 1625547378
Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mtime: 1625547378
Feb 10 07:22:11 00000c4a s3fs[214750]: < Last-Modified: Tue, 06 Jul 2021 04:56:20 GMT
Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Range: bytes 0-5737/5738
Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Length: 5738
Feb 10 07:22:11 00000c4a s3fs[214750]: <
Feb 10 07:22:11 00000c4a s3fs[214750]: * Connection #0 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact
Feb 10 07:22:11 00000c4a s3fs[214750]:       HTTP response code 206
Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=102]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=3]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:OpenExistFdEntity(628): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][flags=0x0]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Open(526): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=-1][time=-1][flags=0x0]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Open(411): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=17][size=-1][time=-1][flags=0x0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:s3fs_getattr(801): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] uid=0, gid=0, mode=100644
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Close(680): [ent->file=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=3]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Close(207): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=3][physical_fd=17]
Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=103]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0]
Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=4]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:GetExistFdEntity(611): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [tpath=][path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2][physical_fd=17]
Feb 10 07:22:11 00000c4a s3fs[214750]:       delete stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png]
Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Close(680): [ent->file=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2]
Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Close(207): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2][physical_fd=17]
Feb 10 07:22:11 00000c4a s3fs[214750]:       [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=-1]
Feb 10 07:22:21 00000c4a s3fs[214750]: < HTTP/1.1 500 Server Error
Feb 10 07:22:21 00000c4a s3fs[214750]: < X-Clv-Request-Id: ead30edf-b0d1-465f-a12b-f9de878835c8
Feb 10 07:22:21 00000c4a s3fs[214750]: < Server: Cleversafe
Feb 10 07:22:21 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5
Feb 10 07:22:21 00000c4a s3fs[214750]: < Accept-Ranges: bytes
Feb 10 07:22:21 00000c4a s3fs[214750]: < x-amz-request-id: ead30edf-b0d1-465f-a12b-f9de878835c8
Feb 10 07:22:21 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:10 GMT
Feb 10 07:22:21 00000c4a s3fs[214750]: < Content-Type: application/xml
Feb 10 07:22:21 00000c4a s3fs[214750]: < Content-Length: 361
Feb 10 07:22:21 00000c4a s3fs[214750]: <
Feb 10 07:22:21 00000c4a s3fs[214750]: * Connection #1 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact
Feb 10 07:22:21 00000c4a s3fs[214750]:       HTTP response code 500 was returned, slowing down
Feb 10 07:22:21 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2387): Body Text:
Feb 10 07:22:23 00000c4a s3fs[214750]: ### retrying...
Feb 10 07:22:23 00000c4a s3fs[214750]:       Retry request. [type=1][url=https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png][path=/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png]
Feb 10 07:22:23 00000c4a kernel: [3206883.738176] s3fs[214751]: segfault at 8 ip 000055db1b6451ed sp 00007fa809bd81f0 error 4 in s3fs[55db1b5e8000+c7000]

Details about issue

This issue is occurring in IBM Cloud Kubernetes Service.
With the help of K8S FlexVolume driver ibmc-s3fs, the bucket is mounted on a worker node and then bind mounted to POD.
S3FS is getting segfault when it is retrying post 500 Server Error

Feb 10 07:13:00 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/639f6b01-8b32-4a73-8d5c-d4929823b469.png HTTP/1.1
Feb 10 07:13:00 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:13:00 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/639f6b01-8b32-4a73-8d5c-d4929823b469.png HTTP/1.1
Feb 10 07:13:01 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content
Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/df72fbdf-812d-4c4e-bb6d-14717bce80b4.png HTTP/1.1
Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b33e1b76-a688-4ab3-bfc7-cae105023825.png HTTP/1.1
Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/c89e2464-ec38-43c1-b09f-4b257c09a1b2.png HTTP/1.1
Feb 10 07:22:04 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/df72fbdf-812d-4c4e-bb6d-14717bce80b4.png HTTP/1.1
Feb 10 07:22:04 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/c89e2464-ec38-43c1-b09f-4b257c09a1b2.png HTTP/1.1
Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/0587c656-4c3f-4fbf-b4b1-f4ce7100dd8e.png HTTP/1.1
Feb 10 07:22:04 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/0587c656-4c3f-4fbf-b4b1-f4ce7100dd8e.png HTTP/1.1
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/0587c656-4c3f-4fbf-b4b1-f4ce7100dd8e.png HTTP/1.1
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content
Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/c89e2464-ec38-43c1-b09f-4b257c09a1b2.png HTTP/1.1
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:05 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b33e1b76-a688-4ab3-bfc7-cae105023825.png HTTP/1.1
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/b33e1b76-a688-4ab3-bfc7-cae105023825.png HTTP/1.1
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content
Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/df72fbdf-812d-4c4e-bb6d-14717bce80b4.png HTTP/1.1
Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content
Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png HTTP/1.1
Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1
Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:11 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1
Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK
Feb 10 07:22:11 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1
Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content
Feb 10 07:22:21 00000c4a s3fs[214750]: < HTTP/1.1 500 Server Error
Originally created by @nkkashyap on GitHub (Feb 15, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1890 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) _Version: V1.90 (commit:cd466eb)_ ``` # s3fs --version Amazon Simple Storage Service File System V1.90 (commit:cd466eb) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) _Fuse Version: 2.9.7-1ubuntu1_ ``` # dpkg -s fuse Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 104 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Architecture: amd64 Version: 2.9.7-1ubuntu1 ``` #### Kernel information (uname -r) _Kernel: 4.15.0-163-generic_ ``` # uname -r 4.15.0-163-generic ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) _Distro: Ubuntu 18.04.6 LTS_ ``` # cat /etc/os-release NAME="Ubuntu" VERSION="18.04.6 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.6 LTS" VERSION_ID="18.04" ``` #### s3fs command line used, if applicable ``` s3fs mybucket /var/data/kubelet/pods/a8fe5d8c-5cd6-4e2b-b956-b59d7b563ded/volumes/ibm~ibmc-s3fs/pvc-7e92b5f4-dee4-4f1e-8423-5283afc80bce -o multireq_max=20 -o use_path_request_style -o passwd_file=/var/lib/ibmc-s3fs/43b90a2450d9588e731283cd4784e4b8e867727ac188367b9ac3930d56d0f58b/passwd -o url=https://s3.private.us-south.cloud-object-storage.appdomain.cloud -o endpoint=us-south-standard -o parallel_count=20 -o multipart_size=52 -o dbglevel=debug -o max_stat_cache_size=100000 -o allow_other -o max_background=1000 -o mp_umask=002 -o cipher_suites=AESGCM -o retries=5 -o curldbg -o kernel_cache -o ibm_iam_auth -o ibm_iam_endpoint=https://iam.cloud.ibm.com ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` Feb 10 07:22:10 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 30 Feb 10 07:22:10 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png Feb 10 07:22:10 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa80401a8a0 [can pipeline] Feb 10 07:22:10 00000c4a s3fs[214750]: * Re-using existing connection! (#1) with host s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:10 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#1) Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png HTTP/1.1 Feb 10 07:22:10 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:10 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL) Feb 10 07:22:10 00000c4a s3fs[214750]: > Accept: */* Feb 10 07:22:10 00000c4a s3fs[214750]: > Authorization: Bearer xxxx Feb 10 07:22:10 00000c4a s3fs[214750]: > Feb 10 07:22:10 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails] Feb 10 07:22:10 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=100] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:10 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:10 00000c4a s3fs[214750]: delete stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:10 00000c4a s3fs[214750]: [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:10 00000c4a s3fs[214750]: [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][bpath=][save=][sseckeypos=18446744073709551615] Feb 10 07:22:10 00000c4a s3fs[214750]: URL is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:10 00000c4a s3fs[214750]: URL changed is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:10 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 29 Feb 10 07:22:10 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:10 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa7b8001bf0 [can pipeline] Feb 10 07:22:10 00000c4a s3fs[214750]: * Re-using existing connection! (#0) with host s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:10 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#0) Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1 Feb 10 07:22:10 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:10 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL) Feb 10 07:22:10 00000c4a s3fs[214750]: > Accept: */* Feb 10 07:22:10 00000c4a s3fs[214750]: > Authorization: Bearer xxxx Feb 10 07:22:10 00000c4a s3fs[214750]: > Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:11 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:11 GMT Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-Request-Id: cfb63826-7137-4341-b366-f087a6e48011 Feb 10 07:22:11 00000c4a s3fs[214750]: < Server: Cleversafe Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5 Feb 10 07:22:11 00000c4a s3fs[214750]: < Accept-Ranges: bytes Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-request-id: cfb63826-7137-4341-b366-f087a6e48011 Feb 10 07:22:11 00000c4a s3fs[214750]: < ETag: "ef00bd95c441461ee53d26978b200b8a" Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Type: image/png Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mode: 33188 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-ctime: 1625547380 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-gid: 0 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-uid: 0 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-atime: 1625547378 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mtime: 1625547378 Feb 10 07:22:11 00000c4a s3fs[214750]: < Last-Modified: Tue, 06 Jul 2021 04:56:20 GMT Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Length: 5738 Feb 10 07:22:11 00000c4a s3fs[214750]: < Feb 10 07:22:11 00000c4a s3fs[214750]: * Connection #0 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact Feb 10 07:22:11 00000c4a s3fs[214750]: HTTP response code 200 Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool Feb 10 07:22:11 00000c4a s3fs[214750]: add stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.247051549][hit count=0] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:OpenExistFdEntity(628): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][flags=0x0] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Open(526): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=-1][time=-1][flags=0x0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:s3fs_getattr(801): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] uid=0, gid=0, mode=100644 Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][mask=F_OK] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.247051549][hit count=1] Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][flags=0x8000] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.247051549][hit count=2] Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=-1] Feb 10 07:22:11 00000c4a s3fs[214750]: delete stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=101] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][bpath=][save=][sseckeypos=18446744073709551615] Feb 10 07:22:11 00000c4a s3fs[214750]: URL is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:11 00000c4a s3fs[214750]: URL changed is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 29 Feb 10 07:22:11 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:11 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa7b8001bf0 [can pipeline] Feb 10 07:22:11 00000c4a s3fs[214750]: * Re-using existing connection! (#0) with host s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:11 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#0) Feb 10 07:22:11 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1 Feb 10 07:22:11 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:11 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL) Feb 10 07:22:11 00000c4a s3fs[214750]: > Accept: */* Feb 10 07:22:11 00000c4a s3fs[214750]: > Authorization: Bearer xxxx Feb 10 07:22:11 00000c4a s3fs[214750]: > Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:11 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:11 GMT Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-Request-Id: 06381d7c-0d13-4827-b06f-31634e74ebbe Feb 10 07:22:11 00000c4a s3fs[214750]: < Server: Cleversafe Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5 Feb 10 07:22:11 00000c4a s3fs[214750]: < Accept-Ranges: bytes Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-request-id: 06381d7c-0d13-4827-b06f-31634e74ebbe Feb 10 07:22:11 00000c4a s3fs[214750]: < ETag: "ef00bd95c441461ee53d26978b200b8a" Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Type: image/png Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mode: 33188 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-ctime: 1625547380 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-gid: 0 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-uid: 0 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-atime: 1625547378 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mtime: 1625547378 Feb 10 07:22:11 00000c4a s3fs[214750]: < Last-Modified: Tue, 06 Jul 2021 04:56:20 GMT Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Length: 5738 Feb 10 07:22:11 00000c4a s3fs[214750]: < Feb 10 07:22:11 00000c4a s3fs[214750]: * Connection #0 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact Feb 10 07:22:11 00000c4a s3fs[214750]: HTTP response code 200 Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool Feb 10 07:22:11 00000c4a s3fs[214750]: add stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=1] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Open(526): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=5738][time=1625547378][flags=0x8000] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Open(411): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=-1][size=5738][time=1625547378][flags=0x8000] Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=17][mtime=1625547378][ctime=1625547378] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:s3fs_read(2325): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=8192][offset=0][pseudo_fd=2] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:GetExistFdEntity(611): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Read(1698): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2][physical_fd=17][offset=0][size=8192] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Load(1001): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=17][offset=0][size=8192] Feb 10 07:22:11 00000c4a s3fs[214750]: [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][start=0][size=5738] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=2] Feb 10 07:22:11 00000c4a s3fs[214750]: [tpath=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][start=0][size=5738] Feb 10 07:22:11 00000c4a s3fs[214750]: URL is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:11 00000c4a s3fs[214750]: URL changed is https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 29 Feb 10 07:22:11 00000c4a s3fs[214750]: downloading... [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][fd=17] Feb 10 07:22:11 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2283): connecting to URL https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png Feb 10 07:22:11 00000c4a s3fs[214750]: * Found bundle for host s3.private.us-south.cloud-object-storage.appdomain.cloud: 0x7fa7b8001bf0 [can pipeline] Feb 10 07:22:11 00000c4a s3fs[214750]: * Re-using existing connection! (#0) with host s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:11 00000c4a s3fs[214750]: * Connected to s3.private.us-south.cloud-object-storage.appdomain.cloud (10.1.129.97) port 443 (#0) Feb 10 07:22:11 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1 Feb 10 07:22:11 00000c4a s3fs[214750]: > Host: s3.private.us-south.cloud-object-storage.appdomain.cloud Feb 10 07:22:11 00000c4a s3fs[214750]: > User-Agent: s3fs/1.90 (commit hash cd466eb; OpenSSL) Feb 10 07:22:11 00000c4a s3fs[214750]: > Accept: */* Feb 10 07:22:11 00000c4a s3fs[214750]: > Authorization: Bearer xxxx Feb 10 07:22:11 00000c4a s3fs[214750]: > Range: bytes=0-5737 Feb 10 07:22:11 00000c4a s3fs[214750]: > Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content Feb 10 07:22:11 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:11 GMT Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-Request-Id: 2366fc68-3a4a-47e2-a7ee-dcacad56fcb1 Feb 10 07:22:11 00000c4a s3fs[214750]: < Server: Cleversafe Feb 10 07:22:11 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5 Feb 10 07:22:11 00000c4a s3fs[214750]: < Accept-Ranges: bytes Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-request-id: 2366fc68-3a4a-47e2-a7ee-dcacad56fcb1 Feb 10 07:22:11 00000c4a s3fs[214750]: < ETag: "ef00bd95c441461ee53d26978b200b8a" Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Type: image/png Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mode: 33188 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-ctime: 1625547380 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-gid: 0 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-uid: 0 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-atime: 1625547378 Feb 10 07:22:11 00000c4a s3fs[214750]: < x-amz-meta-mtime: 1625547378 Feb 10 07:22:11 00000c4a s3fs[214750]: < Last-Modified: Tue, 06 Jul 2021 04:56:20 GMT Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Range: bytes 0-5737/5738 Feb 10 07:22:11 00000c4a s3fs[214750]: < Content-Length: 5738 Feb 10 07:22:11 00000c4a s3fs[214750]: < Feb 10 07:22:11 00000c4a s3fs[214750]: * Connection #0 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact Feb 10 07:22:11 00000c4a s3fs[214750]: HTTP response code 206 Feb 10 07:22:11 00000c4a s3fs[214750]: curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=102] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=3] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:OpenExistFdEntity(628): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][flags=0x0] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Open(526): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][size=-1][time=-1][flags=0x0] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Open(411): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][physical_fd=17][size=-1][time=-1][flags=0x0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:s3fs_getattr(801): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] uid=0, gid=0, mode=100644 Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Close(680): [ent->file=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=3] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Close(207): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=3][physical_fd=17] Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_parent_object_access(642): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/][time=3206301.951216235][hit count=103] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(534): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:check_object_access(539): [pid=222113,uid=0,gid=0] Feb 10 07:22:11 00000c4a s3fs[214750]: s3fs.cpp:get_object_attribute(365): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: cache.cpp:GetStat(295): stat cache hit [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][time=3206871.547051883][hit count=4] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:GetExistFdEntity(611): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2] Feb 10 07:22:11 00000c4a s3fs[214750]: [tpath=][path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2][physical_fd=17] Feb 10 07:22:11 00000c4a s3fs[214750]: delete stat cache entry[path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png] Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2] Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache.cpp:Close(680): [ent->file=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2] Feb 10 07:22:11 00000c4a s3fs[214750]: fdcache_entity.cpp:Close(207): [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=2][physical_fd=17] Feb 10 07:22:11 00000c4a s3fs[214750]: [path=/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png][pseudo_fd=-1] Feb 10 07:22:21 00000c4a s3fs[214750]: < HTTP/1.1 500 Server Error Feb 10 07:22:21 00000c4a s3fs[214750]: < X-Clv-Request-Id: ead30edf-b0d1-465f-a12b-f9de878835c8 Feb 10 07:22:21 00000c4a s3fs[214750]: < Server: Cleversafe Feb 10 07:22:21 00000c4a s3fs[214750]: < X-Clv-S3-Version: 2.5 Feb 10 07:22:21 00000c4a s3fs[214750]: < Accept-Ranges: bytes Feb 10 07:22:21 00000c4a s3fs[214750]: < x-amz-request-id: ead30edf-b0d1-465f-a12b-f9de878835c8 Feb 10 07:22:21 00000c4a s3fs[214750]: < Date: Thu, 10 Feb 2022 07:22:10 GMT Feb 10 07:22:21 00000c4a s3fs[214750]: < Content-Type: application/xml Feb 10 07:22:21 00000c4a s3fs[214750]: < Content-Length: 361 Feb 10 07:22:21 00000c4a s3fs[214750]: < Feb 10 07:22:21 00000c4a s3fs[214750]: * Connection #1 to host s3.private.us-south.cloud-object-storage.appdomain.cloud left intact Feb 10 07:22:21 00000c4a s3fs[214750]: HTTP response code 500 was returned, slowing down Feb 10 07:22:21 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2387): Body Text: Feb 10 07:22:23 00000c4a s3fs[214750]: ### retrying... Feb 10 07:22:23 00000c4a s3fs[214750]: Retry request. [type=1][url=https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png][path=/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png] Feb 10 07:22:23 00000c4a kernel: [3206883.738176] s3fs[214751]: segfault at 8 ip 000055db1b6451ed sp 00007fa809bd81f0 error 4 in s3fs[55db1b5e8000+c7000] ``` ### Details about issue This issue is occurring in IBM Cloud Kubernetes Service. With the help of K8S FlexVolume driver `ibmc-s3fs`, the bucket is mounted on a worker node and then bind mounted to POD. S3FS is getting `segfault` when it is retrying post `500 Server Error` ``` Feb 10 07:13:00 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/639f6b01-8b32-4a73-8d5c-d4929823b469.png HTTP/1.1 Feb 10 07:13:00 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:13:00 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/639f6b01-8b32-4a73-8d5c-d4929823b469.png HTTP/1.1 Feb 10 07:13:01 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/df72fbdf-812d-4c4e-bb6d-14717bce80b4.png HTTP/1.1 Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b33e1b76-a688-4ab3-bfc7-cae105023825.png HTTP/1.1 Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/c89e2464-ec38-43c1-b09f-4b257c09a1b2.png HTTP/1.1 Feb 10 07:22:04 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/df72fbdf-812d-4c4e-bb6d-14717bce80b4.png HTTP/1.1 Feb 10 07:22:04 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/c89e2464-ec38-43c1-b09f-4b257c09a1b2.png HTTP/1.1 Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/0587c656-4c3f-4fbf-b4b1-f4ce7100dd8e.png HTTP/1.1 Feb 10 07:22:04 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:04 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/0587c656-4c3f-4fbf-b4b1-f4ce7100dd8e.png HTTP/1.1 Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/0587c656-4c3f-4fbf-b4b1-f4ce7100dd8e.png HTTP/1.1 Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/c89e2464-ec38-43c1-b09f-4b257c09a1b2.png HTTP/1.1 Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:05 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b33e1b76-a688-4ab3-bfc7-cae105023825.png HTTP/1.1 Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/b33e1b76-a688-4ab3-bfc7-cae105023825.png HTTP/1.1 Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content Feb 10 07:22:05 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/df72fbdf-812d-4c4e-bb6d-14717bce80b4.png HTTP/1.1 Feb 10 07:22:05 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png HTTP/1.1 Feb 10 07:22:10 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1 Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:11 00000c4a s3fs[214750]: > HEAD /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1 Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 200 OK Feb 10 07:22:11 00000c4a s3fs[214750]: > GET /monolith-app-build-encrypt/thumbnails/78fa46a0-a89c-4845-8626-16da171d9adc.png HTTP/1.1 Feb 10 07:22:11 00000c4a s3fs[214750]: < HTTP/1.1 206 Partial Content Feb 10 07:22:21 00000c4a s3fs[214750]: < HTTP/1.1 500 Server Error ```
kerem 2026-03-04 01:50:14 +03:00
  • closed this issue
  • added the
    need info
         label
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Feb 15, 2022):

Could you try running s3fs under valgrind? This should reveal the cause of the segfault. Alternatively you could try using AddressSanitizer:

./configure CXX=clang++ CXXFLAGS="$COMMON_FLAGS -fsanitize=address -fsanitize-address-use-after-scope"
make
ASAN_OPTIONS='detect_leaks=1,detect_stack_use_after_return=1' s3fs $YOUR_OPTIONS_HERE
<!-- gh-comment-id:1040214022 --> @gaul commented on GitHub (Feb 15, 2022): Could you try running s3fs under `valgrind`? This should reveal the cause of the segfault. Alternatively you could try using AddressSanitizer: ``` ./configure CXX=clang++ CXXFLAGS="$COMMON_FLAGS -fsanitize=address -fsanitize-address-use-after-scope" make ASAN_OPTIONS='detect_leaks=1,detect_stack_use_after_return=1' s3fs $YOUR_OPTIONS_HERE ```
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 23, 2022):

@nkkashyap
I want a little more information such as debug logs, too(as @gaul said).
And there are some things I would like to know below.

  • Can this phenomenon be reproduced?
  • If it can be reproduced, how often does it occur? (Is it running for a while? or immediately after startup?)
  • It seems that you are using IBM, but does it coincide with the timing of IBM's token refresh?
<!-- gh-comment-id:1049343896 --> @ggtakec commented on GitHub (Feb 23, 2022): @nkkashyap I want a little more information such as debug logs, too(as @gaul said). And there are some things I would like to know below. - Can this phenomenon be reproduced? - If it can be reproduced, how often does it occur? (Is it running for a while? or immediately after startup?) - It seems that you are using IBM, but does it coincide with the timing of IBM's token refresh?
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Mar 5, 2022):

@nkkashyap
#1917 has been merged.
If the cause of this issue is the same as #1917, you can solve it using the master branch.
If you can, please try to use master branch and we want to know the result.
Thanks in advance for your kindness.

<!-- gh-comment-id:1059636344 --> @ggtakec commented on GitHub (Mar 5, 2022): @nkkashyap #1917 has been merged. If the cause of this issue is the same as #1917, you can solve it using the master branch. If you can, please try to use master branch and we want to know the result. Thanks in advance for your kindness.
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@nkkashyap commented on GitHub (Mar 18, 2022):

Hi @ggtakec @gaul
Sorry for the delay.
This issue is occurring very frequently in the Customer's cluster. Unfortunately I am not able to recreate the issue in my test environment.
Finally, I managed to get the AddressSanitizer trace with Customer's help

=================================================================
==295096==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x55e87648884c bp 0x7fbf6e9f3e30 sp 0x7fbf6e9f3c50 T1)
==295096==The signal is caused by a READ memory access.
==295096==Hint: address points to the zero page.
    #0 0x55e87648884b in curl_slist_remove(curl_slist*, char const*) (/usr/local/bin/s3fstest+0x10684b)
    #1 0x55e876445ce1 in S3fsCurl::RemakeHandle() (/usr/local/bin/s3fstest+0xc3ce1)
    #2 0x55e87644d049 in S3fsCurl::RequestPerform(bool) (/usr/local/bin/s3fstest+0xcb049)
    #3 0x55e876457f9d in S3fsCurl::HeadRequest(char const*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, header_nocase_cmp, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) (/usr/local/bin/s3fstest+0xd5f9d)
    #4 0x55e8763c0225 in get_object_attribute(char const*, stat*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, header_nocase_cmp, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*, bool, bool*, bool) (/usr/local/bin/s3fstest+0x3e225)
    #5 0x55e8763c1a06 in check_object_access(char const*, int, stat*) (/usr/local/bin/s3fstest+0x3fa06)
    #6 0x55e8763df48c in s3fs_open(char const*, fuse_file_info*) (/usr/local/bin/s3fstest+0x5d48c)
    #7 0x7fbf7b1d7a3f in fuse_fs_open (/lib/x86_64-linux-gnu/libfuse.so.2+0xba3f)
    #8 0x7fbf7b1d7b21  (/lib/x86_64-linux-gnu/libfuse.so.2+0xbb21)
    #9 0x7fbf7b1e1f9b  (/lib/x86_64-linux-gnu/libfuse.so.2+0x15f9b)
    #10 0x7fbf7b1e16c0  (/lib/x86_64-linux-gnu/libfuse.so.2+0x156c0)
    #11 0x7fbf7b1dde67  (/lib/x86_64-linux-gnu/libfuse.so.2+0x11e67)
    #12 0x7fbf7a11d6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #13 0x7fbf79e4671e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/local/bin/s3fstest+0x10684b) in curl_slist_remove(curl_slist*, char const*)
Thread T1 created by T0 here:
    #0 0x7fbf7b43fd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x7fbf7b1ddbb4  (/lib/x86_64-linux-gnu/libfuse.so.2+0x11bb4)

==295096==ABORTING

Somehow with clang not able to stack details. I used gcc (g++) compiler.

<!-- gh-comment-id:1072530565 --> @nkkashyap commented on GitHub (Mar 18, 2022): Hi @ggtakec @gaul Sorry for the delay. This issue is occurring very frequently in the Customer's cluster. Unfortunately I am not able to recreate the issue in my test environment. Finally, I managed to get the `AddressSanitizer` trace with Customer's help ``` ================================================================= ==295096==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0x55e87648884c bp 0x7fbf6e9f3e30 sp 0x7fbf6e9f3c50 T1) ==295096==The signal is caused by a READ memory access. ==295096==Hint: address points to the zero page. #0 0x55e87648884b in curl_slist_remove(curl_slist*, char const*) (/usr/local/bin/s3fstest+0x10684b) #1 0x55e876445ce1 in S3fsCurl::RemakeHandle() (/usr/local/bin/s3fstest+0xc3ce1) #2 0x55e87644d049 in S3fsCurl::RequestPerform(bool) (/usr/local/bin/s3fstest+0xcb049) #3 0x55e876457f9d in S3fsCurl::HeadRequest(char const*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, header_nocase_cmp, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) (/usr/local/bin/s3fstest+0xd5f9d) #4 0x55e8763c0225 in get_object_attribute(char const*, stat*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, header_nocase_cmp, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*, bool, bool*, bool) (/usr/local/bin/s3fstest+0x3e225) #5 0x55e8763c1a06 in check_object_access(char const*, int, stat*) (/usr/local/bin/s3fstest+0x3fa06) #6 0x55e8763df48c in s3fs_open(char const*, fuse_file_info*) (/usr/local/bin/s3fstest+0x5d48c) #7 0x7fbf7b1d7a3f in fuse_fs_open (/lib/x86_64-linux-gnu/libfuse.so.2+0xba3f) #8 0x7fbf7b1d7b21 (/lib/x86_64-linux-gnu/libfuse.so.2+0xbb21) #9 0x7fbf7b1e1f9b (/lib/x86_64-linux-gnu/libfuse.so.2+0x15f9b) #10 0x7fbf7b1e16c0 (/lib/x86_64-linux-gnu/libfuse.so.2+0x156c0) #11 0x7fbf7b1dde67 (/lib/x86_64-linux-gnu/libfuse.so.2+0x11e67) #12 0x7fbf7a11d6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da) #13 0x7fbf79e4671e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV (/usr/local/bin/s3fstest+0x10684b) in curl_slist_remove(curl_slist*, char const*) Thread T1 created by T0 here: #0 0x7fbf7b43fd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f) #1 0x7fbf7b1ddbb4 (/lib/x86_64-linux-gnu/libfuse.so.2+0x11bb4) ==295096==ABORTING ``` Somehow with `clang` not able to stack details. I used `gcc (g++)` compiler.
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Mar 18, 2022):

@nkkashyap Thanks for your reply.
We have released a new version, which includes #1917.
Is it still happening with using new version?

In the first log you received, you know the following:

Feb 10 07:22:21 00000c4a s3fs[214750]: < HTTP/1.1 500 Server Error
Feb 10 07:22:21 00000c4a s3fs[214750]: < X-Clv-Request-Id: ead30edf-b0d1-465f-a12b-f9de878835c8
  ...
  ...
Feb 10 07:22:21 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2387): Body Text:
Feb 10 07:22:23 00000c4a s3fs[214750]: ### retrying...
Feb 10 07:22:23 00000c4a s3fs[214750]:       Retry request. [type=1][url=https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png][path=/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png]
Feb 10 07:22:23 00000c4a kernel: [3206883.738176] s3fs[214751]: segfault at 8 ip 000055db1b6451ed

And your content of the trace shows the same thing as follows:

    #0 0x55e87648884b in curl_slist_remove(curl_slist*, char const*) (/usr/local/bin/s3fstest+0x10684b)
    #1 0x55e876445ce1 in S3fsCurl::RemakeHandle() (/usr/local/bin/s3fstest+0xc3ce1)
    #2 0x55e87644d049 in S3fsCurl::RequestPerform(bool) (/usr/local/bin/s3fstest+0xcb049)

What I can see from this log is that your s3fs process received a 500 HTTP status from the S3 server(IBM?), and s3fs is trying to resend the request and is getting segfault.
I didn't know why your s3fs got the 500 status, but assuming from the log output, I think that it happened when s3fs set the token string to the buffer in resend request.
In other words, I imagined that a segfault had occurred for the address of that token buffer.

Then I checked the source and noticed that the buffer was not locked when the token was refreshing.
Then this bug was corrected at #1917.
So far, no other cause has been found yet.

Maybe your problem has been fixed in new v1.91(#1917), but are you still getting the error with this version?

<!-- gh-comment-id:1072570540 --> @ggtakec commented on GitHub (Mar 18, 2022): @nkkashyap Thanks for your reply. We have released a new version, which includes #1917. Is it still happening with using new version? In the first log you received, you know the following: ``` Feb 10 07:22:21 00000c4a s3fs[214750]: < HTTP/1.1 500 Server Error Feb 10 07:22:21 00000c4a s3fs[214750]: < X-Clv-Request-Id: ead30edf-b0d1-465f-a12b-f9de878835c8 ... ... Feb 10 07:22:21 00000c4a s3fs[214750]: curl.cpp:RequestPerform(2387): Body Text: Feb 10 07:22:23 00000c4a s3fs[214750]: ### retrying... Feb 10 07:22:23 00000c4a s3fs[214750]: Retry request. [type=1][url=https://s3.private.us-south.cloud-object-storage.appdomain.cloud/monolith-app-build-encrypt/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png][path=/thumbnails/b9cc0963-3f01-46c5-a689-c00bfb279ddc.png] Feb 10 07:22:23 00000c4a kernel: [3206883.738176] s3fs[214751]: segfault at 8 ip 000055db1b6451ed ``` And your content of the trace shows the same thing as follows: ``` #0 0x55e87648884b in curl_slist_remove(curl_slist*, char const*) (/usr/local/bin/s3fstest+0x10684b) #1 0x55e876445ce1 in S3fsCurl::RemakeHandle() (/usr/local/bin/s3fstest+0xc3ce1) #2 0x55e87644d049 in S3fsCurl::RequestPerform(bool) (/usr/local/bin/s3fstest+0xcb049) ``` What I can see from this log is that your s3fs process received a 500 HTTP status from the S3 server(IBM?), and s3fs is trying to resend the request and is getting segfault. I didn't know why your s3fs got the 500 status, but assuming from the log output, I think that it happened when s3fs set the token string to the buffer in resend request. In other words, I imagined that a segfault had occurred for the address of that token buffer. Then I checked the source and noticed that the buffer was not locked when the token was refreshing. Then this bug was corrected at #1917. So far, no other cause has been found yet. Maybe your problem has been fixed in new v1.91(#1917), but are you still getting the error with this version?
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@nkkashyap commented on GitHub (Mar 18, 2022):

@ggtakec Thanks
Updated s3fs to v1.191 on the cluster. Will monitor it for couple of days.

<!-- gh-comment-id:1072669197 --> @nkkashyap commented on GitHub (Mar 18, 2022): @ggtakec Thanks Updated s3fs to `v1.191` on the cluster. Will monitor it for couple of days.
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@nkkashyap commented on GitHub (Mar 24, 2022):

With s3fs v1.191 the segfault issue is not occurring.
I am closing this issue.
Thanks @ggtakec @gaul

<!-- gh-comment-id:1077433882 --> @nkkashyap commented on GitHub (Mar 24, 2022): With `s3fs v1.191` the `segfault` issue is not occurring. I am closing this issue. Thanks @ggtakec @gaul
kerem commented 2026-03-04 01:50:15 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Mar 26, 2022):

@nkkashyap Thank you for confirmation.
If you run into problems, please reopen or post new issue.

<!-- gh-comment-id:1079636758 --> @ggtakec commented on GitHub (Mar 26, 2022): @nkkashyap Thank you for confirmation. If you run into problems, please reopen or post new issue.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#960
No description provided.