[GH-ISSUE #2394] Error marker_xp->nodesetval is empty #1177

New issue

Open

opened 2026-03-04 01:51:57 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:51:57 +03:00

Owner

Originally created by @sherpaurgen on GitHub (Jan 1, 2024).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2394

Version of s3fs being used (`s3fs --version`)

s3fs version 1.91

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

Fuse v2.9.9-3

Kernel information (`uname -r`)

5.4.0-125-generic

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

(ubuntu 20.04.4)

How to run s3fs, if applicable , cat /etc/fstab . Modified parts below

mybucket:/bucketname   /path/to/destination   fuse.s3fs _netdev,nonempty,allow_other,nosuid,nodev,uid=1234,gid=1234,umask=7022,passwd_file=/some/path,url=https://s3.us-west-1.domain.com,logfile=/path/to/debug.log,dbglevel=info   0 0

s3fs syslog messages


2023-12-28T16:07:11.272Z [ERR] fdcache_entity.cpp:Read(1750): could not reserve disk space for pre-fetch download
2023-12-28T16:07:11.272Z [WAN] s3fs.cpp:s3fs_read(2345): failed to read file(/bucket/smallfiles/loadsOf/12/06/1701820800/_yv7.cfs.bgz.gz). result=-28
2023-12-28T16:07:11.272Z [WAN] fdcache_entity.cpp:Read(1747): could not reserve disk space for pre-fetch download
2023-12-28T16:07:11.273Z [ERR] fdcache_entity.cpp:Read(1750): could not reserve disk space for pre-fetch download

2024-01-01T01:18:40.920Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.
2024-01-01T01:18:40.922Z [INF]       curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01]
2024-01-01T01:18:40.922Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:40.922Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:40.922Z [INF]       curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] []
2024-01-01T01:18:40.922Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com
2024-01-01T01:18:41.013Z [INF]       curl.cpp:RequestPerform(2327): HTTP response code 200
2024-01-01T01:18:41.023Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.
2024-01-01T01:18:41.024Z [INF]       curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01]
2024-01-01T01:18:41.024Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.025Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.025Z [INF]       curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] []
2024-01-01T01:18:41.025Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com
2024-01-01T01:18:41.108Z [INF]       curl.cpp:RequestPerform(2327): HTTP response code 200
2024-01-01T01:18:41.118Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.
2024-01-01T01:18:41.119Z [INF]       curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01]
2024-01-01T01:18:41.119Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.119Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.119Z [INF]       curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] []
2024-01-01T01:18:41.119Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com
2024-01-01T01:18:41.142Z [INF]       curl.cpp:RequestPerform(2327): HTTP response code 200
2024-01-01T01:18:41.144Z [INF]   s3fs.cpp:readdir_multi_head(2586): [path=/mydata/myserverlog/2023/12/01/][list=0]
2024-01-01T01:18:41.145Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800.prop]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800.prop][bpath=1701388800.prop][save=/mydata/myserverlog/2023/12/01/1701388800.prop][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800.prop
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800.prop
2024-01-01T01:18:41.146Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_0.gz]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_0.gz][bpath=1701388800_0.gz][save=/mydata/myserverlog/2023/12/01/1701388800_0.gz][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.gz
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.gz
2024-01-01T01:18:41.146Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_0.sha1]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_0.sha1][bpath=1701388800_0.sha1][save=/mydata/myserverlog/2023/12/01/1701388800_0.sha1][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.sha1
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.sha1
2024-01-01T01:18:41.146Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_1.gz]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_1.gz][bpath=1701388800_1.gz][save=/mydata/myserverlog/2023/12/01/1701388800_1.gz][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_1.gz
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_1.gz

Details about issue

For some reason the s3fs gets un-mounted and nothing is logged in syslog/dmesg or in journalctl, Therefore i have enabled logging in fstab mount option.
What could be the reason for the ERROR message s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty. ?

There are entries in logs that says "could not reserve diskspace for prefetch download" but the use_cache is default/disabled.
https://manpages.ubuntu.com/manpages/focal/man1/s3fs.1.html

Originally created by @sherpaurgen on GitHub (Jan 1, 2024). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2394 #### Version of s3fs being used (`s3fs --version`) s3fs version 1.91 #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) Fuse v2.9.9-3 #### Kernel information (`uname -r`) 5.4.0-125-generic #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) (ubuntu 20.04.4) #### How to run s3fs, if applicable , cat /etc/fstab . Modified parts below ``` mybucket:/bucketname /path/to/destination fuse.s3fs _netdev,nonempty,allow_other,nosuid,nodev,uid=1234,gid=1234,umask=7022,passwd_file=/some/path,url=https://s3.us-west-1.domain.com,logfile=/path/to/debug.log,dbglevel=info 0 0 ``` #### s3fs syslog messages ``` 2023-12-28T16:07:11.272Z [ERR] fdcache_entity.cpp:Read(1750): could not reserve disk space for pre-fetch download 2023-12-28T16:07:11.272Z [WAN] s3fs.cpp:s3fs_read(2345): failed to read file(/bucket/smallfiles/loadsOf/12/06/1701820800/_yv7.cfs.bgz.gz). result=-28 2023-12-28T16:07:11.272Z [WAN] fdcache_entity.cpp:Read(1747): could not reserve disk space for pre-fetch download 2023-12-28T16:07:11.273Z [ERR] fdcache_entity.cpp:Read(1750): could not reserve disk space for pre-fetch download 2024-01-01T01:18:40.920Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty. 2024-01-01T01:18:40.922Z [INF] curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01] 2024-01-01T01:18:40.922Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/ 2024-01-01T01:18:40.922Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/ 2024-01-01T01:18:40.922Z [INF] curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] [] 2024-01-01T01:18:40.922Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com 2024-01-01T01:18:41.013Z [INF] curl.cpp:RequestPerform(2327): HTTP response code 200 2024-01-01T01:18:41.023Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty. 2024-01-01T01:18:41.024Z [INF] curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01] 2024-01-01T01:18:41.024Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/ 2024-01-01T01:18:41.025Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/ 2024-01-01T01:18:41.025Z [INF] curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] [] 2024-01-01T01:18:41.025Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com 2024-01-01T01:18:41.108Z [INF] curl.cpp:RequestPerform(2327): HTTP response code 200 2024-01-01T01:18:41.118Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty. 2024-01-01T01:18:41.119Z [INF] curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01] 2024-01-01T01:18:41.119Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/ 2024-01-01T01:18:41.119Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/ 2024-01-01T01:18:41.119Z [INF] curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] [] 2024-01-01T01:18:41.119Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com 2024-01-01T01:18:41.142Z [INF] curl.cpp:RequestPerform(2327): HTTP response code 200 2024-01-01T01:18:41.144Z [INF] s3fs.cpp:readdir_multi_head(2586): [path=/mydata/myserverlog/2023/12/01/][list=0] 2024-01-01T01:18:41.145Z [INF] cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800.prop] 2024-01-01T01:18:41.146Z [INF] curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800.prop][bpath=1701388800.prop][save=/mydata/myserverlog/2023/12/01/1701388800.prop][sseckeypos=18446744073709551615] 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800.prop 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800.prop 2024-01-01T01:18:41.146Z [INF] cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_0.gz] 2024-01-01T01:18:41.146Z [INF] curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_0.gz][bpath=1701388800_0.gz][save=/mydata/myserverlog/2023/12/01/1701388800_0.gz][sseckeypos=18446744073709551615] 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.gz 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.gz 2024-01-01T01:18:41.146Z [INF] cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_0.sha1] 2024-01-01T01:18:41.146Z [INF] curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_0.sha1][bpath=1701388800_0.sha1][save=/mydata/myserverlog/2023/12/01/1701388800_0.sha1][sseckeypos=18446744073709551615] 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.sha1 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.sha1 2024-01-01T01:18:41.146Z [INF] cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_1.gz] 2024-01-01T01:18:41.146Z [INF] curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_1.gz][bpath=1701388800_1.gz][save=/mydata/myserverlog/2023/12/01/1701388800_1.gz][sseckeypos=18446744073709551615] 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_1.gz 2024-01-01T01:18:41.146Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_1.gz ``` ### Details about issue For some reason the s3fs gets un-mounted and nothing is logged in syslog/dmesg or in journalctl, Therefore i have enabled logging in fstab mount option. What could be the reason for the ERROR message `s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.` ? There are entries in logs that says "could not reserve diskspace for prefetch download" but the use_cache is default/disabled. https://manpages.ubuntu.com/manpages/focal/man1/s3fs.1.html

kerem commented

2026-03-04 01:51:58 +03:00

Author

Owner

@tanguofu commented on GitHub (Jan 19, 2024):

same question

@tanguofu commented on GitHub (Jan 19, 2024): same question

kerem commented

2026-03-04 01:51:58 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 12, 2024):

Your problem seems unrelated to the error message s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty..
Actually, this message shouldn't be at ERR level, so I think I'll create a new PR.
I think this message was confusing and confused you.

The problem with this issue seems to be the could not reserve disk space for pre-fetch download error message that you also noticed.
To read files, s3fs stores downloaded objects as temporary files on the local disk.
I get an error saying there isn't enough space for this.
Note that the use_cache option keeps and caches this temporary file for a certain period of time. This does not mean that this temporary file is not created. So you need free space for download.

@ggtakec commented on GitHub (Feb 12, 2024): Your problem seems unrelated to the error message `s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.`. Actually, this message shouldn't be at ERR level, so I think I'll create a new PR. I think this message was confusing and confused you. The problem with this issue seems to be the `could not reserve disk space for pre-fetch download` error message that you also noticed. To read files, s3fs stores downloaded objects as temporary files on the local disk. I get an error saying there isn't enough space for this. Note that the `use_cache` option keeps and caches this temporary file for a certain period of time. This does not mean that this temporary file is not created. So you need free space for download.