[GH-ISSUE #779] I/O errors when running iozone on s3fs mounted file system #450

New issue

Closed

opened 2026-03-04 01:45:40 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:45:40 +03:00

Owner

Originally created by @apjvanloo on GitHub (Jun 21, 2018).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/779

Additional Information

I am running iozone with the following command line on an s3fs mounted file system:
iozone -R -I -u 3 -r 1m -s 256m -F /mnt/s3fs/s3fs-test-bucket-dv/test/1 /mnt/s3fs/s3fs-test-bucket-dv/test/2 /mnt/s3fs/s3fs-test-bucket-dv/test/3

During the iozone test with 2 concurrent processes the following error occurs:
Error in file: Found ?0? Expecting ?f4f4f4f4f4f4f4f4? addr 7f384c800000
Error in file: Position 267386880
Record # 255 Record size 1024 kb
where 0x7f384c800000x loop 0

In the next test stage I then receive the following error and iozone hangs:
/mnt/s3fs/s3fs-test-bucket-dv/test/1: No such file or directory

Version of s3fs being used (s3fs --version)

V1.83(commit:3bf05da) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse)

2.9.2

System information (uname -r)

3.10.0-862.3.2.el7.x86_64

Distro (cat /etc/issue)

CentOS Linux release 7.5.1804 (Core)

s3fs command line used (if applicable)

s3fs s3fs-test-bucket-dv /mnt/s3fs/s3fs-test-bucket-dv -o use_xattr -o enable_noobj_cache -o stat_cache_expire=30 -o enable_content_md5 -o dbglevel=info -f -o curldbg

/etc/fstab entry (if applicable):

s3fs syslog messages (grep s3fs /var/log/syslog, or s3fs outputs)

[INF] s3fs.cpp:s3fs_getattr(842): [path=/test]
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test]
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test/2]
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test/1]
[INF] s3fs.cpp:s3fs_open(2064): [path=/test/2][flags=49152]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/2]
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/2]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/2][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2] [] []
[INF] s3fs.cpp:s3fs_open(2064): [path=/test/1][flags=49152]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/1]
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/1]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/1][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c8c05c8d0
* Re-using existing connection! (#67) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#67)
> HEAD /test/1 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77754082538c46accde2d206367323d66a7e72fa64001dee2f2bb12257f59d89
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123855Z

[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/2 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4f7b028d24d7635b4bb1985c405d7d884683855342f55fbaaa7a42d58e33de47
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123855Z

< HTTP/1.1 200 OK
< x-amz-id-2: HTUkesjcgV7v+SU6iEDJPYBL9YrVCLZKSo9hRuba78wkjOXPAow3YmeTFqP5J65xkbRL6Mks9Eg=
< x-amz-request-id: D4FAE1FA07D30077
< Date: Thu, 21 Jun 2018 12:38:56 GMT
< Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT
< ETag: "0cb6c9a36d3cad15b2a7e3c82bf753f5-26"
< x-amz-meta-mode: 33184
< x-amz-meta-gid: 0
< x-amz-meta-uid: 0
< x-amz-meta-mtime: 1529584721
< x-amz-version-id: 6QcBJudKgfsGE1fj_gp4mdlRT7swP65W
< Accept-Ranges: bytes
< Content-Type: application/octet-stream
< Content-Length: 268435456
< Server: AmazonS3
<
* Connection #67 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF]       cache.cpp:AddStat(356): add stat cache entry[path=/test/1]

< HTTP/1.1 200 OK
< x-amz-id-2: MsNG1VApIlqLUDJf9cx3xH5UoBai4S4wKyR9D/efEIlMc0qZufYew1s1hlejDN4MChC5f3rFcCs=
< x-amz-request-id: 08367045B3770DC3
< Date: Thu, 21 Jun 2018 12:38:56 GMT
< Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT
< ETag: "13a11ae17c5b77f99c31a85dbdcfa5f8-26"
< x-amz-meta-mode: 33184
< x-amz-meta-gid: 0
< x-amz-meta-uid: 0
< x-amz-meta-mtime: 1529584721
< x-amz-version-id: 4ZIp_OPMgQ9a6MSScp4NRxZEpSM4FrCX
< Accept-Ranges: bytes
< Content-Type: application/octet-stream
< Content-Length: 268435456
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF]       cache.cpp:AddStat(356): add stat cache entry[path=/test/2]
[INF]       fdcache.cpp:SetMtime(1026): [path=/test/2][fd=14][time=1529584721]
[INF]       curl.cpp:GetObjectRequest(3054): [tpath=/test/2][start=267386880][size=1048576]
[INF]       curl.cpp:PreGetObjectRequest(2994): [tpath=/test/2][start=267386880][size=1048576]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2
[INF]       curl.cpp:insertV4Headers(2411): computing signature [GET] [/test/2] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
[INF]       curl.cpp:GetObjectRequest(3069): downloading... [path=/test/2][fd=14]
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> GET /test/2 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;range;x-amz-content-sha256;x-amz-date, Signature=e65cbb494013461ee78c22744ca0e6258b7ee755875897e58332ce12a380c38e
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
Range: bytes=267386880-268435455
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123856Z

< HTTP/1.1 206 Partial Content
< x-amz-id-2: JvSZaxz77bf7iJpk/y24GMLsq20q3rcxR4ilaA7wn464T+A7KL8uXL5Sj+ByO6hBIiuQZn/JKNI=
< x-amz-request-id: 19561A805F7A3D32
< Date: Thu, 21 Jun 2018 12:38:57 GMT
< Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT
< ETag: "13a11ae17c5b77f99c31a85dbdcfa5f8-26"
< x-amz-meta-mode: 33184
< x-amz-meta-gid: 0
< x-amz-meta-uid: 0
< x-amz-meta-mtime: 1529584721
< x-amz-version-id: 4ZIp_OPMgQ9a6MSScp4NRxZEpSM4FrCX
< Accept-Ranges: bytes
< Content-Range: bytes 267386880-268435455/268435456
< Content-Type: application/octet-stream
< Content-Length: 1048576
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 206
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test/2]
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/.fuse_hidden0000000500000002 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f4c2d46c64c90d1c149028ad462097032781d62cefc1959eb461e403280fc5e8
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123856Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: 0E407F34ADEB7C1D
< x-amz-id-2: uQN5CA3rdCf/sx9u3+p/G9YAZkj8scC16fsmeKGhc8JRVg8atBXR5MEjlqBa0RG7c8OxFyRUQH0=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:38:55 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002/]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002/
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002/
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002/] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/.fuse_hidden0000000500000002/ HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f20044e04257828e76f6d87daee5313ff39e2de0d7928a586788ff8d0c354d75
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123856Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: B552E319E550AA09
< x-amz-id-2: E33/CwY4qgWgbgj1xR+gyIvytZ08qPHmlT6OxYGvzsapkE8NZ1IjMUMXi6qG86dShtPhzgsGnYo=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:38:55 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002_$folder$]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002_%24folder%24
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002_%24folder%24
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002_$folder$] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/.fuse_hidden0000000500000002_%24folder%24 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=1f724a6d7e6386fbd3008e9fcdaae51d4e5895eae1245d3190aa168f5690408a
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123856Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: C243C1317EADBC77
< x-amz-id-2: HAvuWt5EHptPObGKiNxCVDJJbgVyMRT3zXWVGCskoobYwazlGWXtIjPa6tIXC/Di0gUmmT6cYu0=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:38:55 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2476): [path=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:ListBucketRequest(3114): [tpath=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv?delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com?delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/
[INF]       curl.cpp:insertV4Headers(2411): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> GET /?delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/ HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8195b62b130e8943beac65ac7e504e5820534b8dce010092c7ad743771e77af3
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123856Z

< HTTP/1.1 200 OK
< x-amz-id-2: KmikVozwhZ1jhV1ZObwP0uck8pgOhM4UMrjJzmNCXBpBwRU4tIWBs5nCJqzz/IeeNWpV3RyE6zs=
< x-amz-request-id: D7BB84DF2D8DE5F1
< Date: Thu, 21 Jun 2018 12:38:57 GMT
< x-amz-bucket-region: eu-west-1
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF]       cache.cpp:AddNoObjectCache(429): add no object cache entry[path=/test/.fuse_hidden0000000500000002]
[INF] s3fs.cpp:s3fs_rename(1504): [from=/test/2][to=/test/.fuse_hidden0000000500000002]
[INF]   s3fs.cpp:rename_object(1238): [from=/test/2][to=/test/.fuse_hidden0000000500000002]
[INF]     s3fs.cpp:put_headers(803): [path=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:PutHeadRequest(2764): [tpath=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:insertV4Headers(2411): computing signature [PUT] [/test/.fuse_hidden0000000500000002] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
[INF]       curl.cpp:PutHeadRequest(2848): copying... [path=/test/.fuse_hidden0000000500000002]
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> PUT /test/.fuse_hidden0000000500000002 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-gid;x-amz-meta-mode;x-amz-meta-mtime;x-amz-meta-uid;x-amz-metadata-directive, Signature=b31fb3ae56b14f52ca309819d1e0efb078b60730ef2b3b14a8aedfc9e855aacc
Content-Type: application/octet-stream
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-acl: private
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-copy-source: /s3fs-test-bucket-dv/test/2
x-amz-date: 20180621T123856Z
x-amz-meta-gid: 0
x-amz-meta-mode: 33184
x-amz-meta-mtime: 1529584721
x-amz-meta-uid: 0
x-amz-metadata-directive: REPLACE
Content-Length: 0
Expect: 100-continue

< HTTP/1.1 100 Continue
[INF] s3fs.cpp:s3fs_fsync(2219): [path=/test/1][fd=7]
[INF]       fdcache.cpp:RowFlush(1441): [tpath=][path=/test/1][fd=7]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/1]
[INF] s3fs.cpp:s3fs_flush(2186): [path=/test/1][fd=7]
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/1]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/1][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c8c05c8d0
* Re-using existing connection! (#67) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#67)
> HEAD /test/1 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f49d87ac3d2789a72f8f6c49055f05166b7d705e8c22f8171fb1e45862beea35
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123856Z

< HTTP/1.1 200 OK
< x-amz-id-2: njntbM6plRajwbUhXXbWv74J7eY7KRzNxFCxCz+A2tldxpBeoW4S9+8IE9dZdZXreLAQFeDnOKc=
< x-amz-request-id: 325231A9B6708D40
< Date: Thu, 21 Jun 2018 12:38:57 GMT
< Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT
< ETag: "0cb6c9a36d3cad15b2a7e3c82bf753f5-26"
< x-amz-meta-mode: 33184
< x-amz-meta-gid: 0
< x-amz-meta-uid: 0
< x-amz-meta-mtime: 1529584721
< x-amz-version-id: 6QcBJudKgfsGE1fj_gp4mdlRT7swP65W
< Accept-Ranges: bytes
< Content-Type: application/octet-stream
< Content-Length: 268435456
< Server: AmazonS3
<
* Connection #67 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF]       cache.cpp:AddStat(356): add stat cache entry[path=/test/1]
[INF]       fdcache.cpp:RowFlush(1441): [tpath=][path=/test/1][fd=7]
[INF] s3fs.cpp:s3fs_release(2239): [path=/test/1][fd=7]
[INF]       fdcache.cpp:GetFdEntity(2020): [path=/test/1][fd=7]
< HTTP/1.1 200 OK
< x-amz-id-2: rSl3tjF6IwOMlix/mUpfqABQ+7rGDk8boY8TKIeT+GdKUqy2WHTXRF6x8JA9vIGnst6dMHeK2+U=
< x-amz-request-id: E80131EA6C9C2F3A
< Date: Thu, 21 Jun 2018 12:38:57 GMT
< x-amz-copy-source-version-id: 4ZIp_OPMgQ9a6MSScp4NRxZEpSM4FrCX
< x-amz-version-id: CHylbSlPunUOvAELFlmCM3yGUeCJWqY_
< Content-Type: application/xml
< Content-Length: 234
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF] s3fs.cpp:s3fs_unlink(1094): [path=/test/2]
[INF]       curl.cpp:DeleteRequest(2482): [tpath=/test/2]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2
[INF]       curl.cpp:insertV4Headers(2411): computing signature [DELETE] [/test/2] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> DELETE /test/2 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4b2be371cda0d5e6b5c5ac37211dd1da19a61425dd6a8954982270d4f93eca19
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123901Z

< HTTP/1.1 204 No Content
< x-amz-id-2: h4iJYYx00eNSEzvVeksBMJq6Y2yvXfvNSQf8lS58hTS+YgR+lljSuVGT+FtZBzj8WCsccXbdYYE=
< x-amz-request-id: 94AE091C7D1B5733
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< x-amz-version-id: c8lVSFiTyMCBFyB.SzlehUwicyjR0UFl
< x-amz-delete-marker: true
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 204
[INF]       fdcache.cpp:DeleteCacheFile(1820): [path=/test/2]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/2]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/.fuse_hidden0000000500000002]
[INF] s3fs.cpp:s3fs_flush(2186): [path=/test/.fuse_hidden0000000500000002][fd=14]
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/.fuse_hidden0000000500000002 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4fe5044f307128748ee62a0defb6d5d9a5883e86a2bf92d0efe6fe74ba6f398f
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123901Z

< HTTP/1.1 200 OK
< x-amz-id-2: 8jyT5f7F0UDFpl5EihvNU9gjVYo3y77hYdxGn69GQwX2BeU1CUzBV+xyvImM5xdYBUNTotnoIKY=
< x-amz-request-id: 5A6B6011CCA68290
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< Last-Modified: Thu, 21 Jun 2018 12:38:57 GMT
< ETag: "e262cb567d77be037ca7896b6a1c5a4e"
< x-amz-meta-mode: 33184
< x-amz-meta-gid: 0
< x-amz-meta-uid: 0
< x-amz-meta-mtime: 1529584721
< x-amz-version-id: CHylbSlPunUOvAELFlmCM3yGUeCJWqY_
< Accept-Ranges: bytes
< Content-Type: application/octet-stream
< Content-Length: 268435456
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF]       cache.cpp:AddStat(356): add stat cache entry[path=/test/.fuse_hidden0000000500000002]
[INF] s3fs.cpp:s3fs_release(2239): [path=/test/.fuse_hidden0000000500000002][fd=14]
[INF]       fdcache.cpp:GetFdEntity(2020): [path=/test/.fuse_hidden0000000500000002][fd=14]
[ERR] s3fs.cpp:s3fs_release(2258): could not find fd(file=/test/.fuse_hidden0000000500000002)
[INF] s3fs.cpp:s3fs_unlink(1094): [path=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:DeleteRequest(2482): [tpath=/test/.fuse_hidden0000000500000002]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002
[INF]       curl.cpp:insertV4Headers(2411): computing signature [DELETE] [/test/.fuse_hidden0000000500000002] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> DELETE /test/.fuse_hidden0000000500000002 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=764215b694035c862198564f3995c8f66566a5cd0f18933f27ba2dff4684ab85
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123901Z

< HTTP/1.1 204 No Content
< x-amz-id-2: Mn+0ylPyaV6dvATxagzUAqMy5xSwra0fVSwFoUxvjjy8MNP2QQfcHwXflY3kq5oH+dD+YPX8/X0=
< x-amz-request-id: CFB6D1411D20E122
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< x-amz-version-id: E87Xq6qIzFrSc1nYmj6k_fMCGiZualiE
< x-amz-delete-marker: true
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 204
[INF]       fdcache.cpp:DeleteCacheFile(1820): [path=/test/.fuse_hidden0000000500000002]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/.fuse_hidden0000000500000002]
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test]
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test/2.DUMMY.1]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/2.DUMMY.1]
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/2.DUMMY.1]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/2.DUMMY.1][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2.DUMMY.1
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2.DUMMY.1
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2.DUMMY.1] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/2.DUMMY.1 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d838059818662bafc9de288babaea03b0d72d8dd7a75f2ed89a37d5c7d0ae033
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: D7A800B110A1795B
< x-amz-id-2: GWRgVVLTd8XxBNwQMW91wJLem5RdZzSv6MznWZiHeRoGk4PNEbavaRR2Xqdv+4DsE4Wt8rzW13o=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/2.DUMMY.1/]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/2.DUMMY.1/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2.DUMMY.1/
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2.DUMMY.1/
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2.DUMMY.1/] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/2.DUMMY.1/ HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3b8522da44f8f16044aa7c07be0219b978d1454bfc8ba7441bedd3c7fbcbeb02
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: C07A637AA9D04747
< x-amz-id-2: 6X58WWYCRIuNre8Kl5XXFUhMNzoudttFW77w8Mq4scQFIbF4MgNahdZrLrTZNoBadfJ4SBBOLXg=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/2.DUMMY.1_$folder$]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/2.DUMMY.1_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2.DUMMY.1_%24folder%24
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2.DUMMY.1_%24folder%24
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2.DUMMY.1_$folder$] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/2.DUMMY.1_%24folder%24 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=1de54be0e917cf2139786ec0f294e60febc144d71de92a15a63ab5a3ec70397c
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: 0F94A1AC612BD260
< x-amz-id-2: iRdRsYEcyKHvMHsR+k0LJ0XVj2irhgaUs21XlSe1FLTLL5Ud4rY1ZdIGFOvNggYR0DlPgSchsAE=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2476): [path=/test/2.DUMMY.1]
[INF]       curl.cpp:ListBucketRequest(3114): [tpath=/test/2.DUMMY.1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv?delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com?delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/
[INF]       curl.cpp:insertV4Headers(2411): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> GET /?delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/ HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=17cf0edfb1032129d03d38b79301f36b5d8eda32abe241d07f5271724b1455ee
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 200 OK
< x-amz-id-2: kqBrwwyhNuiNGtV9MXPQ9FjCx1XXKhpp4cZtwDHOmPsRCEH8GgklAhbbnw6OH+4xfrKqA9cN23s=
< x-amz-request-id: 40B4AF1082FB192C
< Date: Thu, 21 Jun 2018 12:39:04 GMT
< x-amz-bucket-region: eu-west-1
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF]       cache.cpp:AddNoObjectCache(429): add no object cache entry[path=/test/2.DUMMY.1]
[INF] s3fs.cpp:s3fs_getattr(842): [path=/test/1.DUMMY.0]
[INF]       cache.cpp:DelStat(565): delete stat cache entry[path=/test/1.DUMMY.0]
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/1.DUMMY.0]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/1.DUMMY.0][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1.DUMMY.0
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1.DUMMY.0
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1.DUMMY.0] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/1.DUMMY.0 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=22111d76820c087ca0c3a2f6652226484418bc35ca4e4d80c8865f092a2f7fcc
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: 2A1154028E56ACFA
< x-amz-id-2: Sc2JhmPWXTF3V6V6mN9rppyU5Rs4fzclv+6ZV8JAhuSx/4GbAVSstmZmb1b558lr9Y3kjTLrn+A=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/1.DUMMY.0/]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/1.DUMMY.0/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1.DUMMY.0/
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1.DUMMY.0/
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1.DUMMY.0/] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/1.DUMMY.0/ HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c23e5da5d5a7ac2942eac97e2455bc8bf42ff37eb4686e363c5f4bf6f30b8952
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: E497F8EC601675FE
< x-amz-id-2: Q9dkp+hwt7FpEsCwL1AQhzYn/7y+hY1fLwkTsX8HNrrgX2Ee2o4Ox3fckJiovaSxNtE8h3UnTVk=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(2719): [tpath=/test/1.DUMMY.0_$folder$]
[INF]       curl.cpp:PreHeadRequest(2668): [tpath=/test/1.DUMMY.0_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1.DUMMY.0_%24folder%24
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1.DUMMY.0_%24folder%24
[INF]       curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1.DUMMY.0_$folder$] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> HEAD /test/1.DUMMY.0_%24folder%24 HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=6f3c7afad984607b9751b915d770d5da7ff415dac5b29a6f503e473d621097ae
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 404 Not Found
< x-amz-request-id: 0D9732552F817485
< x-amz-id-2: yIrTANkB6WIxFdHwX9aqgJ+TVKDi6uiPqvpUmLK7RVcPMeH0q4Ko1i1X1SnVitcoCc+xymUxk9M=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Thu, 21 Jun 2018 12:39:02 GMT
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2476): [path=/test/1.DUMMY.0]
[INF]       curl.cpp:ListBucketRequest(3114): [tpath=/test/1.DUMMY.0]
[INF]       curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv?delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/
[INF]       curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com?delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/
[INF]       curl.cpp:insertV4Headers(2411): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/] []
[INF]       curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com
* Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990
* Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
* Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62)
> GET /?delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/ HTTP/1.1
User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL)
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e8175651334a2bb879930d8b7300e187f93ef20b1292e547abf1e9d68691b6e1
host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20180621T123903Z

< HTTP/1.1 200 OK
< x-amz-id-2: xQsm45SuSh1604G4VlAmDbvwdbf3m3WFB7h02spMF1seXIBHLNQsV5Ndtgng8vdObkdyvly/XQw=
< x-amz-request-id: E1E29D2BFE124DFF
< Date: Thu, 21 Jun 2018 12:39:04 GMT
< x-amz-bucket-region: eu-west-1
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Server: AmazonS3
<
* Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF]       cache.cpp:AddNoObjectCache(429): add no object cache entry[path=/test/1.DUMMY.0]

Details about issue

iozone output:

iozone -R -I 3 -u 3 -r 1m -s 256m -F /mnt/s3fs/s3fs-test-bucket-dv/test/1 /mnt/s3fs/s3fs-test-bucket-dv/test/2 /mnt/s3fs/s3fs-test-bucket-dv/test/3
        Iozone: Performance Test of File I/O
                Version $Revision: 3.424 $
                Compiled for 64 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa.

        Run began: Thu Jun 21 12:37:09 2018

        Excel chart generation enabled
        O_DIRECT feature enabled
        Record Size 1024 kB
        File size set to 262144 kB
        Command line used: iozone -R -I -u 3 -r 1m -s 256m -F /mnt/s3fs/s3fs-test-bucket-dv/test/1 /mnt/s3fs/s3fs-test-bucket-dv/test/2 /mnt/s3fs/s3fs-test-bucket-dv/test/3 3
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Min process = 1
        Max process = 3
        Throughput test with 1 process
        Each process writes a 262144 kByte file in 1024 kByte records

        Children see throughput for  1 initial writers  = 1197294.88 kB/sec
        Parent sees throughput for  1 initial writers   =   49652.62 kB/sec
        Min throughput per process                      = 1197294.88 kB/sec
        Max throughput per process                      = 1197294.88 kB/sec
        Avg throughput per process                      = 1197294.88 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for  1 rewriters        = 1359047.88 kB/sec
        Parent sees throughput for  1 rewriters         =   37622.45 kB/sec
        Min throughput per process                      = 1359047.88 kB/sec
        Max throughput per process                      = 1359047.88 kB/sec
        Avg throughput per process                      = 1359047.88 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for  1 readers          = 2639758.25 kB/sec
        Parent sees throughput for  1 readers           = 2488175.04 kB/sec
        Min throughput per process                      = 2639758.25 kB/sec
        Max throughput per process                      = 2639758.25 kB/sec
        Avg throughput per process                      = 2639758.25 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 re-readers        = 2718630.75 kB/sec
        Parent sees throughput for 1 re-readers         = 2512159.94 kB/sec
        Min throughput per process                      = 2718630.75 kB/sec
        Max throughput per process                      = 2718630.75 kB/sec
        Avg throughput per process                      = 2718630.75 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 reverse readers   = 2654787.50 kB/sec
        Parent sees throughput for 1 reverse readers    = 2421054.24 kB/sec
        Min throughput per process                      = 2654787.50 kB/sec
        Max throughput per process                      = 2654787.50 kB/sec
        Avg throughput per process                      = 2654787.50 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 stride readers    = 2623539.00 kB/sec
        Parent sees throughput for 1 stride readers     = 2454328.35 kB/sec
        Min throughput per process                      = 2623539.00 kB/sec
        Max throughput per process                      = 2623539.00 kB/sec
        Avg throughput per process                      = 2623539.00 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 random readers    = 2597544.75 kB/sec
        Parent sees throughput for 1 random readers     = 2390191.72 kB/sec
        Min throughput per process                      = 2597544.75 kB/sec
        Max throughput per process                      = 2597544.75 kB/sec
        Avg throughput per process                      = 2597544.75 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 mixed workload    = 2626246.25 kB/sec
        Parent sees throughput for 1 mixed workload     = 2476161.12 kB/sec
        Min throughput per process                      = 2626246.25 kB/sec
        Max throughput per process                      = 2626246.25 kB/sec
        Avg throughput per process                      = 2626246.25 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 random writers    = 1362510.50 kB/sec
        Parent sees throughput for 1 random writers     =   35409.74 kB/sec
        Min throughput per process                      = 1362510.50 kB/sec
        Max throughput per process                      = 1362510.50 kB/sec
        Avg throughput per process                      = 1362510.50 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 pwrite writers    = 1176715.00 kB/sec
        Parent sees throughput for 1 pwrite writers     =   53749.58 kB/sec
        Min throughput per process                      = 1176715.00 kB/sec
        Max throughput per process                      = 1176715.00 kB/sec
        Avg throughput per process                      = 1176715.00 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for 1 pread readers     = 2671883.50 kB/sec
        Parent sees throughput for 1 pread readers      = 2468306.59 kB/sec
        Min throughput per process                      = 2671883.50 kB/sec
        Max throughput per process                      = 2671883.50 kB/sec
        Avg throughput per process                      = 2671883.50 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for  1 fwriters         =  235445.19 kB/sec
        Parent sees throughput for  1 fwriters          =   39019.24 kB/sec
        Min throughput per process                      =  235445.19 kB/sec
        Max throughput per process                      =  235445.19 kB/sec
        Avg throughput per process                      =  235445.19 kB/sec
        Min xfer                                        =  262144.00 kB

        Children see throughput for  1 freaders         = 1754058.00 kB/sec
        Parent sees throughput for  1 freaders          = 1746893.07 kB/sec
        Min throughput per process                      = 1754058.00 kB/sec
        Max throughput per process                      = 1754058.00 kB/sec
        Avg throughput per process                      = 1754058.00 kB/sec
        Min xfer                                        =  262144.00 kB


        Each process writes a 262144 kByte file in 1024 kByte records

        Children see throughput for  2 initial writers  = 1675515.56 kB/sec
        Parent sees throughput for  2 initial writers   =   76689.69 kB/sec
        Min throughput per process                      =  829283.50 kB/sec
        Max throughput per process                      =  846232.06 kB/sec
        Avg throughput per process                      =  837757.78 kB/sec
        Min xfer                                        =  257024.00 kB

        Children see throughput for  2 rewriters        = 1741225.56 kB/sec
        Parent sees throughput for  2 rewriters         =   87488.14 kB/sec
        Min throughput per process                      =  828102.25 kB/sec
        Max throughput per process                      =  913123.31 kB/sec
        Avg throughput per process                      =  870612.78 kB/sec
        Min xfer                                        =  238592.00 kB

        Children see throughput for  2 readers          = 1747556.62 kB/sec
        Parent sees throughput for  2 readers           = 1249274.95 kB/sec
        Min throughput per process                      =       0.00 kB/sec
        Max throughput per process                      = 1747556.62 kB/sec
        Avg throughput per process                      =  873778.31 kB/sec
        Min xfer                                        =       0.00 kB

        Children see throughput for 2 re-readers        = 1040242.66 kB/sec
        Parent sees throughput for 2 re-readers         =  820165.62 kB/sec
        Min throughput per process                      =  233920.78 kB/sec
        Max throughput per process                      =  806321.88 kB/sec
        Avg throughput per process                      =  520121.33 kB/sec
        Min xfer                                        =  102400.00 kB


Error in file: Found ?0? Expecting ?f4f4f4f4f4f4f4f4? addr 7f725af00000
Error in file: Position 267386880
Record # 255 Record size 1024 kb
where 0x7f725af00000x loop 0

        Children see throughput for 2 reverse readers   = 2558754.50 kB/sec
        Parent sees throughput for 2 reverse readers    =   53573.57 kB/sec
        Min throughput per process                      =       0.00 kB/sec
        Max throughput per process                      = 2558754.50 kB/sec
        Avg throughput per process                      = 1279377.25 kB/sec
        Min xfer                                        =       0.00 kB
^C
iozone: interrupted


"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 1024 kBytes "
"Output is in kBytes/sec"

"  Initial write " 1197294.88  1675515.56

"        Rewrite " 1359047.88  1741225.56

"           Read " 2639758.25  1747556.62

"        Re-read " 2718630.75  1040242.66

"   Reverse Read " 2654787.50  2558754.50

exiting iozone

Originally created by @apjvanloo on GitHub (Jun 21, 2018). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/779 ### Additional Information I am running iozone with the following command line on an s3fs mounted file system: iozone -R -I -u 3 -r 1m -s 256m -F /mnt/s3fs/s3fs-test-bucket-dv/test/1 /mnt/s3fs/s3fs-test-bucket-dv/test/2 /mnt/s3fs/s3fs-test-bucket-dv/test/3 During the iozone test with 2 concurrent processes the following error occurs: Error in file: Found ?0? Expecting ?f4f4f4f4f4f4f4f4? addr 7f384c800000 Error in file: Position 267386880 Record # 255 Record size 1024 kb where 0x7f384c800000x loop 0 In the next test stage I then receive the following error and iozone hangs: /mnt/s3fs/s3fs-test-bucket-dv/test/1: No such file or directory #### Version of s3fs being used (s3fs --version) V1.83(commit:3bf05da) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse) 2.9.2 #### System information (uname -r) 3.10.0-862.3.2.el7.x86_64 #### Distro (cat /etc/issue) CentOS Linux release 7.5.1804 (Core) #### s3fs command line used (if applicable) ``` s3fs s3fs-test-bucket-dv /mnt/s3fs/s3fs-test-bucket-dv -o use_xattr -o enable_noobj_cache -o stat_cache_expire=30 -o enable_content_md5 -o dbglevel=info -f -o curldbg ``` #### /etc/fstab entry (if applicable): ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, or s3fs outputs) ``` [INF] s3fs.cpp:s3fs_getattr(842): [path=/test] [INF] s3fs.cpp:s3fs_getattr(842): [path=/test] [INF] s3fs.cpp:s3fs_getattr(842): [path=/test/2] [INF] s3fs.cpp:s3fs_getattr(842): [path=/test/1] [INF] s3fs.cpp:s3fs_open(2064): [path=/test/2][flags=49152] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/2] [INF] curl.cpp:HeadRequest(2719): [tpath=/test/2] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/2][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2] [] [] [INF] s3fs.cpp:s3fs_open(2064): [path=/test/1][flags=49152] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/1] [INF] curl.cpp:HeadRequest(2719): [tpath=/test/1] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/1][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c8c05c8d0 * Re-using existing connection! (#67) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#67) > HEAD /test/1 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=77754082538c46accde2d206367323d66a7e72fa64001dee2f2bb12257f59d89 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123855Z [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/2 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4f7b028d24d7635b4bb1985c405d7d884683855342f55fbaaa7a42d58e33de47 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123855Z < HTTP/1.1 200 OK < x-amz-id-2: HTUkesjcgV7v+SU6iEDJPYBL9YrVCLZKSo9hRuba78wkjOXPAow3YmeTFqP5J65xkbRL6Mks9Eg= < x-amz-request-id: D4FAE1FA07D30077 < Date: Thu, 21 Jun 2018 12:38:56 GMT < Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT < ETag: "0cb6c9a36d3cad15b2a7e3c82bf753f5-26" < x-amz-meta-mode: 33184 < x-amz-meta-gid: 0 < x-amz-meta-uid: 0 < x-amz-meta-mtime: 1529584721 < x-amz-version-id: 6QcBJudKgfsGE1fj_gp4mdlRT7swP65W < Accept-Ranges: bytes < Content-Type: application/octet-stream < Content-Length: 268435456 < Server: AmazonS3 < * Connection #67 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] cache.cpp:AddStat(356): add stat cache entry[path=/test/1] < HTTP/1.1 200 OK < x-amz-id-2: MsNG1VApIlqLUDJf9cx3xH5UoBai4S4wKyR9D/efEIlMc0qZufYew1s1hlejDN4MChC5f3rFcCs= < x-amz-request-id: 08367045B3770DC3 < Date: Thu, 21 Jun 2018 12:38:56 GMT < Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT < ETag: "13a11ae17c5b77f99c31a85dbdcfa5f8-26" < x-amz-meta-mode: 33184 < x-amz-meta-gid: 0 < x-amz-meta-uid: 0 < x-amz-meta-mtime: 1529584721 < x-amz-version-id: 4ZIp_OPMgQ9a6MSScp4NRxZEpSM4FrCX < Accept-Ranges: bytes < Content-Type: application/octet-stream < Content-Length: 268435456 < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] cache.cpp:AddStat(356): add stat cache entry[path=/test/2] [INF] fdcache.cpp:SetMtime(1026): [path=/test/2][fd=14][time=1529584721] [INF] curl.cpp:GetObjectRequest(3054): [tpath=/test/2][start=267386880][size=1048576] [INF] curl.cpp:PreGetObjectRequest(2994): [tpath=/test/2][start=267386880][size=1048576] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2 [INF] curl.cpp:insertV4Headers(2411): computing signature [GET] [/test/2] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com [INF] curl.cpp:GetObjectRequest(3069): downloading... [path=/test/2][fd=14] * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > GET /test/2 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;range;x-amz-content-sha256;x-amz-date, Signature=e65cbb494013461ee78c22744ca0e6258b7ee755875897e58332ce12a380c38e host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com Range: bytes=267386880-268435455 x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123856Z < HTTP/1.1 206 Partial Content < x-amz-id-2: JvSZaxz77bf7iJpk/y24GMLsq20q3rcxR4ilaA7wn464T+A7KL8uXL5Sj+ByO6hBIiuQZn/JKNI= < x-amz-request-id: 19561A805F7A3D32 < Date: Thu, 21 Jun 2018 12:38:57 GMT < Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT < ETag: "13a11ae17c5b77f99c31a85dbdcfa5f8-26" < x-amz-meta-mode: 33184 < x-amz-meta-gid: 0 < x-amz-meta-uid: 0 < x-amz-meta-mtime: 1529584721 < x-amz-version-id: 4ZIp_OPMgQ9a6MSScp4NRxZEpSM4FrCX < Accept-Ranges: bytes < Content-Range: bytes 267386880-268435455/268435456 < Content-Type: application/octet-stream < Content-Length: 1048576 < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 206 [INF] s3fs.cpp:s3fs_getattr(842): [path=/test/2] [INF] s3fs.cpp:s3fs_getattr(842): [path=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/.fuse_hidden0000000500000002 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f4c2d46c64c90d1c149028ad462097032781d62cefc1959eb461e403280fc5e8 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123856Z < HTTP/1.1 404 Not Found < x-amz-request-id: 0E407F34ADEB7C1D < x-amz-id-2: uQN5CA3rdCf/sx9u3+p/G9YAZkj8scC16fsmeKGhc8JRVg8atBXR5MEjlqBa0RG7c8OxFyRUQH0= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:38:55 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002/] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002/ [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002/ [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002/] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/.fuse_hidden0000000500000002/ HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f20044e04257828e76f6d87daee5313ff39e2de0d7928a586788ff8d0c354d75 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123856Z < HTTP/1.1 404 Not Found < x-amz-request-id: B552E319E550AA09 < x-amz-id-2: E33/CwY4qgWgbgj1xR+gyIvytZ08qPHmlT6OxYGvzsapkE8NZ1IjMUMXi6qG86dShtPhzgsGnYo= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:38:55 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002_$folder$] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002_%24folder%24 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002_%24folder%24 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002_$folder$] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/.fuse_hidden0000000500000002_%24folder%24 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=1f724a6d7e6386fbd3008e9fcdaae51d4e5895eae1245d3190aa168f5690408a host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123856Z < HTTP/1.1 404 Not Found < x-amz-request-id: C243C1317EADBC77 < x-amz-id-2: HAvuWt5EHptPObGKiNxCVDJJbgVyMRT3zXWVGCskoobYwazlGWXtIjPa6tIXC/Di0gUmmT6cYu0= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:38:55 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2476): [path=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:ListBucketRequest(3114): [tpath=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv?delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/ [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com?delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/ [INF] curl.cpp:insertV4Headers(2411): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > GET /?delimiter=/&max-keys=2&prefix=test/.fuse_hidden0000000500000002/ HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8195b62b130e8943beac65ac7e504e5820534b8dce010092c7ad743771e77af3 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123856Z < HTTP/1.1 200 OK < x-amz-id-2: KmikVozwhZ1jhV1ZObwP0uck8pgOhM4UMrjJzmNCXBpBwRU4tIWBs5nCJqzz/IeeNWpV3RyE6zs= < x-amz-request-id: D7BB84DF2D8DE5F1 < Date: Thu, 21 Jun 2018 12:38:57 GMT < x-amz-bucket-region: eu-west-1 < Content-Type: application/xml < Transfer-Encoding: chunked < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] cache.cpp:AddNoObjectCache(429): add no object cache entry[path=/test/.fuse_hidden0000000500000002] [INF] s3fs.cpp:s3fs_rename(1504): [from=/test/2][to=/test/.fuse_hidden0000000500000002] [INF] s3fs.cpp:rename_object(1238): [from=/test/2][to=/test/.fuse_hidden0000000500000002] [INF] s3fs.cpp:put_headers(803): [path=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:PutHeadRequest(2764): [tpath=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002 [INF] curl.cpp:insertV4Headers(2411): computing signature [PUT] [/test/.fuse_hidden0000000500000002] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com [INF] curl.cpp:PutHeadRequest(2848): copying... [path=/test/.fuse_hidden0000000500000002] * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > PUT /test/.fuse_hidden0000000500000002 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-gid;x-amz-meta-mode;x-amz-meta-mtime;x-amz-meta-uid;x-amz-metadata-directive, Signature=b31fb3ae56b14f52ca309819d1e0efb078b60730ef2b3b14a8aedfc9e855aacc Content-Type: application/octet-stream host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-acl: private x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-copy-source: /s3fs-test-bucket-dv/test/2 x-amz-date: 20180621T123856Z x-amz-meta-gid: 0 x-amz-meta-mode: 33184 x-amz-meta-mtime: 1529584721 x-amz-meta-uid: 0 x-amz-metadata-directive: REPLACE Content-Length: 0 Expect: 100-continue < HTTP/1.1 100 Continue [INF] s3fs.cpp:s3fs_fsync(2219): [path=/test/1][fd=7] [INF] fdcache.cpp:RowFlush(1441): [tpath=][path=/test/1][fd=7] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/1] [INF] s3fs.cpp:s3fs_flush(2186): [path=/test/1][fd=7] [INF] curl.cpp:HeadRequest(2719): [tpath=/test/1] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/1][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c8c05c8d0 * Re-using existing connection! (#67) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#67) > HEAD /test/1 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f49d87ac3d2789a72f8f6c49055f05166b7d705e8c22f8171fb1e45862beea35 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123856Z < HTTP/1.1 200 OK < x-amz-id-2: njntbM6plRajwbUhXXbWv74J7eY7KRzNxFCxCz+A2tldxpBeoW4S9+8IE9dZdZXreLAQFeDnOKc= < x-amz-request-id: 325231A9B6708D40 < Date: Thu, 21 Jun 2018 12:38:57 GMT < Last-Modified: Thu, 21 Jun 2018 12:38:42 GMT < ETag: "0cb6c9a36d3cad15b2a7e3c82bf753f5-26" < x-amz-meta-mode: 33184 < x-amz-meta-gid: 0 < x-amz-meta-uid: 0 < x-amz-meta-mtime: 1529584721 < x-amz-version-id: 6QcBJudKgfsGE1fj_gp4mdlRT7swP65W < Accept-Ranges: bytes < Content-Type: application/octet-stream < Content-Length: 268435456 < Server: AmazonS3 < * Connection #67 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] cache.cpp:AddStat(356): add stat cache entry[path=/test/1] [INF] fdcache.cpp:RowFlush(1441): [tpath=][path=/test/1][fd=7] [INF] s3fs.cpp:s3fs_release(2239): [path=/test/1][fd=7] [INF] fdcache.cpp:GetFdEntity(2020): [path=/test/1][fd=7] < HTTP/1.1 200 OK < x-amz-id-2: rSl3tjF6IwOMlix/mUpfqABQ+7rGDk8boY8TKIeT+GdKUqy2WHTXRF6x8JA9vIGnst6dMHeK2+U= < x-amz-request-id: E80131EA6C9C2F3A < Date: Thu, 21 Jun 2018 12:38:57 GMT < x-amz-copy-source-version-id: 4ZIp_OPMgQ9a6MSScp4NRxZEpSM4FrCX < x-amz-version-id: CHylbSlPunUOvAELFlmCM3yGUeCJWqY_ < Content-Type: application/xml < Content-Length: 234 < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] s3fs.cpp:s3fs_unlink(1094): [path=/test/2] [INF] curl.cpp:DeleteRequest(2482): [tpath=/test/2] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2 [INF] curl.cpp:insertV4Headers(2411): computing signature [DELETE] [/test/2] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > DELETE /test/2 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4b2be371cda0d5e6b5c5ac37211dd1da19a61425dd6a8954982270d4f93eca19 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123901Z < HTTP/1.1 204 No Content < x-amz-id-2: h4iJYYx00eNSEzvVeksBMJq6Y2yvXfvNSQf8lS58hTS+YgR+lljSuVGT+FtZBzj8WCsccXbdYYE= < x-amz-request-id: 94AE091C7D1B5733 < Date: Thu, 21 Jun 2018 12:39:02 GMT < x-amz-version-id: c8lVSFiTyMCBFyB.SzlehUwicyjR0UFl < x-amz-delete-marker: true < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 204 [INF] fdcache.cpp:DeleteCacheFile(1820): [path=/test/2] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/2] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/.fuse_hidden0000000500000002] [INF] s3fs.cpp:s3fs_flush(2186): [path=/test/.fuse_hidden0000000500000002][fd=14] [INF] curl.cpp:HeadRequest(2719): [tpath=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/.fuse_hidden0000000500000002][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/.fuse_hidden0000000500000002] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/.fuse_hidden0000000500000002 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4fe5044f307128748ee62a0defb6d5d9a5883e86a2bf92d0efe6fe74ba6f398f host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123901Z < HTTP/1.1 200 OK < x-amz-id-2: 8jyT5f7F0UDFpl5EihvNU9gjVYo3y77hYdxGn69GQwX2BeU1CUzBV+xyvImM5xdYBUNTotnoIKY= < x-amz-request-id: 5A6B6011CCA68290 < Date: Thu, 21 Jun 2018 12:39:02 GMT < Last-Modified: Thu, 21 Jun 2018 12:38:57 GMT < ETag: "e262cb567d77be037ca7896b6a1c5a4e" < x-amz-meta-mode: 33184 < x-amz-meta-gid: 0 < x-amz-meta-uid: 0 < x-amz-meta-mtime: 1529584721 < x-amz-version-id: CHylbSlPunUOvAELFlmCM3yGUeCJWqY_ < Accept-Ranges: bytes < Content-Type: application/octet-stream < Content-Length: 268435456 < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] cache.cpp:AddStat(356): add stat cache entry[path=/test/.fuse_hidden0000000500000002] [INF] s3fs.cpp:s3fs_release(2239): [path=/test/.fuse_hidden0000000500000002][fd=14] [INF] fdcache.cpp:GetFdEntity(2020): [path=/test/.fuse_hidden0000000500000002][fd=14] [ERR] s3fs.cpp:s3fs_release(2258): could not find fd(file=/test/.fuse_hidden0000000500000002) [INF] s3fs.cpp:s3fs_unlink(1094): [path=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:DeleteRequest(2482): [tpath=/test/.fuse_hidden0000000500000002] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/.fuse_hidden0000000500000002 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/.fuse_hidden0000000500000002 [INF] curl.cpp:insertV4Headers(2411): computing signature [DELETE] [/test/.fuse_hidden0000000500000002] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > DELETE /test/.fuse_hidden0000000500000002 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=764215b694035c862198564f3995c8f66566a5cd0f18933f27ba2dff4684ab85 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123901Z < HTTP/1.1 204 No Content < x-amz-id-2: Mn+0ylPyaV6dvATxagzUAqMy5xSwra0fVSwFoUxvjjy8MNP2QQfcHwXflY3kq5oH+dD+YPX8/X0= < x-amz-request-id: CFB6D1411D20E122 < Date: Thu, 21 Jun 2018 12:39:02 GMT < x-amz-version-id: E87Xq6qIzFrSc1nYmj6k_fMCGiZualiE < x-amz-delete-marker: true < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 204 [INF] fdcache.cpp:DeleteCacheFile(1820): [path=/test/.fuse_hidden0000000500000002] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/.fuse_hidden0000000500000002] [INF] s3fs.cpp:s3fs_getattr(842): [path=/test] [INF] s3fs.cpp:s3fs_getattr(842): [path=/test/2.DUMMY.1] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/2.DUMMY.1] [INF] curl.cpp:HeadRequest(2719): [tpath=/test/2.DUMMY.1] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/2.DUMMY.1][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2.DUMMY.1 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2.DUMMY.1 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2.DUMMY.1] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/2.DUMMY.1 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d838059818662bafc9de288babaea03b0d72d8dd7a75f2ed89a37d5c7d0ae033 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 404 Not Found < x-amz-request-id: D7A800B110A1795B < x-amz-id-2: GWRgVVLTd8XxBNwQMW91wJLem5RdZzSv6MznWZiHeRoGk4PNEbavaRR2Xqdv+4DsE4Wt8rzW13o= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:39:02 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(2719): [tpath=/test/2.DUMMY.1/] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/2.DUMMY.1/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2.DUMMY.1/ [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2.DUMMY.1/ [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2.DUMMY.1/] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/2.DUMMY.1/ HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3b8522da44f8f16044aa7c07be0219b978d1454bfc8ba7441bedd3c7fbcbeb02 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 404 Not Found < x-amz-request-id: C07A637AA9D04747 < x-amz-id-2: 6X58WWYCRIuNre8Kl5XXFUhMNzoudttFW77w8Mq4scQFIbF4MgNahdZrLrTZNoBadfJ4SBBOLXg= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:39:02 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(2719): [tpath=/test/2.DUMMY.1_$folder$] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/2.DUMMY.1_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/2.DUMMY.1_%24folder%24 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/2.DUMMY.1_%24folder%24 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/2.DUMMY.1_$folder$] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/2.DUMMY.1_%24folder%24 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=1de54be0e917cf2139786ec0f294e60febc144d71de92a15a63ab5a3ec70397c host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 404 Not Found < x-amz-request-id: 0F94A1AC612BD260 < x-amz-id-2: iRdRsYEcyKHvMHsR+k0LJ0XVj2irhgaUs21XlSe1FLTLL5Ud4rY1ZdIGFOvNggYR0DlPgSchsAE= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:39:02 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2476): [path=/test/2.DUMMY.1] [INF] curl.cpp:ListBucketRequest(3114): [tpath=/test/2.DUMMY.1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv?delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/ [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com?delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/ [INF] curl.cpp:insertV4Headers(2411): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > GET /?delimiter=/&max-keys=2&prefix=test/2.DUMMY.1/ HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=17cf0edfb1032129d03d38b79301f36b5d8eda32abe241d07f5271724b1455ee host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 200 OK < x-amz-id-2: kqBrwwyhNuiNGtV9MXPQ9FjCx1XXKhpp4cZtwDHOmPsRCEH8GgklAhbbnw6OH+4xfrKqA9cN23s= < x-amz-request-id: 40B4AF1082FB192C < Date: Thu, 21 Jun 2018 12:39:04 GMT < x-amz-bucket-region: eu-west-1 < Content-Type: application/xml < Transfer-Encoding: chunked < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] cache.cpp:AddNoObjectCache(429): add no object cache entry[path=/test/2.DUMMY.1] [INF] s3fs.cpp:s3fs_getattr(842): [path=/test/1.DUMMY.0] [INF] cache.cpp:DelStat(565): delete stat cache entry[path=/test/1.DUMMY.0] [INF] curl.cpp:HeadRequest(2719): [tpath=/test/1.DUMMY.0] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/1.DUMMY.0][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1.DUMMY.0 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1.DUMMY.0 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1.DUMMY.0] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/1.DUMMY.0 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=22111d76820c087ca0c3a2f6652226484418bc35ca4e4d80c8865f092a2f7fcc host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 404 Not Found < x-amz-request-id: 2A1154028E56ACFA < x-amz-id-2: Sc2JhmPWXTF3V6V6mN9rppyU5Rs4fzclv+6ZV8JAhuSx/4GbAVSstmZmb1b558lr9Y3kjTLrn+A= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:39:02 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(2719): [tpath=/test/1.DUMMY.0/] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/1.DUMMY.0/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1.DUMMY.0/ [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1.DUMMY.0/ [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1.DUMMY.0/] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/1.DUMMY.0/ HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=c23e5da5d5a7ac2942eac97e2455bc8bf42ff37eb4686e363c5f4bf6f30b8952 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 404 Not Found < x-amz-request-id: E497F8EC601675FE < x-amz-id-2: Q9dkp+hwt7FpEsCwL1AQhzYn/7y+hY1fLwkTsX8HNrrgX2Ee2o4Ox3fckJiovaSxNtE8h3UnTVk= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:39:02 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(2719): [tpath=/test/1.DUMMY.0_$folder$] [INF] curl.cpp:PreHeadRequest(2668): [tpath=/test/1.DUMMY.0_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv/test/1.DUMMY.0_%24folder%24 [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com/test/1.DUMMY.0_%24folder%24 [INF] curl.cpp:insertV4Headers(2411): computing signature [HEAD] [/test/1.DUMMY.0_$folder$] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > HEAD /test/1.DUMMY.0_%24folder%24 HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=6f3c7afad984607b9751b915d770d5da7ff415dac5b29a6f503e473d621097ae host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 404 Not Found < x-amz-request-id: 0D9732552F817485 < x-amz-id-2: yIrTANkB6WIxFdHwX9aqgJ+TVKDi6uiPqvpUmLK7RVcPMeH0q4Ko1i1X1SnVitcoCc+xymUxk9M= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Thu, 21 Jun 2018 12:39:02 GMT < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2084): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2476): [path=/test/1.DUMMY.0] [INF] curl.cpp:ListBucketRequest(3114): [tpath=/test/1.DUMMY.0] [INF] curl.cpp:prepare_url(4268): URL is https://s3-eu-west-1.amazonaws.com/s3fs-test-bucket-dv?delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/ [INF] curl.cpp:prepare_url(4300): URL changed is https://s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com?delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/ [INF] curl.cpp:insertV4Headers(2411): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/] [] [INF] curl.cpp:url_to_host(101): url is https://s3-eu-west-1.amazonaws.com * Found bundle for host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com: 0x7f9c84042990 * Re-using existing connection! (#62) with host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com * Connected to s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com (52.218.104.139) port 443 (#62) > GET /?delimiter=/&max-keys=2&prefix=test/1.DUMMY.0/ HTTP/1.1 User-Agent: s3fs/1.83 (commit hash 3bf05da; OpenSSL) Accept: */* Authorization: AWS4-HMAC-SHA256 Credential=AKIAJMCRAISWINXVBJMA/20180621/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e8175651334a2bb879930d8b7300e187f93ef20b1292e547abf1e9d68691b6e1 host: s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20180621T123903Z < HTTP/1.1 200 OK < x-amz-id-2: xQsm45SuSh1604G4VlAmDbvwdbf3m3WFB7h02spMF1seXIBHLNQsV5Ndtgng8vdObkdyvly/XQw= < x-amz-request-id: E1E29D2BFE124DFF < Date: Thu, 21 Jun 2018 12:39:04 GMT < x-amz-bucket-region: eu-west-1 < Content-Type: application/xml < Transfer-Encoding: chunked < Server: AmazonS3 < * Connection #62 to host s3fs-test-bucket-dv.s3-eu-west-1.amazonaws.com left intact [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] cache.cpp:AddNoObjectCache(429): add no object cache entry[path=/test/1.DUMMY.0] ``` ### Details about issue iozone output: ``` iozone -R -I 3 -u 3 -r 1m -s 256m -F /mnt/s3fs/s3fs-test-bucket-dv/test/1 /mnt/s3fs/s3fs-test-bucket-dv/test/2 /mnt/s3fs/s3fs-test-bucket-dv/test/3 Iozone: Performance Test of File I/O Version $Revision: 3.424 $ Compiled for 64 bit mode. Build: linux Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa. Run began: Thu Jun 21 12:37:09 2018 Excel chart generation enabled O_DIRECT feature enabled Record Size 1024 kB File size set to 262144 kB Command line used: iozone -R -I -u 3 -r 1m -s 256m -F /mnt/s3fs/s3fs-test-bucket-dv/test/1 /mnt/s3fs/s3fs-test-bucket-dv/test/2 /mnt/s3fs/s3fs-test-bucket-dv/test/3 3 Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Min process = 1 Max process = 3 Throughput test with 1 process Each process writes a 262144 kByte file in 1024 kByte records Children see throughput for 1 initial writers = 1197294.88 kB/sec Parent sees throughput for 1 initial writers = 49652.62 kB/sec Min throughput per process = 1197294.88 kB/sec Max throughput per process = 1197294.88 kB/sec Avg throughput per process = 1197294.88 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 rewriters = 1359047.88 kB/sec Parent sees throughput for 1 rewriters = 37622.45 kB/sec Min throughput per process = 1359047.88 kB/sec Max throughput per process = 1359047.88 kB/sec Avg throughput per process = 1359047.88 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 readers = 2639758.25 kB/sec Parent sees throughput for 1 readers = 2488175.04 kB/sec Min throughput per process = 2639758.25 kB/sec Max throughput per process = 2639758.25 kB/sec Avg throughput per process = 2639758.25 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 re-readers = 2718630.75 kB/sec Parent sees throughput for 1 re-readers = 2512159.94 kB/sec Min throughput per process = 2718630.75 kB/sec Max throughput per process = 2718630.75 kB/sec Avg throughput per process = 2718630.75 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 reverse readers = 2654787.50 kB/sec Parent sees throughput for 1 reverse readers = 2421054.24 kB/sec Min throughput per process = 2654787.50 kB/sec Max throughput per process = 2654787.50 kB/sec Avg throughput per process = 2654787.50 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 stride readers = 2623539.00 kB/sec Parent sees throughput for 1 stride readers = 2454328.35 kB/sec Min throughput per process = 2623539.00 kB/sec Max throughput per process = 2623539.00 kB/sec Avg throughput per process = 2623539.00 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 random readers = 2597544.75 kB/sec Parent sees throughput for 1 random readers = 2390191.72 kB/sec Min throughput per process = 2597544.75 kB/sec Max throughput per process = 2597544.75 kB/sec Avg throughput per process = 2597544.75 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 mixed workload = 2626246.25 kB/sec Parent sees throughput for 1 mixed workload = 2476161.12 kB/sec Min throughput per process = 2626246.25 kB/sec Max throughput per process = 2626246.25 kB/sec Avg throughput per process = 2626246.25 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 random writers = 1362510.50 kB/sec Parent sees throughput for 1 random writers = 35409.74 kB/sec Min throughput per process = 1362510.50 kB/sec Max throughput per process = 1362510.50 kB/sec Avg throughput per process = 1362510.50 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 pwrite writers = 1176715.00 kB/sec Parent sees throughput for 1 pwrite writers = 53749.58 kB/sec Min throughput per process = 1176715.00 kB/sec Max throughput per process = 1176715.00 kB/sec Avg throughput per process = 1176715.00 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 pread readers = 2671883.50 kB/sec Parent sees throughput for 1 pread readers = 2468306.59 kB/sec Min throughput per process = 2671883.50 kB/sec Max throughput per process = 2671883.50 kB/sec Avg throughput per process = 2671883.50 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 fwriters = 235445.19 kB/sec Parent sees throughput for 1 fwriters = 39019.24 kB/sec Min throughput per process = 235445.19 kB/sec Max throughput per process = 235445.19 kB/sec Avg throughput per process = 235445.19 kB/sec Min xfer = 262144.00 kB Children see throughput for 1 freaders = 1754058.00 kB/sec Parent sees throughput for 1 freaders = 1746893.07 kB/sec Min throughput per process = 1754058.00 kB/sec Max throughput per process = 1754058.00 kB/sec Avg throughput per process = 1754058.00 kB/sec Min xfer = 262144.00 kB Each process writes a 262144 kByte file in 1024 kByte records Children see throughput for 2 initial writers = 1675515.56 kB/sec Parent sees throughput for 2 initial writers = 76689.69 kB/sec Min throughput per process = 829283.50 kB/sec Max throughput per process = 846232.06 kB/sec Avg throughput per process = 837757.78 kB/sec Min xfer = 257024.00 kB Children see throughput for 2 rewriters = 1741225.56 kB/sec Parent sees throughput for 2 rewriters = 87488.14 kB/sec Min throughput per process = 828102.25 kB/sec Max throughput per process = 913123.31 kB/sec Avg throughput per process = 870612.78 kB/sec Min xfer = 238592.00 kB Children see throughput for 2 readers = 1747556.62 kB/sec Parent sees throughput for 2 readers = 1249274.95 kB/sec Min throughput per process = 0.00 kB/sec Max throughput per process = 1747556.62 kB/sec Avg throughput per process = 873778.31 kB/sec Min xfer = 0.00 kB Children see throughput for 2 re-readers = 1040242.66 kB/sec Parent sees throughput for 2 re-readers = 820165.62 kB/sec Min throughput per process = 233920.78 kB/sec Max throughput per process = 806321.88 kB/sec Avg throughput per process = 520121.33 kB/sec Min xfer = 102400.00 kB Error in file: Found ?0? Expecting ?f4f4f4f4f4f4f4f4? addr 7f725af00000 Error in file: Position 267386880 Record # 255 Record size 1024 kb where 0x7f725af00000x loop 0 Children see throughput for 2 reverse readers = 2558754.50 kB/sec Parent sees throughput for 2 reverse readers = 53573.57 kB/sec Min throughput per process = 0.00 kB/sec Max throughput per process = 2558754.50 kB/sec Avg throughput per process = 1279377.25 kB/sec Min xfer = 0.00 kB ^C iozone: interrupted "Throughput report Y-axis is type of test X-axis is number of processes" "Record size = 1024 kBytes " "Output is in kBytes/sec" " Initial write " 1197294.88 1675515.56 " Rewrite " 1359047.88 1741225.56 " Read " 2639758.25 1747556.62 " Re-read " 2718630.75 1040242.66 " Reverse Read " 2654787.50 2558754.50 exiting iozone ```

kerem closed this issue

2026-03-04 01:45:40 +03:00

kerem commented

2026-03-04 01:45:41 +03:00

Author

Owner

@gaul commented on GitHub (Jan 24, 2019):

Could you test again against master? #918 addresses symptoms similar to this issue.

@gaul commented on GitHub (Jan 24, 2019): Could you test again against master? #918 addresses symptoms similar to this issue.

kerem commented

2026-03-04 01:45:41 +03:00

Author

Owner

@gaul commented on GitHub (Apr 9, 2019):

Closing due to inactivity. Please reopen if symptoms persist.

@gaul commented on GitHub (Apr 9, 2019): Closing due to inactivity. Please reopen if symptoms persist.