starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00

[GH-ISSUE #1474] Encryption and Permission Issues with use_sse=kmsid (RHEL7 EC2 AWS) #777

New issue

Open

opened 2026-03-04 01:48:41 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:48:41 +03:00

Owner

Originally created by @robzdeloitte on GitHub (Nov 12, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1474

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.86 (commit:5614155) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse)

2.9.2

Kernel information (uname -r)

3.10.0-1127.18.2.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

Red Hat Enterprise Linux Server release 7.8 (Maipo)

s3fs command line used, if applicable

try 1: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse masked-bucket-name my-directory/

try 2: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse=kmsid masked-bucket-name my-directory/

in attempt 2 we have added the AWSSSEKMSID env variable

/etc/fstab entry, if applicable

n/a

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

no relevant logs

Details about issue

Overall context: Our ec2 instance is attempting to put an object into a S3 bucket, which is mounted to a directory using s3fs. A Lambda function then executes the s3 copy command to move this file to a new S3 bucket. Both S3 buckets in this scenario are encrypted with a KMS master key from an external AWS account. This instance is able to successfully copy files to these buckets using aws s3 cp some-file.txt s3://our-bucket --sse aws:kms, but we run into issues with our second attempt at the s3fs command, where we observe that we can see objects from S3 and move them from the mount directory onto another location in the server, but receive errors when attempting to mv or cp a file into it (permission denied). The goal would be to understand what the appropriate equivalent to the above cli command with aws:kms would be.

We are facing two separate situations:

Try 1:

try1: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse masked-bucket-name my-directory/

When we use -o use_sse everything works fine - we can put objects into a bucket, but receive a (AccessDenied) when calling the CopyObject operation: Access Denied message during the execution of the Lambda function. We are worried that because we are not specific about the key being used we are not encrypting the object properly. For this scenario we would like to confirm if the approach to the put for the encrypted object is correct.

**Try 2 **

try 2: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse=kmsid masked-bucket-name my-directory/

In this scenario we have specified the environment variable for the KMS ID, which is the full arn of the KMS master key for the external AWS account that is used to encrypt the two buckets. When we mount using this command, we are able to pull objects from this mounted directory, but we are unable to put objects into it. We receive permission denied when it happens:

[root@our-server-hostname our-mounted-directory]# mv example-file-name our-mounted-directory/ mv: cannot create regular file 'example-file-name': Operation not permitted

Goal

Given our situation, we are hoping to identify which path we should be chasing and how to appropriately adjust for the correct path. Thanks in advance for any help!

Originally created by @robzdeloitte on GitHub (Nov 12, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1474 #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.86 (commit:5614155) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse) 2.9.2 #### Kernel information (uname -r) 3.10.0-1127.18.2.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) Red Hat Enterprise Linux Server release 7.8 (Maipo) #### s3fs command line used, if applicable ``` try 1: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse masked-bucket-name my-directory/ try 2: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse=kmsid masked-bucket-name my-directory/ in attempt 2 we have added the AWSSSEKMSID env variable ``` #### /etc/fstab entry, if applicable n/a #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ no relevant logs ### Details about issue **Overall context**: Our ec2 instance is attempting to put an object into a S3 bucket, which is mounted to a directory using s3fs. A Lambda function then executes the s3 copy command to move this file to a new S3 bucket. Both S3 buckets in this scenario are encrypted with a KMS master key from an external AWS account. This instance is able to successfully copy files to these buckets using `aws s3 cp some-file.txt s3://our-bucket --sse aws:kms`, but we run into issues with our second attempt at the s3fs command, where we observe that we can see objects from S3 and move them from the mount directory onto another location in the server, but receive errors when attempting to mv or cp a file into it (permission denied). **The goal would be to understand what the appropriate equivalent to the above cli command with aws:kms would be.** We are facing two separate situations: **Try 1**: `try1: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse masked-bucket-name my-directory/` When we use `-o use_sse` everything works fine - we can put objects into a bucket, but receive a `(AccessDenied) when calling the CopyObject operation: Access Denied` message during the execution of the Lambda function. We are worried that because we are not specific about the key being used we are not encrypting the object properly. **For this scenario we would like to confirm if the approach to the put for the encrypted object is correct.** **Try 2 ** `try 2: s3fs -o iam_role='masked-iam-role-name' -o 'nonempty' -o use_sse=kmsid masked-bucket-name my-directory/` In this scenario we have specified the environment variable for the KMS ID, which is the full arn of the KMS master key for the external AWS account that is used to encrypt the two buckets. When we mount using this command, we are able to pull objects from this mounted directory, but we are unable to put objects into it. We receive permission denied when it happens: `[root@our-server-hostname our-mounted-directory]# mv example-file-name our-mounted-directory/ mv: cannot create regular file 'example-file-name': Operation not permitted` **Goal** Given our situation, we are hoping to identify which path we should be chasing and how to appropriately adjust for the correct path. Thanks in advance for any help!

kerem commented

2026-03-04 01:48:42 +03:00

Author

Owner

@gaul commented on GitHub (Nov 15, 2020):

@robzdeloitte I successfully interacted with s3fs using kms after some trial and error. My issue was both using a bucket in the wrong region as my KMS ID and using the incorrect format for the kms id. The latter should have the form xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. You can get more debugging information from s3fs by running s3fs -f -d which should display any error logging.

@gaul commented on GitHub (Nov 15, 2020): @robzdeloitte I successfully interacted with s3fs using kms after some trial and error. My issue was both using a bucket in the wrong region as my KMS ID and using the incorrect format for the kms id. The latter should have the form `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`. You can get more debugging information from s3fs by running `s3fs -f -d` which should display any error logging.

kerem commented

2026-03-04 01:48:42 +03:00

Author

Owner

@robzdeloitte commented on GitHub (Nov 16, 2020):

Thanks for your response @gaul!

Ideally we would like to get option 2 (try 2) listed above working. The good news I see is at the IAM role was applied sucessfully. I ran the following command and received the following output. While this was running, I tried to copy a file s3fs-logs-test into the bucket, but received this error: cp: cannot create regular file 'my-mount-directory/s3fs-logs-test': Operation not permitted

I also tried to create a file robz123 directly in the directory, which failed.

It appears there were two primary errors:

404 Error

[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT

403 Permission Denied

[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[ERR] curl.cpp:RequestPerform(2436): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6R6H3Z5V1T0HFJ1G</RequestId><HostId>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</HostId></Error>
[INF]       cache.cpp:DelStat(579): delete stat cache entry[path=/s3fs-logs-test]

Full Log Output

Log Output

** Also updated the environment variable for AWSSSEKMSID

[root@hostname working-directory]# s3fs -o iam_role='my-iam-role' -o 'nonempty' -o use_sse=kmsid my-bucket my-directory/ -f -d

[CRT] s3fs.cpp:set_s3fs_log_level(297): change debug level from [CRT] to [INF]
[INF]     s3fs.cpp:set_mountpoint_attribute(4401): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
[INF] s3fs.cpp:s3fs_init(3493): init v1.86(commit:5614155) with OpenSSL
[INF] s3fs.cpp:s3fs_check_service(3828): check services.
[INF]       curl.cpp:GetIAMCredentials(2862): [IAM role=XXXXXXXXXXXXXXXXX]
[WAN] curl.cpp:ResetHandle(2044): The CURLOPT_SSL_ENABLE_ALPN option could not be unset. S3 server does not support ALPN, then this option should be disabled to maximize performance. you need to use libcurl 7.36.0 or later.
[WAN] curl.cpp:ResetHandle(2047): The S3FS_CURLOPT_KEEP_SENDING_ON_ERROR option could not be set. For maximize performance you need to enable this option and you should use libcurl 7.51.0 or later.
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:SetIAMCredentials(1870): IAM credential response = "{
  "Code" : "Success",
  "LastUpdated" : "2020-11-16T19:54:06Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "XXXXXXXXXXXXXXXXX",
  "SecretAccessKey" : "XXXXXXXXXXXXXXXXX",
  "Token" : "XXXXXXXXXXXXXXXXX",
  "Expiration" : "2020-11-17T02:08:24Z"
}"
[INF] curl.cpp:ReturnHandler(318): Pool full: destroy the oldest handler
[INF]       curl.cpp:CheckBucket(3413): check a bucket.
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/s3fs-logs-test]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF] s3fs.cpp:s3fs_getattr(876): [path=/s3fs-logs-test]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF] s3fs.cpp:s3fs_getattr(876): [path=/s3fs-logs-test]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF] s3fs.cpp:s3fs_create(1061): [path=/s3fs-logs-test][mode=100644][flags=0x80c1]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]     s3fs.cpp:create_file_object(1018): [path=/s3fs-logs-test][mode=100644]
[INF]       curl.cpp:PutRequest(3218): [tpath=/s3fs-logs-test]
[INF]       curl.cpp:PutRequest(3236): create zero byte file object.
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test
[INF]       curl.cpp:PutRequest(3319): uploading... [path=/s3fs-logs-test][fd=-1][size=0]
[INF]       curl.cpp:insertV4Headers(2753): computing signature [PUT] [/s3fs-logs-test] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[ERR] curl.cpp:RequestPerform(2436): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6R6H3Z5V1T0HFJ1G</RequestId><HostId>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</HostId></Error>
[INF]       cache.cpp:DelStat(579): delete stat cache entry[path=/s3fs-logs-test]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_access(3560): [path=/][mask=X_OK ]
[INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800]
[INF] s3fs.cpp:s3fs_readdir(2553): [path=/]
[INF]   s3fs.cpp:list_bucket(2596): [path=/]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]   s3fs.cpp:readdir_multi_head(2469): [path=/][list=0]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/keytest.txt][bpath=keytest.txt][save=/keytest.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest.txt
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/keytest2.txt][bpath=keytest2.txt][save=/keytest2.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest2.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest2.txt
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/keytest3.txt][bpath=keytest3.txt][save=/keytest3.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest3.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest3.txt
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/s3-logs][bpath=s3-logs][save=/s3-logs][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3-logs
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3-logs
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/testfile.txt][bpath=testfile.txt][save=/testfile.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/testfile.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/testfile.txt
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/xxxx.txt][bpath=xxxx.txt][save=/xxxx.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx.txt
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/xxxx1.txt][bpath=xxxx1.txt][save=/xxxx1.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx1.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx1.txt
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/xxxx2.txt][bpath=xxxx2.txt][save=/xxxx2.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx2.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx2.txt
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/xxxx3.txt][bpath=xxxx3.txt][save=/xxxx3.txt][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx3.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx3.txt
[INF]       curl.cpp:Request(4430): [count=9]
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/testfile.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx1.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest2.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3-logs] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest3.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx3.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx2.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/keytest.txt]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/keytest2.txt]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/keytest3.txt]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/s3-logs]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/testfile.txt]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/xxxx.txt]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/xxxx1.txt]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/xxxx2.txt]
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/xxxx3.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest2.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest3.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/s3-logs]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/testfile.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx1.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx2.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx3.txt]
[INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800]
[INF] s3fs.cpp:s3fs_readdir(2553): [path=/]
[INF]   s3fs.cpp:list_bucket(2596): [path=/]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]   s3fs.cpp:readdir_multi_head(2469): [path=/][list=0]
[INF]       curl.cpp:Request(4430): [count=0]
[INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800]
[INF] s3fs.cpp:s3fs_readdir(2553): [path=/]
[INF]   s3fs.cpp:list_bucket(2596): [path=/]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]   s3fs.cpp:readdir_multi_head(2469): [path=/][list=0]
[INF]       curl.cpp:Request(4430): [count=0]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/rz-123]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/rz-123]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/rz-123]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/rz-123]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/rz-123]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/rz-123]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF] s3fs.cpp:s3fs_open(2170): [path=/keytest.txt][flags=0x8000]
[INF] s3fs.cpp:s3fs_flush(2302): [path=/keytest.txt][fd=0]
[INF]       cache.cpp:DelStat(579): delete stat cache entry[path=/keytest.txt]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/keytest.txt]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/keytest.txt][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest.txt
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest.txt
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest.txt] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]       cache.cpp:AddStat(371): add stat cache entry[path=/keytest.txt]
[INF]       fdcache.cpp:SetMtime(1469): [path=/keytest.txt][fd=15][time=1605125568]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/rz-123]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/rz-123]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/rz-123]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF] s3fs.cpp:s3fs_create(1061): [path=/rz-123][mode=100640][flags=0x80c1]
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123/]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]       curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$]
[INF]       curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24
[INF]       curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT
[INF]   s3fs.cpp:list_bucket(2596): [path=/rz-123]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/rz-123]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]     s3fs.cpp:create_file_object(1018): [path=/rz-123][mode=100640]
[INF]       curl.cpp:PutRequest(3218): [tpath=/rz-123]
[INF]       curl.cpp:PutRequest(3236): create zero byte file object.
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123
[INF]       curl.cpp:PutRequest(3319): uploading... [path=/rz-123][fd=-1][size=0]
[INF]       curl.cpp:insertV4Headers(2753): computing signature [PUT] [/rz-123] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[ERR] curl.cpp:RequestPerform(2436): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>A1114A0375A5CA82</RequestId><HostId>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</HostId></Error>
[INF]       cache.cpp:DelStat(579): delete stat cache entry[path=/rz-123]
[INF] s3fs.cpp:s3fs_flush(2302): [path=/keytest.txt][fd=15]
[INF]       fdcache.cpp:RowFlush(1885): [tpath=][path=/keytest.txt][fd=15]
[INF] s3fs.cpp:s3fs_release(2357): [path=/keytest.txt][fd=15]
[INF]       fdcache.cpp:GetFdEntity(2505): [path=/keytest.txt][fd=15]
[INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800]
[INF] s3fs.cpp:s3fs_readdir(2553): [path=/]
[INF]   s3fs.cpp:list_bucket(2596): [path=/]
[INF]       curl.cpp:ListBucketRequest(3446): [tpath=/]
[INF]       curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(2416): HTTP response code 200
[INF]   s3fs.cpp:readdir_multi_head(2469): [path=/][list=0]
[INF]       curl.cpp:Request(4430): [count=0]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest2.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest3.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/s3-logs]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/testfile.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx1.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx2.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx3.txt]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]
[INF] s3fs.cpp:s3fs_getattr(876): [path=/]

@robzdeloitte commented on GitHub (Nov 16, 2020): Thanks for your response @gaul! Ideally we would like to get option 2 (try 2) listed above working. The good news I see is at the IAM role was applied sucessfully. I ran the following command and received the following output. While this was running, I tried to copy a file **s3fs-logs-test** into the bucket, but received this error: `cp: cannot create regular file 'my-mount-directory/s3fs-logs-test': Operation not permitted` I also tried to create a file **robz123** directly in the directory, which failed. It appears there were two primary errors: **404 Error** ``` [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT ``` **403 Permission Denied** ``` [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [ERR] curl.cpp:RequestPerform(2436): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6R6H3Z5V1T0HFJ1G</RequestId><HostId>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</HostId></Error> [INF] cache.cpp:DelStat(579): delete stat cache entry[path=/s3fs-logs-test] ``` **Full Log Output** ``` Log Output ** Also updated the environment variable for AWSSSEKMSID [root@hostname working-directory]# s3fs -o iam_role='my-iam-role' -o 'nonempty' -o use_sse=kmsid my-bucket my-directory/ -f -d [CRT] s3fs.cpp:set_s3fs_log_level(297): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(4401): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) [INF] s3fs.cpp:s3fs_init(3493): init v1.86(commit:5614155) with OpenSSL [INF] s3fs.cpp:s3fs_check_service(3828): check services. [INF] curl.cpp:GetIAMCredentials(2862): [IAM role=XXXXXXXXXXXXXXXXX] [WAN] curl.cpp:ResetHandle(2044): The CURLOPT_SSL_ENABLE_ALPN option could not be unset. S3 server does not support ALPN, then this option should be disabled to maximize performance. you need to use libcurl 7.36.0 or later. [WAN] curl.cpp:ResetHandle(2047): The S3FS_CURLOPT_KEEP_SENDING_ON_ERROR option could not be set. For maximize performance you need to enable this option and you should use libcurl 7.51.0 or later. [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:SetIAMCredentials(1870): IAM credential response = "{ "Code" : "Success", "LastUpdated" : "2020-11-16T19:54:06Z", "Type" : "AWS-HMAC", "AccessKeyId" : "XXXXXXXXXXXXXXXXX", "SecretAccessKey" : "XXXXXXXXXXXXXXXXX", "Token" : "XXXXXXXXXXXXXXXXX", "Expiration" : "2020-11-17T02:08:24Z" }" [INF] curl.cpp:ReturnHandler(318): Pool full: destroy the oldest handler [INF] curl.cpp:CheckBucket(3413): check a bucket. [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/s3fs-logs-test] [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:s3fs_getattr(876): [path=/s3fs-logs-test] [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:s3fs_getattr(876): [path=/s3fs-logs-test] [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:s3fs_create(1061): [path=/s3fs-logs-test][mode=100644][flags=0x80c1] [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/s3fs-logs-test_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3fs-logs-test_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3fs-logs-test_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/s3fs-logs-test] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/s3fs-logs-test] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=s3fs-logs-test/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=s3fs-logs-test/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:create_file_object(1018): [path=/s3fs-logs-test][mode=100644] [INF] curl.cpp:PutRequest(3218): [tpath=/s3fs-logs-test] [INF] curl.cpp:PutRequest(3236): create zero byte file object. [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3fs-logs-test [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3fs-logs-test [INF] curl.cpp:PutRequest(3319): uploading... [path=/s3fs-logs-test][fd=-1][size=0] [INF] curl.cpp:insertV4Headers(2753): computing signature [PUT] [/s3fs-logs-test] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [ERR] curl.cpp:RequestPerform(2436): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6R6H3Z5V1T0HFJ1G</RequestId><HostId>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</HostId></Error> [INF] cache.cpp:DelStat(579): delete stat cache entry[path=/s3fs-logs-test] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_access(3560): [path=/][mask=X_OK ] [INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800] [INF] s3fs.cpp:s3fs_readdir(2553): [path=/] [INF] s3fs.cpp:list_bucket(2596): [path=/] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:readdir_multi_head(2469): [path=/][list=0] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/keytest.txt][bpath=keytest.txt][save=/keytest.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest.txt [INF] curl.cpp:PreHeadRequest(3009): [tpath=/keytest2.txt][bpath=keytest2.txt][save=/keytest2.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest2.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest2.txt [INF] curl.cpp:PreHeadRequest(3009): [tpath=/keytest3.txt][bpath=keytest3.txt][save=/keytest3.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest3.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest3.txt [INF] curl.cpp:PreHeadRequest(3009): [tpath=/s3-logs][bpath=s3-logs][save=/s3-logs][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/s3-logs [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/s3-logs [INF] curl.cpp:PreHeadRequest(3009): [tpath=/testfile.txt][bpath=testfile.txt][save=/testfile.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/testfile.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/testfile.txt [INF] curl.cpp:PreHeadRequest(3009): [tpath=/xxxx.txt][bpath=xxxx.txt][save=/xxxx.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx.txt [INF] curl.cpp:PreHeadRequest(3009): [tpath=/xxxx1.txt][bpath=xxxx1.txt][save=/xxxx1.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx1.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx1.txt [INF] curl.cpp:PreHeadRequest(3009): [tpath=/xxxx2.txt][bpath=xxxx2.txt][save=/xxxx2.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx2.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx2.txt [INF] curl.cpp:PreHeadRequest(3009): [tpath=/xxxx3.txt][bpath=xxxx3.txt][save=/xxxx3.txt][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/xxxx3.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/xxxx3.txt [INF] curl.cpp:Request(4430): [count=9] [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/testfile.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx1.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest2.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/s3-logs] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest3.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx3.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/xxxx2.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] cache.cpp:AddStat(371): add stat cache entry[path=/keytest.txt] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/keytest2.txt] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/keytest3.txt] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/s3-logs] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/testfile.txt] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/xxxx.txt] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/xxxx1.txt] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/xxxx2.txt] [INF] cache.cpp:AddStat(371): add stat cache entry[path=/xxxx3.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest2.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest3.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/s3-logs] [INF] s3fs.cpp:s3fs_getattr(876): [path=/testfile.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx1.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx2.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx3.txt] [INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800] [INF] s3fs.cpp:s3fs_readdir(2553): [path=/] [INF] s3fs.cpp:list_bucket(2596): [path=/] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:readdir_multi_head(2469): [path=/][list=0] [INF] curl.cpp:Request(4430): [count=0] [INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800] [INF] s3fs.cpp:s3fs_readdir(2553): [path=/] [INF] s3fs.cpp:list_bucket(2596): [path=/] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:readdir_multi_head(2469): [path=/][list=0] [INF] curl.cpp:Request(4430): [count=0] [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/rz-123] [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/rz-123] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/rz-123] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/rz-123] [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/rz-123] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/rz-123] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:s3fs_open(2170): [path=/keytest.txt][flags=0x8000] [INF] s3fs.cpp:s3fs_flush(2302): [path=/keytest.txt][fd=0] [INF] cache.cpp:DelStat(579): delete stat cache entry[path=/keytest.txt] [INF] curl.cpp:HeadRequest(3049): [tpath=/keytest.txt] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/keytest.txt][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/keytest.txt [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/keytest.txt [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/keytest.txt] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] cache.cpp:AddStat(371): add stat cache entry[path=/keytest.txt] [INF] fdcache.cpp:SetMtime(1469): [path=/keytest.txt][fd=15][time=1605125568] [INF] s3fs.cpp:s3fs_getattr(876): [path=/rz-123] [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/rz-123] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/rz-123] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:s3fs_create(1061): [path=/rz-123][mode=100640][flags=0x80c1] [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123/] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123/][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] curl.cpp:HeadRequest(3049): [tpath=/rz-123_$folder$] [INF] curl.cpp:PreHeadRequest(3009): [tpath=/rz-123_$folder$][bpath=][save=][sseckeypos=-1] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123_%24folder%24 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123_%24folder%24 [INF] curl.cpp:insertV4Headers(2753): computing signature [HEAD] [/rz-123_$folder$] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2441): HTTP response code 404 was returned, returning ENOENT [INF] s3fs.cpp:list_bucket(2596): [path=/rz-123] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/rz-123] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=rz-123/ [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=rz-123/] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:create_file_object(1018): [path=/rz-123][mode=100640] [INF] curl.cpp:PutRequest(3218): [tpath=/rz-123] [INF] curl.cpp:PutRequest(3236): create zero byte file object. [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket/rz-123 [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com/rz-123 [INF] curl.cpp:PutRequest(3319): uploading... [path=/rz-123][fd=-1][size=0] [INF] curl.cpp:insertV4Headers(2753): computing signature [PUT] [/rz-123] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [ERR] curl.cpp:RequestPerform(2436): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>A1114A0375A5CA82</RequestId><HostId>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</HostId></Error> [INF] cache.cpp:DelStat(579): delete stat cache entry[path=/rz-123] [INF] s3fs.cpp:s3fs_flush(2302): [path=/keytest.txt][fd=15] [INF] fdcache.cpp:RowFlush(1885): [tpath=][path=/keytest.txt][fd=15] [INF] s3fs.cpp:s3fs_release(2357): [path=/keytest.txt][fd=15] [INF] fdcache.cpp:GetFdEntity(2505): [path=/keytest.txt][fd=15] [INF] s3fs.cpp:s3fs_opendir(2404): [path=/][flags=0x18800] [INF] s3fs.cpp:s3fs_readdir(2553): [path=/] [INF] s3fs.cpp:list_bucket(2596): [path=/] [INF] curl.cpp:ListBucketRequest(3446): [tpath=/] [INF] curl.cpp:prepare_url(4703): URL is https://s3.amazonaws.com/my-bucket?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:prepare_url(4736): URL changed is https://my-bucket.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:insertV4Headers(2753): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] [] [INF] curl.cpp:url_to_host(99): url is https://s3.amazonaws.com [INF] curl.cpp:RequestPerform(2416): HTTP response code 200 [INF] s3fs.cpp:readdir_multi_head(2469): [path=/][list=0] [INF] curl.cpp:Request(4430): [count=0] [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest2.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/keytest3.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/s3-logs] [INF] s3fs.cpp:s3fs_getattr(876): [path=/testfile.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx1.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx2.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/xxxx3.txt] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] [INF] s3fs.cpp:s3fs_getattr(876): [path=/] ```