[GH-ISSUE #1068] Consider using the AWS C++ SDK

kerem commented

2026-03-04 01:46:57 +03:00

Owner

Originally created by @gaul on GitHub (Jul 2, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1068

AWS provides an SDK which could eliminate 1/4 of all s3fs lines of code:

https://aws.amazon.com/sdk-for-cpp/

Using this has drawbacks including forcing a C++11 dependency and likely less SSL library flexibility.

Originally created by @gaul on GitHub (Jul 2, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1068 AWS provides an SDK which could eliminate 1/4 of all s3fs lines of code: https://aws.amazon.com/sdk-for-cpp/ Using this has drawbacks including forcing a C++11 dependency and likely less SSL library flexibility.

kerem commented

2026-03-04 01:46:58 +03:00

Author

Owner

@gaul commented on GitHub (Aug 9, 2020):

s3fs currently has an implicit dependency on RHEL 7. Here is a chart of enterprise distribution GCC versions:

Distro	GCC
Debian 7	4.7.2
Debian 8	4.9.2
Debian 9	6.3.0
Debian 10	8.3.0
RHEL 6	4.4.7
RHEL 7	4.8.5
RHEL 8	8.3.1
Ubuntu 14.04	4.8.2
Ubuntu 16.04	5.3.1
Ubuntu 18.04	7.3.0
Ubuntu 20.04	9.3.0

It appears that all of C++11 was implemented in GCC in 4.8:

https://gcc.gnu.org/projects/cxx-status.html#cxx11

However, the AWS SDK has a minimum requirement of GCC 4.9:

https://github.com/aws/aws-sdk-cpp/issues/1198#issuecomment-513438794

@gaul commented on GitHub (Aug 9, 2020): s3fs currently has an implicit dependency on RHEL 7. Here is a chart of enterprise distribution GCC versions: | Distro | GCC | | --- | --- | | Debian 7 | 4.7.2 | | Debian 8 | 4.9.2 | | Debian 9 | 6.3.0 | | Debian 10 | 8.3.0 | | RHEL 6 | 4.4.7 | | RHEL 7 | 4.8.5 | | RHEL 8 | 8.3.1 | | Ubuntu 14.04 | 4.8.2 | | Ubuntu 16.04 | 5.3.1 | | Ubuntu 18.04 | 7.3.0 | | Ubuntu 20.04 | 9.3.0 | It appears that all of C++11 was implemented in GCC in 4.8: https://gcc.gnu.org/projects/cxx-status.html#cxx11 However, the AWS SDK has a minimum requirement of GCC 4.9: https://github.com/aws/aws-sdk-cpp/issues/1198#issuecomment-513438794

kerem commented

2026-03-04 01:46:58 +03:00

Author

Owner

@gaul commented on GitHub (Aug 9, 2020):

Using the official SDK might help work around the many IAM issues that users report.

@gaul commented on GitHub (Aug 9, 2020): Using the official SDK might help work around the many IAM issues that users report.

kerem commented

2026-03-04 01:46:58 +03:00

Author

Owner

@mariadb-JeffBachtel commented on GitHub (Nov 19, 2020):

Additional motivation: s3fs currently does not implement the AWS_WEB_IDENTITY_TOKEN_FILE environment variable auth chain (used with service-linked roles in EKS), which is available in recent AWS SDKs.

@mariadb-JeffBachtel commented on GitHub (Nov 19, 2020): Additional motivation: s3fs currently does not implement the `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable auth chain (used with service-linked roles in EKS), which is available in recent AWS SDKs.

kerem commented

2026-03-04 01:46:58 +03:00

Author

Owner

@mariadb-JeffBachtel commented on GitHub (Nov 19, 2020):

I don't know if this is helpful, but RHEL7 has devtoolset-8 which provides GCC 8 for use. I'm sure it'll be annoying for people compiling from scratch to install that as a prerequisite, but it would allow usable RHEL7 builds for folks, if s3fs were using the aws cpp sdk.

@mariadb-JeffBachtel commented on GitHub (Nov 19, 2020): I don't know if this is helpful, but RHEL7 has `devtoolset-8` which provides GCC 8 for use. I'm sure it'll be annoying for people compiling from scratch to install that as a prerequisite, but it would allow usable RHEL7 builds for folks, if s3fs were using the aws cpp sdk.

kerem commented

2026-03-04 01:46:58 +03:00

Author

Owner

@gaul commented on GitHub (Apr 29, 2021):

Ubuntu 16.04 LTS reaches end of standard support this month which removes a potential objection to requiring C++ 11.

@gaul commented on GitHub (Apr 29, 2021): Ubuntu 16.04 LTS reaches end of standard support this month which removes a potential objection to requiring C++ 11.

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@gaul commented on GitHub (Jul 25, 2021):

Using the AWS SDK would allow more options to configure credentials. References #1122, #1290, and #1724.

@gaul commented on GitHub (Jul 25, 2021): Using the AWS SDK would allow more options to configure credentials. References #1122, #1290, and #1724.

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@gaul commented on GitHub (Jan 27, 2022):

I would like to prototype using the SDK to see how much work it might be to port s3fs. One of the biggest problems is how to handle the dependency. Could we use a git submodule to bring in the external repo? This would allow us to compile it as if it were our own code and statically link the library. After doing this we can experiment with a single RPC like check_bucket.

@gaul commented on GitHub (Jan 27, 2022): I would like to prototype using the SDK to see how much work it might be to port s3fs. One of the biggest problems is how to handle the dependency. Could we use a git submodule to bring in the external repo? This would allow us to compile it as if it were our own code and statically link the library. After doing this we can experiment with a single RPC like `check_bucket`.

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 29, 2022):

I would like to confirm the following in addition to @gaul opinion.

First, is aws-sdk-cpp provided as a package?
(I think that it is not provided as a binary from the official or equivalent repository)

And if it is not distributed as a package, should s3fs be provided as a package containing aws-sdk-cpp?
I think we can include it as a DSO(in this case, I think it's okay if the redistributable of aws-sdk-cpp is permitted by the license), or statically link it.

@ggtakec commented on GitHub (Jan 29, 2022): I would like to confirm the following in addition to @gaul opinion. First, is aws-sdk-cpp provided as a package? (I think that it is not provided as a binary from the official or equivalent repository) And if it is not distributed as a package, should s3fs be provided as a package containing aws-sdk-cpp? I think we can include it as a DSO(in this case, I think it's okay if the redistributable of aws-sdk-cpp is permitted by the license), or statically link it.

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@gaul commented on GitHub (Jan 29, 2022):

The AWS SDK uses Apache 2.0 which GNU says is not compatible with GPL 2:

https://www.gnu.org/licenses/license-list.en.html#apache2

s3fs specifies GPL 2 or later, which s3fs specifies. Thus we would effectively relicense to GPL 3 or later and probably should change the license headers to reflect this.

I don't think that any distributions have packages for the SDK, and certainly not all of the platforms we support. I think static linking is the easiest option and to treat it as part of our codebase. Can we look at what other projects do?

@gaul commented on GitHub (Jan 29, 2022): The AWS SDK uses Apache 2.0 which GNU says is _not_ compatible with GPL 2: https://www.gnu.org/licenses/license-list.en.html#apache2 s3fs specifies GPL 2 or later, which s3fs specifies. Thus we would effectively relicense to GPL 3 or later and probably should change the license headers to reflect this. I don't think that any distributions have packages for the SDK, and certainly not all of the platforms we support. I think static linking is the easiest option and to treat it as part of our codebase. Can we look at what other projects do?

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 29, 2022):

Whether user use vcpkg or clone yourself, user will be building aws-sdk-cpp locally now.
If we follow this, I think s3fs will also be a just-in-time build at install.

One of my suggestions is to wrap the functions using aws-sdk-cpp, make it a DSO, and distribute this DSO.
NOTE: This proposal assumes that binaries containing aws-sdk-cpp can be distributed.
By setting the Apache2.0 license to this DSO, I think we'll be able to eliminate the mix of source code in s3fs.

This is just one suggestion yet.

@ggtakec commented on GitHub (Jan 29, 2022): Whether user use vcpkg or clone yourself, user will be building aws-sdk-cpp locally now. If we follow this, I think s3fs will also be a just-in-time build at install. One of my suggestions is to wrap the functions using aws-sdk-cpp, make it a DSO, and distribute this DSO. _NOTE: This proposal assumes that binaries containing aws-sdk-cpp can be distributed._ By setting the Apache2.0 license to this DSO, I think we'll be able to eliminate the mix of source code in s3fs. This is just one suggestion yet.

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 18, 2022):

I posted the PR(#1927) with code that made it possible to use the aws-sdk-cpp library for credential(token) processing.
The process using the aws-sdk-cpp library is specified as a shared library by s3fs option(credlib), and it is loaded and used in s3fs.
If anyone wants to test this, you can give it a try.
Thanks in advance for your assistance.

@ggtakec commented on GitHub (Mar 18, 2022): I posted the PR(#1927) with code that made it possible to use the `aws-sdk-cpp` library for credential(token) processing. The process using the `aws-sdk-cpp` library is specified as a shared library by s3fs option(`credlib`), and it is loaded and used in s3fs. If anyone wants to test this, you can give it a try. Thanks in advance for your assistance.

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@ggtakec commented on GitHub (Oct 22, 2022):

Sorry for the delay in replying.

I merged PR #1927 into master branch.
I hope this fix will be one of the solutions for this issue.

This change allows authentication logic implemented in DSO to be loaded at startup and built into s3fs.
(Note that s3fs-fuse does not provide this DSO)
Newly added options(credlib and credlib_opts) can be used to load the DSO.

See the repository below for an example of this DSO:
https://github.com/ggtakec/s3fs-fuse-awscred-lib
This repository code allows you to wrap aws-sdk-cpp and generate a DSO.
This created DSO can be loaded when s3fs-fuse starts using the new option above.

Loading s3fs-fuse-awscred-lib gives you access to all the authentication logic supported by aws-sdk-cpp.
That includes things like Token auto-renewal.

Could anyone please try this?

@ggtakec commented on GitHub (Oct 22, 2022): _Sorry for the delay in replying._ I merged PR #1927 into master branch. I hope this fix will be one of the solutions for this issue. This change allows authentication logic implemented in `DSO` to be loaded at startup and built into s3fs. _(Note that s3fs-fuse does not provide this `DSO`)_ Newly added options(`credlib` and `credlib_opts`) can be used to load the `DSO`. See the repository below for an example of this `DSO`: https://github.com/ggtakec/s3fs-fuse-awscred-lib This repository code allows you to wrap `aws-sdk-cpp` and generate a `DSO`. This created `DSO` can be loaded when s3fs-fuse starts using the new option above. Loading `s3fs-fuse-awscred-lib` gives you access to all the authentication logic supported by `aws-sdk-cpp`. That includes things like Token auto-renewal. Could anyone please try this?

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@sheromon commented on GitHub (Jun 23, 2023):

Seems like this issue should be closed since #1927 has been merged?

@sheromon commented on GitHub (Jun 23, 2023): Seems like this issue should be closed since #1927 has been merged?

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@gaul commented on GitHub (Jun 24, 2023):

I doubt many users actually configure the credential library. I prefer that s3fs relies directly on the AWS library to generally improve compatibility and reduce the overall maintenance burden.

@gaul commented on GitHub (Jun 24, 2023): I doubt many users actually configure the credential library. I prefer that s3fs relies directly on the AWS library to generally improve compatibility and reduce the overall maintenance burden.

kerem commented

2026-03-04 01:46:59 +03:00

Author

Owner

@ggtakec commented on GitHub (Jun 25, 2023):

@sheromon I commented at https://github.com/s3fs-fuse/s3fs-fuse/issues/1290#issuecomment-1606045109
Please see it.
Thanks.

@ggtakec commented on GitHub (Jun 25, 2023): @sheromon I commented at https://github.com/s3fs-fuse/s3fs-fuse/issues/1290#issuecomment-1606045109 Please see it. Thanks.

kerem referenced this issue

2026-03-04 02:01:03 +03:00

[PR #585] [MERGED] Fixed failure to upload/copy with SSE_C and SSE_KMS #1558

Rows
Columns

[GH-ISSUE #1068] Consider using the AWS C++ SDK #585