[GH-ISSUE #2639] Boto3 upgrade - MissingContentMD5) when calling the DeleteObjects operation: Missing required header for this request: Content-Md5. #1259

New issue

Closed

opened 2026-03-04 01:52:36 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:52:36 +03:00

Owner

Originally created by @andretheronsa on GitHub (Jan 22, 2025).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2639

We use fspec, aiobotocore & s3fs to manage files on our Minio S3 buckets.

We do a move operation like so

fs = fsspec.core.url_to_fs(src, s3={
  "client_kwargs": {
     "endpoint_url"=minio_url,
     "verify": False
    },
  "key": access_key,
  "secret": secret
})
fs.cp(src, dst, recursive=True)

With these versions it works:
aiobotocore==2.17.0
boto3==1.35.93
s3transfer==0.10.4

With these we get the following error:
aiobotocore==2.18.0
boto3==1.36.
s3transfer==0.11.1

Traceback (most recent call last):
  File "/local/lib/python3.11/site-packages/s3fs/core.py", line 114, in _error_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/local/lib/python3.11/site-packages/aiobotocore/client.py", line 412, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (MissingContentMD5) when calling the DeleteObjects operation: Missing required header for this request: Content-Md5.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/fsspec/spec.py", line 1193, in mv
    self.rm(path1, recursive=recursive)
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 2052, in _rm
    out = await _run_coros_in_chunks(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 268, in _run_coros_in_chunks
    result, k = await done.pop()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 245, in _run_coro
    return await asyncio.wait_for(coro, timeout=timeout), i
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 452, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 2026, in _bulk_delete
    out = await self._call_s3(
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 371, in _call_s3
    return await _error_wrapper(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 146, in _error_wrapper
    raise err
OSError: [Errno 5] An error occurred (MissingContentMD5) when calling the DeleteObjects operation: Missing required header for this request: Content-Md5.

I suspect these changes from boto3 1.35 to 1.36 are responsible.

feature:s3: [botocore] The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using the when_supported and when_required options - in code using the response_checksum_validation parameter for botocore.config.Config, in the shared AWS config file using response_checksum_validation, or as an env variable using AWS_RESPONSE_CHECKSUM_VALIDATION.

feature:s3: [botocore] Added support for the CRC64NVME checksum algorithm in the S3 client through the optional AWS CRT (awscrt) dependency.

feature:s3: [botocore] S3 client behavior is updated to always calculate a CRC32 checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). Checksum behavior can be configured using when_supported and when_required options - in code using the request_checksum_calculation parameter for botocore.config.Config, in the shared AWS config file using request_checksum_calculation, or as an env variable using AWS_REQUEST_CHECKSUM_CALCULATION. Note: Botocore will no longer automatically compute and populate the Content-MD5 header.

I am not 100% this library is the cause. This s3transfer issue seem related.

Originally created by @andretheronsa on GitHub (Jan 22, 2025). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2639 We use fspec, aiobotocore & s3fs to manage files on our Minio S3 buckets. We do a move operation like so ``` fs = fsspec.core.url_to_fs(src, s3={ "client_kwargs": { "endpoint_url"=minio_url, "verify": False }, "key": access_key, "secret": secret }) fs.cp(src, dst, recursive=True) ``` With these versions it works: aiobotocore==2.17.0 boto3==1.35.93 s3transfer==0.10.4 With these we get the following error: aiobotocore==2.18.0 boto3==1.36. s3transfer==0.11.1 ``` Traceback (most recent call last): File "/local/lib/python3.11/site-packages/s3fs/core.py", line 114, in _error_wrapper return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/local/lib/python3.11/site-packages/aiobotocore/client.py", line 412, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (MissingContentMD5) when calling the DeleteObjects operation: Missing required header for this request: Content-Md5. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python3.11/site-packages/fsspec/spec.py", line 1193, in mv self.rm(path1, recursive=recursive) File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper return sync(self.loop, func, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 103, in sync raise return_result File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner result[0] = await coro ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 2052, in _rm out = await _run_coros_in_chunks( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 268, in _run_coros_in_chunks result, k = await done.pop() ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/fsspec/asyn.py", line 245, in _run_coro return await asyncio.wait_for(coro, timeout=timeout), i ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/tasks.py", line 452, in wait_for return await fut ^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 2026, in _bulk_delete out = await self._call_s3( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 371, in _call_s3 return await _error_wrapper( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/s3fs/core.py", line 146, in _error_wrapper raise err OSError: [Errno 5] An error occurred (MissingContentMD5) when calling the DeleteObjects operation: Missing required header for this request: Content-Md5. ``` I suspect [these changes](https://github.com/boto/boto3/blob/1e2006c3c5b8d702b3ec21f939a684ed5f1ad666/CHANGELOG.rst#L35) from boto3 1.35 to 1.36 are responsible. > * feature:``s3``: [``botocore``] The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using the ``when_supported`` and ``when_required`` options - in code using the ``response_checksum_validation`` parameter for ``botocore.config.Config``, in the shared AWS config file using ``response_checksum_validation``, or as an env variable using ``AWS_RESPONSE_CHECKSUM_VALIDATION``. > * feature:``s3``: [``botocore``] Added support for the CRC64NVME checksum algorithm in the S3 client through the optional AWS CRT (``awscrt``) dependency. > * feature:``s3``: [``botocore``] S3 client behavior is updated to always calculate a CRC32 checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). Checksum behavior can be configured using ``when_supported`` and ``when_required`` options - in code using the ``request_checksum_calculation`` parameter for ``botocore.config.Config``, in the shared AWS config file using ``request_checksum_calculation``, or as an env variable using ``AWS_REQUEST_CHECKSUM_CALCULATION``. Note: Botocore will no longer automatically compute and populate the Content-MD5 header. I am not 100% this library is the cause. This [s3transfer issue](https://github.com/boto/s3transfer/issues/327) seem related.

kerem closed this issue

2026-03-04 01:52:36 +03:00

kerem commented

2026-03-04 01:52:37 +03:00

Author

Owner

@thispejo commented on GitHub (Feb 25, 2025):

In version 1.37.0 the same error continues.

@thispejo commented on GitHub (Feb 25, 2025): In version 1.37.0 the same error continues.

kerem commented

2026-03-04 01:52:37 +03:00

Author

Owner

@juliogonzalez commented on GitHub (Feb 25, 2025):

I am not 100% sure, but... @thispejo @andretheronsa could it be you are reporting your issue at the wrong repository?

Looks to me your issue should be reported at https://github.com/fsspec/s3fs/, which is written in Python and indeed uses aibotocore and boto3.

s3fs-fuse is written in C and uses FUSE to be able to mount buckets at Unix-like systems.

@juliogonzalez commented on GitHub (Feb 25, 2025): I am not 100% sure, but... @thispejo @andretheronsa could it be you are reporting your issue at the wrong repository? Looks to me your issue should be reported at https://github.com/fsspec/s3fs/, which is written in Python and indeed uses `aibotocore` and `boto3`. `s3fs-fuse` is written in C and uses FUSE to be able to mount buckets at Unix-like systems.