[GH-ISSUE #777] s3fs Memory Error #446

New issue

Closed

opened 2026-03-04 01:45:38 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:45:38 +03:00

Owner

Originally created by @yssefunc on GitHub (Jun 13, 2018).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/777

I am working on AWS EMR Cluster. I have a data on my S3 storage. After I clean my data, i am sending to my S3 storage again via s3fs library. The code works with files which size are between 200-500 mb. However, when i am uploading between 2.0 and 2.5 gb size. The code gives a error which is "MemoryError". Do you guys any ideas or expericence about this issue?

import s3fs
bytes_to_write = nyc_green_20161.to_csv(None).encode()
fs = s3fs.S3FileSystem(key='#', secret='#')
with fs.open('s3://ludditiesnyctaxi/new/2016/yellow/yellow_1.csv', 'wb') as f:
f.write(bytes_to_write)

Originally created by @yssefunc on GitHub (Jun 13, 2018). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/777 I am working on AWS EMR Cluster. I have a data on my S3 storage. After I clean my data, i am sending to my S3 storage again via s3fs library. The code works with files which size are between 200-500 mb. However, when i am uploading between 2.0 and 2.5 gb size. The code gives a error which is "MemoryError". Do you guys any ideas or expericence about this issue? import s3fs bytes_to_write = nyc_green_20161.to_csv(None).encode() fs = s3fs.S3FileSystem(key='#', secret='#') with fs.open('s3://ludditiesnyctaxi/new/2016/yellow/yellow_1.csv', 'wb') as f: f.write(bytes_to_write)

kerem closed this issue

2026-03-04 01:45:39 +03:00

kerem commented

2026-03-04 01:45:39 +03:00

Author

Owner

@gaul commented on GitHub (Jun 13, 2018):

@yssefunc Please close this issue and follow up with the Python s3fs project. Unfortunately we share the same name which cause confusion.

@gaul commented on GitHub (Jun 13, 2018): @yssefunc Please close this issue and follow up with the [Python s3fs](https://github.com/dask/s3fs) project. Unfortunately we share the same name which cause confusion.

kerem commented

2026-03-04 01:45:39 +03:00

Author

Owner

@yssefunc commented on GitHub (Jun 13, 2018):

Ok..

@yssefunc commented on GitHub (Jun 13, 2018): Ok..