Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace minio client with aws sdk #547

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

mguella
Copy link
Contributor

@mguella mguella commented Apr 16, 2024

The Minio client has issues connecting to different providers (e.g. Cloudflare R2, see minio/minio-js#619 (comment)). Moving to the AWS SDK solves that problem.
This should also support files up to 5GB so it helps with #479 but doesn't solve it completely, as for bigger ones (up to 5TB) the multipart upload is probably needed.
However it's a first step toward solving the issue of big files and it's already a big improvement over the current Minio client as it allows to use S3 compatible providers different from AWS and Minio.

@ikreymer
Copy link
Member

Thanks, we're just doing some additional testing and should be able to merge this soon.

It does seem like Minio's fputObject will not do automated multi-part upload, while AWS S3 Client has:
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property which is a more higher-level command that can automatically use multipart upload if necessary. That may address some of the remaining issues, if we were to switch to use that.

@ikreymer
Copy link
Member

It does seem like Minio's fputObject will not do automated multi-part upload

Er, no, that's not quite right. fPutObject does support multi-part upload (was it working previously?) https://min.io/docs/minio/linux/developers/javascript/API.html#fPutObject so we probably do want to switch to the corresponding AWS S3 API to ensure that we don't break larger uploads. The WACZ files can get large (and there's no sizeLimit by default)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants