mirror of
https://github.com/flyimg/flyimg.git
synced 2026-04-25 01:35:56 +03:00
[GH-ISSUE #529] S3 performance #797
Labels
No labels
Docs
Docs
Docs
Security
UnitTest
bug
dependencies
duplicate
enhancement
enhancement
enhancement
hacktoberfest
help wanted
invalid
pull-request
question
stale
version 1
version 2
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/flyimg#797
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @francescor on GitHub (Oct 18, 2024).
Original GitHub issue: https://github.com/flyimg/flyimg/issues/529
Originally assigned to: @sadok-f on GitHub.
Describe the bug
When we enable S3, performace are very slow
To Reproduce
Steps to reproduce the behavior if possible:
https://flyimg-qa.opencityitalia.it is a testing setup with S3 (almost no load), while https://flyimg.opencityitalia.it is the production without S3 (with a lot of load normally).
So far we've been using flyimg with no S3: a week ago we upgraded production to
flyimg/flyimg:1.4.8with S3 enabled: everything worked, but images wherevery slow, so in production we are still using local storage (but we really wish to use S3).Can you suggest us a way to test performance of the two setup, one with S3 and the other with local storage? It would really help an even bash script that we can execute for some kind of beachmark ... something like a
curlwith 100 images: if you suggest something we can help coding itcan you suggest what can we do
@sadok-f commented on GitHub (Oct 21, 2024):
@francescor for the S3 setup, it depends on where the instances of flyimg are running and in which S3 region it stores the images, it might be latency between them.
in Flyimg's root folder, there is a bash script called
benchmark.shwhich benchmarks Flyimg instance locally with 1000 requests with 4 as concurrent requests, usingab - Apache HTTP server benchmarking toolyou can change it to test the performance of the two setups.
@francescor commented on GitHub (Oct 21, 2024):
oh, great!
so, I in our context I guess I have to adapt benchmark.sh by using something like:
not sure if the outcome is the same as your script with docker, actually, anyhow with the above (so just 300 request, vs. 1000) I have this for https://flyimg-qa.opencityitalia.it (2 replicas, all with 100% cpu usage during the test, with S3 enabled)
@francescor commented on GitHub (Oct 21, 2024):
while with https://flyimg.opencityitalia.it (without S3, and with 8 replicas ... so 8 totally independend flyimg containers, none of which reach 100% cpu during the test) the first time I execute the script I see
@francescor commented on GitHub (Oct 21, 2024):
but then, the second time, I have
@francescor commented on GitHub (Oct 21, 2024):
I re-execute the benchmark for the flyimg version in S3, with 8 flyimg intance as the latest above test without S3, and I get
In this context, what
abmetric from the above benchmark should I care most to test difference between adopting S3 for storage vs. local storage?Requests per second?@sadok-f commented on GitHub (Oct 25, 2024):
overall yes you should check the value of
Requests per secondthe more the value the better the performance is.@francescor commented on GitHub (Oct 25, 2024):
thanks, we will (anyhow we realized that the deploy in production with s3 enabled had some other issues, not related to S3, so this issue may be completely irrelevant, sorry. Anyhow wee plan to enable S3 in the next 10 days in the production infrastructure, at that time we'll have enough data for a before/after evalutation, and we may or not confirm the issue.
@github-actions[bot] commented on GitHub (Nov 25, 2024):
This issue is stale (30 days with no activity)
@francescor commented on GitHub (Dec 6, 2024):
Hi, we are using S3 for images in this https://www.comune-qa.bugliano.pi.it (development)
while we do not use S3 in this other one https://www.comune.vicenza.it (production)
@sadok-f can I ask your opinion on their flyimg speed?
test says
@sadok-f commented on GitHub (Dec 9, 2024):
Hi @francescor
Using S3 will definitely be slower than saving the images and serving them from local storage. From the numbers you shared, the overall performance seems acceptable, but you can validate that in your QA environment before rolling it out to production.
@francescor commented on GitHub (Dec 9, 2024):
omg, I was expecting the opposite:
so, maybe only if we have 6 severs with no S3, and each has to create the image: then we have a 6x time, while if all the 6 servers use S3, then only the first has to create the image for the others: but then, once the image is created having it locally compared to S3 will always be faster
am I correct?
@sadok-f commented on GitHub (Dec 10, 2024):
you have a good point here, it all depends on the generated images, when you pass some options, the generation will take longer.
keep in mind also that s3 is in another location and it could be some latency fetching the images, while you could use a shared storage between your instances, and it'll be for sure faster than s3
@github-actions[bot] commented on GitHub (Jan 10, 2025):
This issue is stale (30 days with no activity)
@github-actions[bot] commented on GitHub (Jan 25, 2025):
This issue was closed (14 days since being marked as stale)