mirror of
https://github.com/floccusaddon/floccus.git
synced 2026-04-25 14:16:12 +03:00
[GH-ISSUE #1011] sync large amount of bookmarks (40k) in normal sync mode not responsing extension failed #665
Labels
No labels
browser-specific
bug
correctness issues
enhancement
feature: Google Drive
feature: Linkwarden
feature: git
feature: nextcloud-bookmarks
feature: tabs
feature: webdav
help wanted
native-app
priority: high
priority: low
priority: medium
pull-request
question
question
stale
upstream
waiting for more information
wontfix
🙁 Not following issue template
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/floccus#665
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @xiaosuv on GitHub (Dec 18, 2021).
Original GitHub issue: https://github.com/floccusaddon/floccus/issues/1011
Describe the bug
storage using webdav, sync large amount of bookmarks (40k) in normal sync mode not responsing extension failed or slow and no progress.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
bookmarks should sync success
Desktop
(please complete the following information)
Server
(please complete the following information)
@marcelklehr commented on GitHub (Dec 28, 2021):
oof. 40k bookmarks could be enough to break the memory limit :/
@lazymonkey2 commented on GitHub (Jan 5, 2022):
I have 35k bookmarks and floccus is really slow. it takes at least 15 minutes to sync on gdrive, and the fan on my laptop is spinning furiously. also during the sync the floccus icon in the toolbar becomes not reponsive after some minutes (clicking on it has no effect). but at least it seems to complete successffuly on my device. I haven't even tried on android with kiwi... it probably would eat a ton of battery. in any case currently floccus is not usable for me :-(
I don't know if it's possible but could you consider using web assembly (wasm) to reduce cpu usage and speed up the sync? (I believe the merge process in particular could benefit from this)
thanks.
@a-litsov commented on GitHub (Jan 19, 2022):
Yes, even for 2.5k bookmarks it is very slow. It seems that it sends separate GET request for each bookmark and folder, so it is very ineffective. It even crashed my nextcloud instance several times while syncing (cheap $5 droplet).
@marcelklehr commented on GitHub (Feb 26, 2022):
Note that it only sends a request per changed bookmark with Nextcloud Bookmarks, so unless you regularly change 1000s of bookmarks, network is not the main bottleneck (although it is a bottleneck, granted).
@marcelklehr commented on GitHub (Feb 26, 2022):
I may have been able to improve this situation with #1053 (by not constructing a hash tree for the whole bookmarks tree anymore which presumably would block a lot of CPU cycles, if you have 1000s of bookmarks).
@marcelklehr commented on GitHub (Feb 26, 2022):
I'd be grateful, if you could help test whether this makes a difference. I'll provide you with an alpha build.
@xiaosuv commented on GitHub (Feb 27, 2022):
I will test and report
@marcelklehr commented on GitHub (Mar 1, 2022):
@xiaosuv Thank you! You can find the alpha build here: https://github.com/floccusaddon/floccus/releases/tag/v4.11.0-alpha.0
@xiaosuv commented on GitHub (Mar 3, 2022):
sync slow, didn't finished in 15 minutes, extension icon not responsed, high CPU usage.

@MarkusLoyen commented on GitHub (Mar 9, 2022):
@marcelklehr
I also tried the alpha mentioned before with firefox, SRWareIron (Chromium) and Vivaldi.

With all I get ridicolous high cpu load and all the browser become unusable in the end.
here for SRWare Iron (Chromium)
I tried syncing with a webdav-target (Strato HiDrive) and in addition to that my own nextcloud with nextcloud bookmarks.

from time to time Iron (running on windows 11) comes up with this:
The impact is in the end so high, that I need to disable floccus to be able to work again. This needs urgent fixing..
In case you need more information let me know.
@ShapeShifter499 commented on GitHub (Apr 17, 2022):
I'm in the same boat, my Firefox install has around 14K+ bookmarks. It appears to run for a little bit but then it stalls and hangs the whole web browser.
I did notice a lot of errors with bookmarks using Nextcloud as a backend in Nextcloud logs.
A couple of those errors expanded below.
@exstntlmsnthrp commented on GitHub (Jun 16, 2022):
I also have some of the issues referenced here. Using Firefox 101.0.1, floccus extension 4.15.0. It maxes out my CPU (i7) and RAM (32gb) on my local machine when I attempt to sync with a self-hosted instance of Nextcloud v.24.0.1 using Nextcloud bookmarks v.10.4.0. After 10-15 minutes Firefox crashes before the sync has completed. Similar problems with Brave. I have ~6000 bookmarks.
I have checked my NC logs (good idea @ShapeShifter499!) and do not have any errors listed. However, during sync it does max out the CPU and RAM on my instance.
On my phone (lineageos 19.1), sync is fast with the floccus app, which is confusing given the above.
Edit: Trying again at the moment, with Firefox as the only open app, sync has been stuck at approximately 50% complete for almost 30 minutes.
@marcelklehr commented on GitHub (Jun 25, 2022):
I have finally had time to look into this. I was able to reproduce this and have a fix.
I've also reaped some low hanging performance gains and reduced the bulk import limit from 300 bookmarks to 75 bookmarks, so that the bulk import bundles aren't overloading the server anymore, potentially causing status 504 errors and causing floccus to resort to individual requests for each bookmark.
With these changes 6k bookmarks took me ~6min.
@marcelklehr commented on GitHub (Jun 26, 2022):
Please try the latest release once it has passed review and report back if this is fixed for you ♥️
@c608345 commented on GitHub (Jun 28, 2022):
This is not a solution for me as I have 40k bookmarks, sync takes too much time.
I will wait patiently or I fix this someday.
@marcelklehr commented on GitHub (Jun 28, 2022):
@c608345 Note, that I gave the time only for a sync where 6k bookmarks are added, ie. the initial sync. Syncing without changes should be much faster.
@c608345 commented on GitHub (Jun 28, 2022):
I should try this release.
@marcelklehr commented on GitHub (Jun 29, 2022):
Just tried with 18k bookmarks with Firefox and NC Bookmarks: initial sync took ~18min (because of uploading), second sync took ~3min, third sync took 35s.
@xiaosuv commented on GitHub (Jun 30, 2022):
PC A 45k bookmarks, sync type push up to a local webdav server, it takes about 25 minites.
PC B a 2016 notebook pull down in 17 mins, sync needs 1 GB memory.
It works, thank you for your work.
@github-actions[bot] commented on GitHub (Jul 1, 2023):
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.