[GH-ISSUE #1011] sync large amount of bookmarks (40k) in normal sync mode not responsing extension failed #665

Closed
opened 2026-02-25 22:37:47 +03:00 by kerem · 20 comments
Owner

Originally created by @xiaosuv on GitHub (Dec 18, 2021).
Original GitHub issue: https://github.com/floccusaddon/floccus/issues/1011

Describe the bug

storage using webdav, sync large amount of bookmarks (40k) in normal sync mode not responsing extension failed or slow and no progress.

To Reproduce

Steps to reproduce the behavior:

  1. get 40k bookmarks
  2. setup floccus webdav
  3. using normal sync mode
  4. extension failed need reload or slow and no progress.

Expected behavior

bookmarks should sync success

Desktop

(please complete the following information)

  • OS: debian 10
  • Browser chromium
  • Browser Version 96
  • Floccus version: 4.8.7
  • Floccus sync method: webdav

Server

(please complete the following information)

Originally created by @xiaosuv on GitHub (Dec 18, 2021). Original GitHub issue: https://github.com/floccusaddon/floccus/issues/1011 ### Describe the bug storage using webdav, sync large amount of bookmarks (40k) in normal sync mode not responsing extension failed or slow and no progress. ### To Reproduce Steps to reproduce the behavior: 1. get 40k bookmarks 2. setup floccus webdav 3. using normal sync mode 4. extension failed need reload or slow and no progress. ### Expected behavior bookmarks should sync success ### Desktop (please complete the following information) - OS: debian 10 - Browser chromium - Browser Version 96 - Floccus version: 4.8.7 - Floccus sync method: webdav ### Server (please complete the following information) - OS: debian 10 - webdav: https://github.com/hacdias/webdav
Author
Owner

@marcelklehr commented on GitHub (Dec 28, 2021):

oof. 40k bookmarks could be enough to break the memory limit :/

<!-- gh-comment-id:1002016743 --> @marcelklehr commented on GitHub (Dec 28, 2021): oof. 40k bookmarks could be enough to break the memory limit :/
Author
Owner

@lazymonkey2 commented on GitHub (Jan 5, 2022):

I have 35k bookmarks and floccus is really slow. it takes at least 15 minutes to sync on gdrive, and the fan on my laptop is spinning furiously. also during the sync the floccus icon in the toolbar becomes not reponsive after some minutes (clicking on it has no effect). but at least it seems to complete successffuly on my device. I haven't even tried on android with kiwi... it probably would eat a ton of battery. in any case currently floccus is not usable for me :-(
I don't know if it's possible but could you consider using web assembly (wasm) to reduce cpu usage and speed up the sync? (I believe the merge process in particular could benefit from this)
thanks.

<!-- gh-comment-id:1005537076 --> @lazymonkey2 commented on GitHub (Jan 5, 2022): I have 35k bookmarks and floccus is really slow. it takes at least 15 minutes to sync on gdrive, and the fan on my laptop is spinning furiously. also during the sync the floccus icon in the toolbar becomes not reponsive after some minutes (clicking on it has no effect). but at least it seems to complete successffuly on my device. I haven't even tried on android with kiwi... it probably would eat a ton of battery. in any case currently floccus is not usable for me :-( I don't know if it's possible but could you consider using web assembly (wasm) to reduce cpu usage and speed up the sync? (I believe the merge process in particular could benefit from this) thanks.
Author
Owner

@a-litsov commented on GitHub (Jan 19, 2022):

Yes, even for 2.5k bookmarks it is very slow. It seems that it sends separate GET request for each bookmark and folder, so it is very ineffective. It even crashed my nextcloud instance several times while syncing (cheap $5 droplet).

<!-- gh-comment-id:1016704032 --> @a-litsov commented on GitHub (Jan 19, 2022): Yes, even for 2.5k bookmarks it is very slow. It seems that it sends separate GET request for each bookmark and folder, so it is very ineffective. It even crashed my nextcloud instance several times while syncing (cheap $5 droplet).
Author
Owner

@marcelklehr commented on GitHub (Feb 26, 2022):

sends separate GET request for each bookmark and folder, so it is very ineffective

Note that it only sends a request per changed bookmark with Nextcloud Bookmarks, so unless you regularly change 1000s of bookmarks, network is not the main bottleneck (although it is a bottleneck, granted).

<!-- gh-comment-id:1052170795 --> @marcelklehr commented on GitHub (Feb 26, 2022): > sends separate GET request for each bookmark and folder, so it is very ineffective Note that it only sends a request per *changed* bookmark with Nextcloud Bookmarks, so unless you regularly change 1000s of bookmarks, network is not the main bottleneck (although it is *a* bottleneck, granted).
Author
Owner

@marcelklehr commented on GitHub (Feb 26, 2022):

I may have been able to improve this situation with #1053 (by not constructing a hash tree for the whole bookmarks tree anymore which presumably would block a lot of CPU cycles, if you have 1000s of bookmarks).

<!-- gh-comment-id:1052171304 --> @marcelklehr commented on GitHub (Feb 26, 2022): I may have been able to improve this situation with #1053 (by not constructing a hash tree for the whole bookmarks tree anymore which presumably would block a lot of CPU cycles, if you have 1000s of bookmarks).
Author
Owner

@marcelklehr commented on GitHub (Feb 26, 2022):

I'd be grateful, if you could help test whether this makes a difference. I'll provide you with an alpha build.

<!-- gh-comment-id:1052172447 --> @marcelklehr commented on GitHub (Feb 26, 2022): I'd be grateful, if you could help test whether this makes a difference. I'll provide you with an alpha build.
Author
Owner

@xiaosuv commented on GitHub (Feb 27, 2022):

I will test and report

<!-- gh-comment-id:1053098140 --> @xiaosuv commented on GitHub (Feb 27, 2022): I will test and report
Author
Owner

@marcelklehr commented on GitHub (Mar 1, 2022):

@xiaosuv Thank you! You can find the alpha build here: https://github.com/floccusaddon/floccus/releases/tag/v4.11.0-alpha.0

<!-- gh-comment-id:1055506747 --> @marcelklehr commented on GitHub (Mar 1, 2022): @xiaosuv Thank you! You can find the alpha build here: https://github.com/floccusaddon/floccus/releases/tag/v4.11.0-alpha.0
Author
Owner

@xiaosuv commented on GitHub (Mar 3, 2022):

sync slow, didn't finished in 15 minutes, extension icon not responsed, high CPU usage.
Screenshot_2022-03-03_09-10-53

<!-- gh-comment-id:1057565996 --> @xiaosuv commented on GitHub (Mar 3, 2022): sync slow, didn't finished in 15 minutes, extension icon not responsed, high CPU usage. ![Screenshot_2022-03-03_09-10-53](https://user-images.githubusercontent.com/1177060/156477771-b00588af-5239-4f9d-af14-8572df76ac28.png)
Author
Owner

@MarkusLoyen commented on GitHub (Mar 9, 2022):

@marcelklehr

I also tried the alpha mentioned before with firefox, SRWareIron (Chromium) and Vivaldi.
With all I get ridicolous high cpu load and all the browser become unusable in the end.
here for SRWare Iron (Chromium)
image

I tried syncing with a webdav-target (Strato HiDrive) and in addition to that my own nextcloud with nextcloud bookmarks.
from time to time Iron (running on windows 11) comes up with this:
image

The impact is in the end so high, that I need to disable floccus to be able to work again. This needs urgent fixing..

In case you need more information let me know.

<!-- gh-comment-id:1063479347 --> @MarkusLoyen commented on GitHub (Mar 9, 2022): @marcelklehr I also tried the alpha mentioned before with firefox, SRWareIron (Chromium) and Vivaldi. With all I get ridicolous high cpu load and all the browser become unusable in the end. here for SRWare Iron (Chromium) ![image](https://user-images.githubusercontent.com/58221933/157555406-4150bf39-cf46-4f03-bd9f-d2f5765f73c3.png) I tried syncing with a webdav-target (Strato HiDrive) and in addition to that my own nextcloud with nextcloud bookmarks. from time to time Iron (running on windows 11) comes up with this: ![image](https://user-images.githubusercontent.com/58221933/157555308-c3180a22-cea0-4eac-892f-436e826097a6.png) The impact is in the end so high, that I need to disable floccus to be able to work again. This needs urgent fixing.. In case you need more information let me know.
Author
Owner

@ShapeShifter499 commented on GitHub (Apr 17, 2022):

I'm in the same boat, my Firefox install has around 14K+ bookmarks. It appears to run for a little bit but then it stalls and hangs the whole web browser.

I did notice a lot of errors with bookmarks using Nextcloud as a backend in Nextcloud logs.

Screenshot from 2022-04-16 19-42-32

A couple of those errors expanded below.

{"reqId":"6kBZqD6gMYTb69YayVma","level":2,"time":"2022-04-16T14:30:01+00:00","remoteAddr":"<remote_IP>","user":"<USER>","app":"bookmarks","method":"POST","url":"/index.php/apps/bookmarks/public/rest/v2/bookmark","message":"cURL error 28: Operation timed out after 10001 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://www.bunniestudios.com/blog/?page_id=1022","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0","version":"23.0.2.1","id":"625b7e7d82ab3"}
{"reqId":"kRamzI5ZblqVpQLThURb","level":3,"time":"2022-04-16T14:30:38+00:00","remoteAddr":"<remote_IP>","user":"<USER>","app":"PHP","method":"POST","url":"/index.php/apps/bookmarks/public/rest/v2/bookmark","message":"dns_get_record(): A temporary server error occurred. at /snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php#57","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0","version":"23.0.2.1","exception":{"Exception":"Error","Message":"dns_get_record(): A temporary server error occurred. at /snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php#57","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php","line":57,"function":"dns_get_record"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php","line":74,"function":"soaRecord","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php","line":128,"function":"dnsResolve","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/Client.php","line":223,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Http/Client.php","line":50,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/vendor/marcelklehr/link-preview/src/Models/Link.php","line":63,"function":"sendRequest","class":"OCA\\Bookmarks\\Http\\Client","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/vendor/marcelklehr/link-preview/src/Models/Link.php","line":73,"function":"fetch","class":"Marcelklehr\\LinkPreview\\Models\\Link","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Service/LinkExplorer.php","line":54,"function":"getPreview","class":"Marcelklehr\\LinkPreview\\Models\\Link","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Service/BookmarkService.php","line":200,"function":"get","class":"OCA\\Bookmarks\\Service\\LinkExplorer","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Service/BookmarkService.php","line":147,"function":"_addBookmark","class":"OCA\\Bookmarks\\Service\\BookmarkService","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Controller/BookmarkController.php","line":428,"function":"create","class":"OCA\\Bookmarks\\Service\\BookmarkService","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":217,"function":"newBookmark","class":"OCA\\Bookmarks\\Controller\\BookmarkController","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":126,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/AppFramework/App.php","line":157,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Route/Router.php","line":302,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/snap/nextcloud/29938/htdocs/lib/base.php","line":1006,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/snap/nextcloud/29938/htdocs/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"id":"625b7e7d8299c"}
<!-- gh-comment-id:1100792206 --> @ShapeShifter499 commented on GitHub (Apr 17, 2022): I'm in the same boat, my Firefox install has around 14K+ bookmarks. It appears to run for a little bit but then it stalls and hangs the whole web browser. I did notice a lot of errors with bookmarks using Nextcloud as a backend in Nextcloud logs. ![Screenshot from 2022-04-16 19-42-32](https://user-images.githubusercontent.com/388665/163698284-94eeed2c-659d-4f96-aea8-89e023cac507.png) A couple of those errors expanded below. ``` {"reqId":"6kBZqD6gMYTb69YayVma","level":2,"time":"2022-04-16T14:30:01+00:00","remoteAddr":"<remote_IP>","user":"<USER>","app":"bookmarks","method":"POST","url":"/index.php/apps/bookmarks/public/rest/v2/bookmark","message":"cURL error 28: Operation timed out after 10001 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://www.bunniestudios.com/blog/?page_id=1022","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0","version":"23.0.2.1","id":"625b7e7d82ab3"} ``` ``` {"reqId":"kRamzI5ZblqVpQLThURb","level":3,"time":"2022-04-16T14:30:38+00:00","remoteAddr":"<remote_IP>","user":"<USER>","app":"PHP","method":"POST","url":"/index.php/apps/bookmarks/public/rest/v2/bookmark","message":"dns_get_record(): A temporary server error occurred. at /snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php#57","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0","version":"23.0.2.1","exception":{"Exception":"Error","Message":"dns_get_record(): A temporary server error occurred. at /snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php#57","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php","line":57,"function":"dns_get_record"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php","line":74,"function":"soaRecord","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/DnsPinMiddleware.php","line":128,"function":"dnsResolve","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Http/Client/Client.php","line":223,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Http/Client.php","line":50,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/vendor/marcelklehr/link-preview/src/Models/Link.php","line":63,"function":"sendRequest","class":"OCA\\Bookmarks\\Http\\Client","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/vendor/marcelklehr/link-preview/src/Models/Link.php","line":73,"function":"fetch","class":"Marcelklehr\\LinkPreview\\Models\\Link","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Service/LinkExplorer.php","line":54,"function":"getPreview","class":"Marcelklehr\\LinkPreview\\Models\\Link","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Service/BookmarkService.php","line":200,"function":"get","class":"OCA\\Bookmarks\\Service\\LinkExplorer","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Service/BookmarkService.php","line":147,"function":"_addBookmark","class":"OCA\\Bookmarks\\Service\\BookmarkService","type":"->"},{"file":"/var/snap/nextcloud/29938/nextcloud/extra-apps/bookmarks/lib/Controller/BookmarkController.php","line":428,"function":"create","class":"OCA\\Bookmarks\\Service\\BookmarkService","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":217,"function":"newBookmark","class":"OCA\\Bookmarks\\Controller\\BookmarkController","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":126,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/AppFramework/App.php","line":157,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/lib/private/Route/Router.php","line":302,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/snap/nextcloud/29938/htdocs/lib/base.php","line":1006,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/snap/nextcloud/29938/htdocs/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/snap/nextcloud/29938/htdocs/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"id":"625b7e7d8299c"} ```
Author
Owner

@exstntlmsnthrp commented on GitHub (Jun 16, 2022):

I also have some of the issues referenced here. Using Firefox 101.0.1, floccus extension 4.15.0. It maxes out my CPU (i7) and RAM (32gb) on my local machine when I attempt to sync with a self-hosted instance of Nextcloud v.24.0.1 using Nextcloud bookmarks v.10.4.0. After 10-15 minutes Firefox crashes before the sync has completed. Similar problems with Brave. I have ~6000 bookmarks.

I have checked my NC logs (good idea @ShapeShifter499!) and do not have any errors listed. However, during sync it does max out the CPU and RAM on my instance.

On my phone (lineageos 19.1), sync is fast with the floccus app, which is confusing given the above.

Edit: Trying again at the moment, with Firefox as the only open app, sync has been stuck at approximately 50% complete for almost 30 minutes.

<!-- gh-comment-id:1157909595 --> @exstntlmsnthrp commented on GitHub (Jun 16, 2022): I also have some of the issues referenced here. Using Firefox 101.0.1, floccus extension 4.15.0. It maxes out my CPU (i7) and RAM (32gb) on my local machine when I attempt to sync with a self-hosted instance of Nextcloud v.24.0.1 using Nextcloud bookmarks v.10.4.0. After 10-15 minutes Firefox crashes before the sync has completed. Similar problems with Brave. I have ~6000 bookmarks. I have checked my NC logs (good idea @ShapeShifter499!) and do not have any errors listed. However, during sync it does max out the CPU and RAM on my instance. On my phone (lineageos 19.1), sync is fast with the floccus app, which is confusing given the above. Edit: Trying again at the moment, with Firefox as the only open app, sync has been stuck at approximately 50% complete for almost 30 minutes.
Author
Owner

@marcelklehr commented on GitHub (Jun 25, 2022):

I have finally had time to look into this. I was able to reproduce this and have a fix.

I've also reaped some low hanging performance gains and reduced the bulk import limit from 300 bookmarks to 75 bookmarks, so that the bulk import bundles aren't overloading the server anymore, potentially causing status 504 errors and causing floccus to resort to individual requests for each bookmark.

With these changes 6k bookmarks took me ~6min.

<!-- gh-comment-id:1166314945 --> @marcelklehr commented on GitHub (Jun 25, 2022): I have finally had time to look into this. I was able to reproduce this and have a fix. I've also reaped some low hanging performance gains and reduced the bulk import limit from 300 bookmarks to 75 bookmarks, so that the bulk import bundles aren't overloading the server anymore, potentially causing status 504 errors and causing floccus to resort to individual requests for each bookmark. With these changes 6k bookmarks took me ~6min.
Author
Owner

@marcelklehr commented on GitHub (Jun 26, 2022):

Please try the latest release once it has passed review and report back if this is fixed for you ♥️

<!-- gh-comment-id:1166535349 --> @marcelklehr commented on GitHub (Jun 26, 2022): Please try the latest release once it has passed review and report back if this is fixed for you :hearts:
Author
Owner

@c608345 commented on GitHub (Jun 28, 2022):

This is not a solution for me as I have 40k bookmarks, sync takes too much time.
I will wait patiently or I fix this someday.

<!-- gh-comment-id:1168656270 --> @c608345 commented on GitHub (Jun 28, 2022): This is not a solution for me as I have 40k bookmarks, sync takes too much time. I will wait patiently or I fix this someday.
Author
Owner

@marcelklehr commented on GitHub (Jun 28, 2022):

@c608345 Note, that I gave the time only for a sync where 6k bookmarks are added, ie. the initial sync. Syncing without changes should be much faster.

<!-- gh-comment-id:1168728977 --> @marcelklehr commented on GitHub (Jun 28, 2022): @c608345 Note, that I gave the time only for a sync where 6k bookmarks are *added*, ie. the initial sync. Syncing without changes should be much faster.
Author
Owner

@c608345 commented on GitHub (Jun 28, 2022):

I should try this release.

<!-- gh-comment-id:1168766040 --> @c608345 commented on GitHub (Jun 28, 2022): I should try this release.
Author
Owner

@marcelklehr commented on GitHub (Jun 29, 2022):

Just tried with 18k bookmarks with Firefox and NC Bookmarks: initial sync took ~18min (because of uploading), second sync took ~3min, third sync took 35s.

<!-- gh-comment-id:1170059336 --> @marcelklehr commented on GitHub (Jun 29, 2022): Just tried with 18k bookmarks with Firefox and NC Bookmarks: initial sync took ~18min (because of uploading), second sync took ~3min, third sync took 35s.
Author
Owner

@xiaosuv commented on GitHub (Jun 30, 2022):

PC A 45k bookmarks, sync type push up to a local webdav server, it takes about 25 minites.
PC B a 2016 notebook pull down in 17 mins, sync needs 1 GB memory.
It works, thank you for your work.

<!-- gh-comment-id:1171270983 --> @xiaosuv commented on GitHub (Jun 30, 2022): PC A 45k bookmarks, sync type push up to a local webdav server, it takes about 25 minites. PC B a 2016 notebook pull down in 17 mins, sync needs 1 GB memory. It works, thank you for your work.
Author
Owner

@github-actions[bot] commented on GitHub (Jul 1, 2023):

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

<!-- gh-comment-id:1615351980 --> @github-actions[bot] commented on GitHub (Jul 1, 2023): This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/floccus#665
No description provided.