mirror of
https://github.com/floccusaddon/floccus.git
synced 2026-04-25 22:26:06 +03:00
[GH-ISSUE #1383] Initial Sync Causing Server CPU Saturation #911
Labels
No labels
browser-specific
bug
correctness issues
enhancement
feature: Google Drive
feature: Linkwarden
feature: git
feature: nextcloud-bookmarks
feature: tabs
feature: webdav
help wanted
native-app
priority: high
priority: low
priority: medium
pull-request
question
question
stale
upstream
waiting for more information
wontfix
🙁 Not following issue template
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/floccus#911
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @sproggit on GitHub (May 5, 2023).
Original GitHub issue: https://github.com/floccusaddon/floccus/issues/1383
Which version of floccus are you using?
v4.19.1
Sync method
Nextcloud Bookmarks
Which browser are you using? In case you are using the phone App, specify the Android or iOS version and device please.
Firefox 112.0.2/64-bit
Which version of Nextcloud Bookmarks are you using? (if relevant)
v13.0.1
Which version of Nextcloud? (if relevant)
v26.0.1
What kind of WebDAV server are you using? (if relevant)
Not Applicable
Describe the Bug
Tech Setup:-
Client: Home-built workstation using ASRock Z390M-ITX/AC motherboard, Intel Core i9-9900 CPU and 32Gb RAM
NextCloud Server: Raspbery Pi 4 running Raspberry Pi OS Bullseye with apt-get update/apt-get upgrade reporting no new packages
Database Server: QNAP TVS-672XT NAS/Core i5/32Gb RAM/MariaDB v10.5.8
Observed Behaviour:
Having configured Floccus and set the initial synchronisation from browser to NextCloud using "Push Up", an extremely slow sync rate is observed.
A check of the workstation using "System Monitor" shows long periods of no network transfer and then brief bursts of up to 2kb/s, but more often showing around 600 bytes/sec...
Examining the Raspberry Pi, however, the BASH command "top" shows 3 instances of php-fpm8.2, but with CPU utilisation ofg 91.4, 88.4 and 7.6% respectively - more often than not I will see 4 instances of fpm running and with a higher CPU load.
I am not sure if the next item is relevant, but including for completeness... The NextCloud Administrator's console screen, in the section titled "Security & setup warnings" on the "Overview" page, reports, "The PHP memory limit is below the recommended value of 512MB". However, from the "/etc/php" folder, the following bash command:-
grep -R memory_limit *
Returns the following:-
7.4/fpm/php.ini:memory_limit = 512M
8.1/cli/php.ini:memory_limit = -1
8.2/cgi/php.ini:memory_limit = 512M
8.2/cli/php.ini:memory_limit = -1
8.2/fpm/php.ini:memory_limit = 512M
(All values of "512M" were manually altered from their original values of "128M".
Going to the "home page" of the Bookmarks Plug-in and checking the status values in the panel on the left, I can [at the time of writing] see "All Bookmarks: 3.8K" and "Duplicates: 1.5K"
Refreshing this page in the browser every few minutes shows these values incrementing steadily - if very slowly.
Assessment
There is something happening in the way that the Bookmarks plug-in for NextCloud is handling a "Push Up" synchronisation that is extremely inefficient - data transfer rates of between 0 and 2kb/s... of bookmarks data... simply should not be capable of pushing a Pi 4B with 4Gb RAM to CPU saturation. (I've now done a few more checks and I'm pretty sure that there is now also some CPU throttling going on).
There is nothing else in the configuration which is remotely constrained.
If this was a 2-way sync that required more logical effort I could understand the CPU load, but this seems unexpected given the one-way delivery.
Expected Behavior
Ideally, I would expect the "initial setup" i.e. "Push Up" from browser to Pi to happen much more quickly.
This is anecdotal, not scientific... but today I also installed the NextCloud client for iOS to my iPhone and then used that the synchronize my camera roll from the iPhone to my "Photos" App on Nextcloud. Despite the total data volume being many times higher than the bookmarks (an HTML export of my browser bookmark results in a 4.8Mb file - around the size of one of my Phone images... - takes a few minutes to complete. Floccus has been syncing for at least the last 3 hours and is still grinding along.
To Reproduce
Reproduce the configuration of hardware and software described above and simply start a "Push Up" sync.
I've cancelled and re-started this several times now. I've completely re-installed NextCloud from scratch, to eliminate any possibility of other Apps causing issues. I've shut down everything else on the Pi4B and switched off all other network-connected devices at their power sockets. In short, I've eliminated everything I can f
rom this setup, but I get the same symptoms, reliably, every time.
Debug log provided
@sproggit commented on GitHub (May 5, 2023):
Apologies - just in case it is relevant, I realised that in my description of the setup I am using, I did not state that my workstation is running Linux Mint 20.3/64-bit.
Also, for clarity, the 2 screen shots included above - the System Monitor is from my Mint workstation, the bash console running "top" is from my Raspberry Pi.
@marcelklehr commented on GitHub (May 21, 2023):
Do you have apcu enabled?
@sproggit commented on GitHub (May 21, 2023):
Hello Marcel, thank you for responding.
I just checked and it does not look as though I have APCu installed or enabled. I had not previously encountered this language feature... so as soon as I can [working right now] I will update my configuration.
Unfortunately [unless you can suggest one] I am not sure that there is a quick, easy and safe way to test the addition of APCu after I add it... I have managed to complete the initial synchronisation between my browser and NextCloud... so I suspect that in order for me to test this, I would have to somehow clear out the NextCloud library and start over? That seems like a high risk strategy.
However, if you believe that this would have avoided the issue, we could close this ticket and I can make a note in my documentation to make sure this is set if I am ever forced to rebuild from scratch.
Thank you
Clive
@marcelklehr commented on GitHub (May 21, 2023):
You should also see apcu make a difference for normal operation.
@marcelklehr commented on GitHub (May 21, 2023):
I'll leave this open until you've had time to test :)
@sproggit commented on GitHub (May 22, 2023):
Marcel,
This evening I've applied APCu for PHP 8.2 on my Raspberry Pi (NextCloud Server), restarted Apache and run tests by forcing the Floccus plug-in to initiate a sync. I'm watching it run now. I have 3 discretely configured replicas in my browser:
Bookmarks Toolbar
Other Bookmarks
Bookmarks Menu
In total, the NC Bookmarks plug-in reports I have "3.8K" unique URLs bookmarked... so replication time is going to be quite long regardless. However, as I'm sitting here watching, I'm seeing a User CPU utilisation of max 5%, which is a world apart from seeing 95%+ before the installation. I'm also seeing just once instance of php-fpm8.2 running, whereas before the application of APCu I was seeing 8 running and watching them saturate the CPU and bring the machine to a standstill.
It's always difficult to prove a negative... but having said that the difference between performance either side of APCu application suggests that the change has made a significant difference to performance.
Thanks for the fix!
@marcelklehr commented on GitHub (May 23, 2023):
Yay, thank you for the update! I'm happy I could help :)
@github-actions[bot] commented on GitHub (May 23, 2024):
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.