[GH-ISSUE #476] High Load in Nextcloud at sync

kerem commented

2026-02-25 22:36:53 +03:00

Owner

Originally created by @superwinni2 on GitHub (Mar 24, 2020).
Original GitHub issue: https://github.com/floccusaddon/floccus/issues/476

Describe the bug
When I try to sync my booksmarks with my nextcloud, my cloud gets a very high load at about ~5-9
Cloud is installed on LXC on Proxmox. 4 Cores. 1.5 GB RAM 10 GB of memory.
Everything is good until I start sync in floccur. RAM runs full from 165 MB to 1,3 GB + a little bit of SWAP. Also the sync fails.
At beginngs when I used floccus there wasn't such problems. For normal use my nextcloud got ~256 MB RAM and only one core. I gave it more because of the problems.

Expected behavior
Normal Load and RAM usage on nextcloud when syncing.

Desktop (please complete the following information):

OS: Wondows 10 - 1909
Browser Chrome
Browser Version 80.0.3987.132 (64 bit)
Floccus version: 4.0.2
Floccus sync method: Nextcloud

Server (please complete the following information):

OS: ProxMox 6.1-7 LXC Container Debian Buster
Nextcloud version: 18.0.3 (problem was also in 18.02 and earlier)
Bookmarks app version: 2.3.4

Debug log
See in your Cloud :)

[x ] Debug log provided

Originally created by @superwinni2 on GitHub (Mar 24, 2020). Original GitHub issue: https://github.com/floccusaddon/floccus/issues/476 **Describe the bug** When I try to sync my booksmarks with my nextcloud, my cloud gets a very high load at about ~5-9 Cloud is installed on LXC on Proxmox. 4 Cores. 1.5 GB RAM 10 GB of memory. Everything is good until I start sync in floccur. RAM runs full from 165 MB to 1,3 GB + a little bit of SWAP. Also the sync fails. At beginngs when I used floccus there wasn't such problems. For normal use my nextcloud got ~256 MB RAM and only one core. I gave it more because of the problems. **Expected behavior** Normal Load and RAM usage on nextcloud when syncing. **Desktop (please complete the following information):** - OS: Wondows 10 - 1909 - Browser Chrome - Browser Version 80.0.3987.132 (64 bit) - Floccus version: 4.0.2 - Floccus sync method: Nextcloud **Server (please complete the following information):** - OS: ProxMox 6.1-7 LXC Container Debian Buster - Nextcloud version: 18.0.3 (problem was also in 18.02 and earlier) - Bookmarks app version: 2.3.4 **Debug log** See in your Cloud :) - [x ] Debug log provided

kerem

2026-02-25 22:36:53 +03:00

closed this issue
added the
bug
label

kerem commented

2026-02-25 22:36:53 +03:00

Author

Owner

@forrestab commented on GitHub (Apr 9, 2020):

I would just like to add, I didn't see this problem on nextcloud@17 and started noticing it after upgrading.

I am running nextcloud in a docker container on unraid.

@forrestab commented on GitHub (Apr 9, 2020): I would just like to add, I didn't see this problem on nextcloud@17 and started noticing it after upgrading. I am running nextcloud in a docker container on unraid.

kerem commented

2026-02-25 22:36:53 +03:00

Author

Owner

@superwinni2 commented on GitHub (Apr 9, 2020):

I found out what it was today...
After some search, I found out, that if I'm taking my IP which I'm connecting to into the (bruteforce) whitelist, it's working again...
But I don't know why so.... I haven't changed any settings in floccus or account details in nextcloud. Due to that, after setting the whitlisted IP, everything was working like normal again.

Also tried it out on another Cloud... Same error... There are going to be some bookmarks synchronisized and after some times, there will be the error....
Btw... Some bookmark mean about 76 Bookmarks somewhere between 29 folders which has been synchonised... Some doubled at the server... On my client I*ve got about ~130 in 41 folders...

@superwinni2 commented on GitHub (Apr 9, 2020): I found out what it was today... After some search, I found out, that if I'm taking my IP which I'm connecting to into the (bruteforce) whitelist, it's working again... But I don't know why so.... I haven't changed any settings in floccus or account details in nextcloud. Due to that, after setting the whitlisted IP, everything was working like normal again. Also tried it out on another Cloud... Same error... There are going to be some bookmarks synchronisized and after some times, there will be the error.... Btw... Some bookmark mean about 76 Bookmarks somewhere between 29 folders which has been synchonised... Some doubled at the server... On my client I*ve got about ~130 in 41 folders...

kerem commented

2026-02-25 22:36:53 +03:00

Author

Owner

@marcelklehr commented on GitHub (Apr 9, 2020):

Ah, yes. Floccus can potentially trigger the brute force app :/

@marcelklehr commented on GitHub (Apr 9, 2020): Ah, yes. Floccus can potentially trigger the brute force app :/

kerem commented

2026-02-25 22:36:53 +03:00

Author

Owner

@superwinni2 commented on GitHub (Apr 10, 2020):

Okay... :/
Do you have any idea why?

@superwinni2 commented on GitHub (Apr 10, 2020): Okay... :/ Do you have any idea why?

kerem commented

2026-02-25 22:36:53 +03:00

Author

Owner

@marcelklehr commented on GitHub (Apr 10, 2020):

Floccus makes a lot of requests for syncing, especially on the first run when a lot of bookmarks have to be created.

@marcelklehr commented on GitHub (Apr 10, 2020): Floccus makes a lot of requests for syncing, especially on the first run when a lot of bookmarks have to be created.

kerem commented

2026-02-25 22:36:53 +03:00

Author

Owner

@marcelklehr commented on GitHub (Apr 23, 2020):

Bookmarks v3 will hopefully reduce server load significantly.

@marcelklehr commented on GitHub (Apr 23, 2020): Bookmarks v3 will hopefully reduce server load significantly.

kerem commented

2026-02-25 22:36:53 +03:00

Author

Owner

@mobamoba commented on GitHub (May 4, 2020):

I'm having this problem with 3.08 of Bookmarks. I reported high CPU usage in the Bookmarks git because I'm not sure if the problem is with Bookmarks or Floccus but having Floccus enabled wipes out my system's RAM and CPU. It was working fine until recently and nothing changed at my end.

I have Floccus on 3 different browsers and the only way to make it work is to disable it in each browser then turn them back on one at a time. Thus it would seem that the problem is that there's no queuing with the various Floccus instances, i.e. each of them is hitting Nextcloud at the same time and overwhelming the system.

@mobamoba commented on GitHub (May 4, 2020): I'm having this problem with 3.08 of Bookmarks. I reported high CPU usage in the Bookmarks git because I'm not sure if the problem is with Bookmarks or Floccus but having Floccus enabled wipes out my system's RAM and CPU. It was working fine until recently and nothing changed at my end. I have Floccus on 3 different browsers and the only way to make it work is to disable it in each browser then turn them back on one at a time. Thus it would seem that the problem is that there's no queuing with the various Floccus instances, i.e. each of them is hitting Nextcloud at the same time and overwhelming the system.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@superwinni2 commented on GitHub (May 5, 2020):

I'm having this problem with 3.08 of Bookmarks. I reported high CPU usage in the Bookmarks git because I'm not sure if the problem is with Bookmarks or Floccus but having Floccus enabled wipes out my system's RAM and CPU. It was working fine until recently and nothing changed at my end.

I have Floccus on 3 different browsers and the only way to make it work is to disable it in each browser then turn them back on one at a time. Thus it would seem that the problem is that there's no queuing with the various Floccus instances, i.e. each of them is hitting Nextcloud at the same time and overwhelming the system.

Looks likethe same problem of mine...
Too many requests in an amount of time so the nextcloud "bruteforce" protection is blocking the connection...
Try whitelistening the IP from where you are trying to connect. One after another. After everything is synced you can delete the whitlists again.

Did the same and now everything works like it should.

@superwinni2 commented on GitHub (May 5, 2020): > I'm having this problem with 3.08 of Bookmarks. I reported high CPU usage in the Bookmarks git because I'm not sure if the problem is with Bookmarks or Floccus but having Floccus enabled wipes out my system's RAM and CPU. It was working fine until recently and nothing changed at my end. > > I have Floccus on 3 different browsers and the only way to make it work is to disable it in each browser then turn them back on one at a time. Thus it would seem that the problem is that there's no queuing with the various Floccus instances, i.e. each of them is hitting Nextcloud at the same time and overwhelming the system. Looks likethe same problem of mine... Too many requests in an amount of time so the nextcloud "bruteforce" protection is blocking the connection... Try whitelistening the IP from where you are trying to connect. One after another. After everything is synced you can delete the whitlists again. Did the same and now everything works like it should.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@mobamoba commented on GitHub (May 7, 2020):

Unfortunately that didn't solve my problem. It happens every single time Floccus syncs (1 Firefox and 2 Chrome instances). And I access my Nextcloud from a dynamic IP so have no real way to whitelist anything in the Brute Force settings.

Since there's realistically no need to all instances of Floccus to sync at the same time, is there a way for the app the check and see if another Floccus instance is syncing and, if so, wait 5 minutes before trying again?

@mobamoba commented on GitHub (May 7, 2020): Unfortunately that didn't solve my problem. It happens every single time Floccus syncs (1 Firefox and 2 Chrome instances). And I access my Nextcloud from a dynamic IP so have no real way to whitelist anything in the Brute Force settings. Since there's realistically no need to all instances of Floccus to sync at the same time, is there a way for the app the check and see if another Floccus instance is syncing and, if so, wait 5 minutes before trying again?

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (May 7, 2020):

@mobamoba With the bookmarks app that's not possible, but the WebDAV method does exactly that.

@marcelklehr commented on GitHub (May 7, 2020): @mobamoba With the bookmarks app that's not possible, but the WebDAV method does exactly that.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@kai-uwe-rommel commented on GitHub (May 8, 2020):

I see the same problem. I have Floccus 4.1.0 in Firefox 76. When I sync the bookmarks menu, the sync takes 16 minutes (!). At the server end I have Nextcloud 17.06 with the Bookmarks 3.0.12 app. During the 16 minutes of sync, the Nextcloud server is at 100% CPU, it has 2 cores (Xeon 2.0 GHz) and 4 GB RAM, it is a VM in a vSphere cluster. The Brute Force Whitelist does not help. I wonder what Nextcloud/Bookmarks is doing with all those CPU cycles ...

@kai-uwe-rommel commented on GitHub (May 8, 2020): I see the same problem. I have Floccus 4.1.0 in Firefox 76. When I sync the bookmarks menu, the sync takes 16 minutes (!). At the server end I have Nextcloud 17.06 with the Bookmarks 3.0.12 app. During the 16 minutes of sync, the Nextcloud server is at 100% CPU, it has 2 cores (Xeon 2.0 GHz) and 4 GB RAM, it is a VM in a vSphere cluster. The Brute Force Whitelist does not help. I wonder what Nextcloud/Bookmarks is doing with all those CPU cycles ...

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (May 8, 2020):

@kai-uwe-rommel How many bookmarks do you have? Have you enabled APCu/some other form of caching on the server? How long is your PHP execution timeout?

@marcelklehr commented on GitHub (May 8, 2020): @kai-uwe-rommel How many bookmarks do you have? Have you enabled APCu/some other form of caching on the server? How long is your PHP execution timeout?

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@kai-uwe-rommel commented on GitHub (May 8, 2020):

Difficult to count the bookmarks. I estimate about 50 folders with around 500 ... 1000 bookmarks. PHP execution timeout is 3600 seconds. I did not install any extra caching solution (e.g. Redis). I don't know about APCu? Probably I should have opcode caching? Whenever I upgrade Nextcloud it complains and warns me that I do not have but should have opcode caching. A couple of time I already spent some time to make sure I have enabled opcode caching and I am pretty sure I have set everything correctly but Nextcloud still complains. Form googling around I gathered that this seems to be a "known" problem ... more people seem to see this issue.

@kai-uwe-rommel commented on GitHub (May 8, 2020): Difficult to count the bookmarks. I estimate about 50 folders with around 500 ... 1000 bookmarks. PHP execution timeout is 3600 seconds. I did not install any extra caching solution (e.g. Redis). I don't know about APCu? Probably I should have opcode caching? Whenever I upgrade Nextcloud it complains and warns me that I do not have but should have opcode caching. A couple of time I already spent some time to make sure I have enabled opcode caching and I am pretty sure I have set everything correctly but Nextcloud still complains. Form googling around I gathered that this seems to be a "known" problem ... more people seem to see this issue.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@kai-uwe-rommel commented on GitHub (May 8, 2020):

I have installed APCu now. No change of the problem.

@kai-uwe-rommel commented on GitHub (May 8, 2020): I have installed APCu now. No change of the problem.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (May 8, 2020):

APCu should allow progressive caching of folder hashes, so that the server state can be compared to the browser state very quickly. It might take a while for the cache to "warm up", though.

@marcelklehr commented on GitHub (May 8, 2020): APCu should allow progressive caching of folder hashes, so that the server state can be compared to the browser state very quickly. It might take a while for the cache to "warm up", though.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@mobamoba commented on GitHub (May 8, 2020):

I've had APCu caching for ages and I still have this problem.

@mobamoba commented on GitHub (May 8, 2020): I've had APCu caching for ages and I still have this problem.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (May 8, 2020):

Mmh. You could enable profiling in PHP and examine the profiler output (or send it to me to analyze it), to find out where it spends that time.

@marcelklehr commented on GitHub (May 8, 2020): Mmh. You could enable profiling in PHP and examine the profiler output (or send it to me to analyze it), to find out where it spends that time.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@kai-uwe-rommel commented on GitHub (May 8, 2020):

Can you tell me how to do this profiling? I am not really familiar with PHP. Thanks!

@kai-uwe-rommel commented on GitHub (May 8, 2020): Can you tell me how to do this profiling? I am not really familiar with PHP. Thanks!

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (May 8, 2020):

See https://stackoverflow.com/a/21145/5508501

@marcelklehr commented on GitHub (May 8, 2020): See https://stackoverflow.com/a/21145/5508501

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@mobamoba commented on GitHub (May 9, 2020):

I don't know if this will help but I disable Brute Force and disabled Floccus on 2 Chrome instance so literally the only Floccus instance I was running was on Firefox. It still goes to 100% and grinds my machine to a halt. In other words, the problem is not Brute Force (since it's disabled) nor is it multiple instances running at once (since there was only one instance running); it's something though I don't know how to you debug it.

@mobamoba commented on GitHub (May 9, 2020): I don't know if this will help but I disable Brute Force and disabled Floccus on 2 Chrome instance so literally the only Floccus instance I was running was on Firefox. It still goes to 100% and grinds my machine to a halt. In other words, the problem is not Brute Force (since it's disabled) nor is it multiple instances running at once (since there was only one instance running); it's something though I don't know how to you debug it.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (May 10, 2020):

It might be hash calculations for the tree comparison, but those should be cached.

@marcelklehr commented on GitHub (May 10, 2020): It might be hash calculations for the tree comparison, but those *should* be cached.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@kai-uwe-rommel commented on GitHub (May 10, 2020):

I did not have time yet to tackle xdebug. However it looks like since I added APCu, the 100% CPU peak became much narrower. Instead of 15 minutes it now takes only 3 for a sync of my bookmarks menu. Impressive, if it stays this way. I have also much increased the sync interval (because I only change bookmarks in my main browser) to 8 hours so the load from this on my Nextcloud instance should become negligible. Before it caused an almost constant 40% load (on average).

@kai-uwe-rommel commented on GitHub (May 10, 2020): I did not have time yet to tackle xdebug. However it looks like since I added APCu, the 100% CPU peak became much narrower. Instead of 15 minutes it now takes only 3 for a sync of my bookmarks menu. Impressive, if it stays this way. I have also much increased the sync interval (because I only change bookmarks in my main browser) to 8 hours so the load from this on my Nextcloud instance should become negligible. Before it caused an almost constant 40% load (on average).

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@mobamoba commented on GitHub (May 10, 2020):

I just double-checked my phpinfo and, yes, apcu, memcached, redis, & zend opcache are all up and running so if it is the hash calculation you mentioned then for whatever reason perhaps it's not being cached.

@mobamoba commented on GitHub (May 10, 2020): I just double-checked my phpinfo and, yes, apcu, memcached, redis, & zend opcache are all up and running so if it is the hash calculation you mentioned then for whatever reason perhaps it's not being cached.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@kai-uwe-rommel commented on GitHub (May 15, 2020):

See my comment from 5 days ago. The situation has stayed this way. So for me the problem is more or less solved. APCu caching has significantly improved the situation. And I was able to further lessen the load by much increasing the sync interval (because not needed so often). I do not currently have time available to spend on xdebug'ing this case.

@kai-uwe-rommel commented on GitHub (May 15, 2020): See my comment from 5 days ago. The situation has stayed this way. So for me the problem is more or less solved. APCu caching has significantly improved the situation. And I was able to further lessen the load by much increasing the sync interval (because not needed so often). I do not currently have time available to spend on xdebug'ing this case.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@ghost commented on GitHub (Jun 5, 2020):

I have to report the same problem: On the Nextcloud server the load is about 90 %, several php-fpm7.3 processes come up. On the client the process WebExtensions does a load of about 80 %.

Server:
Raspbian Buster
Nextcloud 18.0.4.2
Bookmark app 3.0.13

Client:
Ubuntu 18.04
Firefox 77.0.1
Floccus 4.2.0.5

I have more than 3.000 objects in oc_bookmarks and about 375 objects in oc_bookmarks_folders.

@ghost commented on GitHub (Jun 5, 2020): I have to report the same problem: On the Nextcloud server the load is about 90 %, several php-fpm7.3 processes come up. On the client the process WebExtensions does a load of about 80 %. Server: Raspbian Buster Nextcloud 18.0.4.2 Bookmark app 3.0.13 Client: Ubuntu 18.04 Firefox 77.0.1 Floccus 4.2.0.5 I have more than 3.000 objects in oc_bookmarks and about 375 objects in oc_bookmarks_folders.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@stlehmann commented on GitHub (Oct 23, 2020):

I have the same problem with Nextcloud 20 and Bookmarks 4. Very high load on the server when syncing with Floccus.
How about limiting the number of sync calls per seconds in Floccus? There could be a parameter for this. I wouldn' t mind if Floccus syncs slowly in the background for a longer time duration. But a massive workload on the server is potentially problematic especially if multiple clients try to sync at the same time.

@stlehmann commented on GitHub (Oct 23, 2020): I have the same problem with Nextcloud 20 and Bookmarks 4. Very high load on the server when syncing with Floccus. How about limiting the number of sync calls per seconds in Floccus? There could be a parameter for this. I wouldn' t mind if Floccus syncs slowly in the background for a longer time duration. But a massive workload on the server is potentially problematic especially if multiple clients try to sync at the same time.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@omocinteractive commented on GitHub (Oct 26, 2020):

Same problem - very high load with v3.4.4 on Ubuntu 16.04/NC 19 and a good performed hardware server.

@omocinteractive commented on GitHub (Oct 26, 2020): Same problem - very high load with v3.4.4 on Ubuntu 16.04/NC 19 and a good performed hardware server.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (Oct 27, 2020):

I can confirm this. It's a regression in the server app. v3.4.5 is out now, which should fix this. v4.0.1 will be released soon with the same fix.

@marcelklehr commented on GitHub (Oct 27, 2020): I can confirm this. It's a regression in the server app. v3.4.5 is out now, which should fix this. v4.0.1 will be released soon with the same fix.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (Oct 29, 2020):

Releases are out. Closing. Shout, if this is not fixed for you.

@marcelklehr commented on GitHub (Oct 29, 2020): Releases are out. Closing. Shout, if this is not fixed for you.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@stlehmann commented on GitHub (Oct 29, 2020):

Thanks @marcelklehr. I just tried the new release and server load looks great. Thanks loads.

@stlehmann commented on GitHub (Oct 29, 2020): Thanks @marcelklehr. I just tried the new release and server load looks great. Thanks loads.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@faaaaabi commented on GitHub (Nov 6, 2020):

Is still have massive CPU load when using Bookmarks v4.0.5, Floccus v4.4.6, Nextcloud v20.0.1 in combination with syncing floccus. MySQL process list shows following (short):

MariaDB [(none)]> show processlist;
+-----+-----------+------------------+-----------+---------+------+---------------------+------------------------------------------------------------------------------------------------------+----------+
| Id  | User      | Host             | db        | Command | Time | State               | Info                                                                                                 | Progress |
+-----+-----------+------------------+-----------+---------+------+---------------------+------------------------------------------------------------------------------------------------------+----------+
| 254 | nextcloud | 172.23.0.3:43528 | nextcloud | Query   |   66 | Creating sort index | SELECT `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`cl |    0.000 |
| 261 | root      | localhost        | NULL      | Query   |    0 | starting            | show processlist                                                                                     |    0.000 |
+-----+-----------+------------------+-----------+---------+------+---------------------+------------------------------------------------------------------------------------------------------+----------+

(long):

*************************** 1. row ***************************
      Id: 261
    User: root
    Host: localhost
      db: NULL
 Command: Query
    Time: 0
   State: starting
    Info: show full processlist
Progress: 0.000
*************************** 2. row ***************************
      Id: 270
    User: nextcloud
    Host: 172.23.0.3:43874
      db: nextcloud
 Command: Query
    Time: 115
   State: Creating sort index
    Info: SELECT `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, GROUP_CONCAT(`tree`.`parent_folder`) AS `folders`, GROUP_CONCAT(`t`.`tag`) AS `tags` FROM `oc_bookmarks` `b` LEFT JOIN `oc_bookmarks_tree` `tree` ON (`b`.`id` = `tree`.`id`) AND (`tree`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_tags` `t` ON `t`.`bookmark_id` = `b`.`id` LEFT JOIN `oc_bookmarks_tree` `tr` ON (`tr`.`id` = `b`.`id`) AND (`tr`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_shared_folders` `sf` ON `tr`.`parent_folder` = `sf`.`folder_id` LEFT JOIN `oc_bookmarks_tree` `tr2` ON (`tr2`.`id` = `tr`.`parent_folder`) AND (`tr2`.`type` = 'folder') LEFT JOIN `oc_bookmarks_shared_folders` `sf2` ON `tr2`.`parent_folder` = `sf`.`folder_id` WHERE (`b`.`user_id` = 'fabi') OR (`sf`.`user_id` = 'fabi') OR (`sf2`.`user_id` = 'fabi') GROUP BY `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, `b`.`id` ORDER BY `b`.`lastmodified` DESC, `b`.`id` ASC LIMIT 300
Progress: 0.000

Seems to be busy with creating an index for sorting

@faaaaabi commented on GitHub (Nov 6, 2020): Is still have massive CPU load when using Bookmarks v4.0.5, Floccus v4.4.6, Nextcloud v20.0.1 in combination with syncing floccus. MySQL process list shows following **(short)**: ``` MariaDB [(none)]> show processlist; +-----+-----------+------------------+-----------+---------+------+---------------------+------------------------------------------------------------------------------------------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +-----+-----------+------------------+-----------+---------+------+---------------------+------------------------------------------------------------------------------------------------------+----------+ | 254 | nextcloud | 172.23.0.3:43528 | nextcloud | Query | 66 | Creating sort index | SELECT `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`cl | 0.000 | | 261 | root | localhost | NULL | Query | 0 | starting | show processlist | 0.000 | +-----+-----------+------------------+-----------+---------+------+---------------------+------------------------------------------------------------------------------------------------------+----------+ ``` **(long)**: ``` *************************** 1. row *************************** Id: 261 User: root Host: localhost db: NULL Command: Query Time: 0 State: starting Info: show full processlist Progress: 0.000 *************************** 2. row *************************** Id: 270 User: nextcloud Host: 172.23.0.3:43874 db: nextcloud Command: Query Time: 115 State: Creating sort index Info: SELECT `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, GROUP_CONCAT(`tree`.`parent_folder`) AS `folders`, GROUP_CONCAT(`t`.`tag`) AS `tags` FROM `oc_bookmarks` `b` LEFT JOIN `oc_bookmarks_tree` `tree` ON (`b`.`id` = `tree`.`id`) AND (`tree`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_tags` `t` ON `t`.`bookmark_id` = `b`.`id` LEFT JOIN `oc_bookmarks_tree` `tr` ON (`tr`.`id` = `b`.`id`) AND (`tr`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_shared_folders` `sf` ON `tr`.`parent_folder` = `sf`.`folder_id` LEFT JOIN `oc_bookmarks_tree` `tr2` ON (`tr2`.`id` = `tr`.`parent_folder`) AND (`tr2`.`type` = 'folder') LEFT JOIN `oc_bookmarks_shared_folders` `sf2` ON `tr2`.`parent_folder` = `sf`.`folder_id` WHERE (`b`.`user_id` = 'fabi') OR (`sf`.`user_id` = 'fabi') OR (`sf2`.`user_id` = 'fabi') GROUP BY `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, `b`.`id` ORDER BY `b`.`lastmodified` DESC, `b`.`id` ASC LIMIT 300 Progress: 0.000 ``` **Seems to be busy with creating an index for sorting**

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@mnalis commented on GitHub (Nov 10, 2020):

@faaaaabi what might be interesting is if you could paste the output of mysql commands in nextcloud DB:

EXPLAIN of slow query (just add full slow query after EXPLAIN command), so in this case:

EXPLAIN SELECT `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, GROUP_CONCAT(`tree`.`parent_folder`) AS `folders`, GROUP_CONCAT(`t`.`tag`) AS `tags` FROM `oc_bookmarks` `b` LEFT JOIN `oc_bookmarks_tree` `tree` ON (`b`.`id` = `tree`.`id`) AND (`tree`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_tags` `t` ON `t`.`bookmark_id` = `b`.`id` LEFT JOIN `oc_bookmarks_tree` `tr` ON (`tr`.`id` = `b`.`id`) AND (`tr`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_shared_folders` `sf` ON `tr`.`parent_folder` = `sf`.`folder_id` LEFT JOIN `oc_bookmarks_tree` `tr2` ON (`tr2`.`id` = `tr`.`parent_folder`) AND (`tr2`.`type` = 'folder') LEFT JOIN `oc_bookmarks_shared_folders` `sf2` ON `tr2`.`parent_folder` = `sf`.`folder_id` WHERE (`b`.`user_id` = 'fabi') OR (`sf`.`user_id` = 'fabi') OR (`sf2`.`user_id` = 'fabi') GROUP BY `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, `b`.`id` ORDER BY `b`.`lastmodified` DESC, `b`.`id` ASC LIMIT 300

INDEXs for tables mentioned in FROM/JOIN, in this case:

SHOW INDEX FROM oc_bookmarks;
SHOW INDEX FROM oc_bookmarks_tree;
SHOW INDEX FROM oc_bookmarks_tags;
SHOW INDEX FROM oc_bookmarks_shared_folders;

related mysql params:

SHOW variables like "%buffer%";
SHOW variables like "%table_size%";

@mnalis commented on GitHub (Nov 10, 2020): @faaaaabi what might be interesting is if you could paste the output of mysql commands in nextcloud DB: - EXPLAIN of slow query (just add full slow query after `EXPLAIN` command), so in this case: ``` EXPLAIN SELECT `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, GROUP_CONCAT(`tree`.`parent_folder`) AS `folders`, GROUP_CONCAT(`t`.`tag`) AS `tags` FROM `oc_bookmarks` `b` LEFT JOIN `oc_bookmarks_tree` `tree` ON (`b`.`id` = `tree`.`id`) AND (`tree`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_tags` `t` ON `t`.`bookmark_id` = `b`.`id` LEFT JOIN `oc_bookmarks_tree` `tr` ON (`tr`.`id` = `b`.`id`) AND (`tr`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_shared_folders` `sf` ON `tr`.`parent_folder` = `sf`.`folder_id` LEFT JOIN `oc_bookmarks_tree` `tr2` ON (`tr2`.`id` = `tr`.`parent_folder`) AND (`tr2`.`type` = 'folder') LEFT JOIN `oc_bookmarks_shared_folders` `sf2` ON `tr2`.`parent_folder` = `sf`.`folder_id` WHERE (`b`.`user_id` = 'fabi') OR (`sf`.`user_id` = 'fabi') OR (`sf2`.`user_id` = 'fabi') GROUP BY `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, `b`.`id` ORDER BY `b`.`lastmodified` DESC, `b`.`id` ASC LIMIT 300 ``` - INDEXs for tables mentioned in FROM/JOIN, in this case: ``` SHOW INDEX FROM oc_bookmarks; SHOW INDEX FROM oc_bookmarks_tree; SHOW INDEX FROM oc_bookmarks_tags; SHOW INDEX FROM oc_bookmarks_shared_folders; ``` - related mysql params: ``` SHOW variables like "%buffer%"; SHOW variables like "%table_size%"; ```

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@faaaaabi commented on GitHub (Nov 13, 2020):

Sure, here you go

MariaDB [nextcloud]> EXPLAIN SELECT `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, GROUP_CONCAT(`tree`.`parent_folder`) AS `folders`, GROUP_CONCAT(`t`.`tag`) AS `tags` FROM `oc_bookmarks` `b` LEFT JOIN `oc_bookmarks_tree` `tree` ON (`b`.`id` = `tree`.`id`) AND (`tree`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_tags` `t` ON `t`.`bookmark_id` = `b`.`id` LEFT JOIN `oc_bookmarks_tree` `tr` ON (`tr`.`id` = `b`.`id`) AND (`tr`.`type` = 'bookmark') LEFT JOIN `oc_bookmarks_shared_folders` `sf` ON `tr`.`parent_folder` = `sf`.`folder_id` LEFT JOIN `oc_bookmarks_tree` `tr2` ON (`tr2`.`id` = `tr`.`parent_folder`) AND (`tr2`.`type` = 'folder') LEFT JOIN `oc_bookmarks_shared_folders` `sf2` ON `tr2`.`parent_folder` = `sf`.`folder_id` WHERE (`b`.`user_id` = 'fabi') OR (`sf`.`user_id` = 'fabi') OR (`sf2`.`user_id` = 'fabi') GROUP BY `b`.`id`, `b`.`url`, `b`.`title`, `b`.`description`, `b`.`lastmodified`, `b`.`added`, `b`.`clickcount`, `b`.`last_preview`, `b`.`available`, `b`.`archived_file`, `b`.`user_id`, `b`.`id` ORDER BY `b`.`lastmodified` DESC, `b`.`id` ASC LIMIT 300
    -> ;
+------+-------------+-------+-------+-------------------------------------------+-----------------------+---------+----------------------------------+------+--------------------------------------------------------------+
| id   | select_type | table | type  | possible_keys                             | key                   | key_len | ref                              | rows | Extra                                                        |
+------+-------------+-------+-------+-------------------------------------------+-----------------------+---------+----------------------------------+------+--------------------------------------------------------------+
|    1 | SIMPLE      | b     | ALL   | IDX_3EE1CD04A76ED395,bookmarks_deadbyuser | NULL                  | NULL    | NULL                             | 796  | Using temporary; Using filesort                              |
|    1 | SIMPLE      | tree  | ref   | PRIMARY                                   | PRIMARY               | 90      | nextcloud.b.id,const             | 1    | Using where; Using index                                     |
|    1 | SIMPLE      | t     | ref   | bookmark_tag                              | bookmark_tag          | 9       | nextcloud.b.id                   | 1    | Using index                                                  |
|    1 | SIMPLE      | tr    | ref   | PRIMARY                                   | PRIMARY               | 90      | nextcloud.b.id,const             | 1    | Using where; Using index                                     |
|    1 | SIMPLE      | sf    | ALL   | bookmarks_shared_folder                   | NULL                  | NULL    | NULL                             | 1    | Using where; Using join buffer (flat, BNL join)              |
|    1 | SIMPLE      | tr2   | ref   | PRIMARY                                   | PRIMARY               | 90      | nextcloud.tr.parent_folder,const | 1    | Using where; Using index                                     |
|    1 | SIMPLE      | sf2   | index | NULL                                      | bookmarks_shared_user | 1022    | NULL                             | 1    | Using where; Using index; Using join buffer (flat, BNL join) |
+------+-------------+-------+-------+-------------------------------------------+-----------------------+---------+----------------------------------+------+--------------------------------------------------------------+

MariaDB [nextcloud]> SHOW INDEX FROM oc_bookmarks;
+--------------+------------+----------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table        | Non_unique | Key_name             | Seq_in_index | Column_name  | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------+------------+----------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| oc_bookmarks |          0 | PRIMARY              |            1 | id           | A         |         796 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks |          1 | IDX_3EE1CD04A76ED395 |            1 | user_id      | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks |          1 | IDX_3EE1CD04DF091378 |            1 | last_preview | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
| oc_bookmarks |          1 | bookmarks_modified   |            1 | lastmodified | A         |         796 |     NULL | NULL   | YES  | BTREE      |         |               |
| oc_bookmarks |          1 | bookmarks_deadbyuser |            1 | user_id      | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks |          1 | bookmarks_deadbyuser |            2 | available    | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks |          1 | bookmarks_dead       |            1 | available    | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
+--------------+------------+----------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

MariaDB [nextcloud]> SHOW INDEX FROM oc_bookmarks_tree;
+-------------------+------------+-------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table             | Non_unique | Key_name                | Seq_in_index | Column_name   | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------+------------+-------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| oc_bookmarks_tree |          0 | PRIMARY                 |            1 | id            | A         |       55116 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks_tree |          0 | PRIMARY                 |            2 | type          | A         |       55116 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks_tree |          0 | PRIMARY                 |            3 | parent_folder | A         |       55116 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks_tree |          1 | bookmarks_tree_parent   |            1 | parent_folder | A         |       27558 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks_tree |          1 | bookmarks_tree_parent_i |            1 | parent_folder | A         |       18372 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks_tree |          1 | bookmarks_tree_parent_i |            2 | index         | A         |       55116 |     NULL | NULL   |      | BTREE      |         |               |
+-------------------+------------+-------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
6 rows in set (0.001 sec)

MariaDB [nextcloud]> SHOW INDEX FROM oc_bookmarks_tags;
+-------------------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table             | Non_unique | Key_name     | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| oc_bookmarks_tags |          0 | bookmark_tag |            1 | bookmark_id | A         |           0 |     NULL | NULL   | YES  | BTREE      |         |               |
| oc_bookmarks_tags |          0 | bookmark_tag |            2 | tag         | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
+-------------------+------------+--------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

MariaDB [nextcloud]> SHOW INDEX FROM oc_bookmarks_shared_folders;
+-----------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table                       | Non_unique | Key_name                | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| oc_bookmarks_shared_folders |          0 | PRIMARY                 |            1 | id          | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks_shared_folders |          1 | bookmarks_shared_user   |            1 | user_id     | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
| oc_bookmarks_shared_folders |          1 | bookmarks_shared_folder |            1 | folder_id   | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
+-----------------------------+------------+-------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

MariaDB [nextcloud]> SHOW variables like "%buffer%";
+-------------------------------------+----------------+
| Variable_name                       | Value          |
+-------------------------------------+----------------+
| aria_pagecache_buffer_size          | 134217728      |
| aria_sort_buffer_size               | 268434432      |
| bulk_insert_buffer_size             | 16777216       |
| innodb_buffer_pool_chunk_size       | 134217728      |
| innodb_buffer_pool_dump_at_shutdown | ON             |
| innodb_buffer_pool_dump_now         | OFF            |
| innodb_buffer_pool_dump_pct         | 25             |
| innodb_buffer_pool_filename         | ib_buffer_pool |
| innodb_buffer_pool_instances        | 1              |
| innodb_buffer_pool_load_abort       | OFF            |
| innodb_buffer_pool_load_at_startup  | ON             |
| innodb_buffer_pool_load_now         | OFF            |
| innodb_buffer_pool_size             | 1073741824     |
| innodb_change_buffer_max_size       | 25             |
| innodb_change_buffering             | all            |
| innodb_log_buffer_size              | 8388608        |
| innodb_sort_buffer_size             | 1048576        |
| join_buffer_size                    | 262144         |
| join_buffer_space_limit             | 2097152        |
| key_buffer_size                     | 134217728      |
| mrr_buffer_size                     | 262144         |
| myisam_sort_buffer_size             | 536870912      |
| net_buffer_length                   | 16384          |
| preload_buffer_size                 | 32768          |
| read_buffer_size                    | 2097152        |
| read_rnd_buffer_size                | 1048576        |
| sort_buffer_size                    | 4194304        |
| sql_buffer_result                   | OFF            |
+-------------------------------------+----------------+
28 rows in set (0.003 sec)

MariaDB [nextcloud]> SHOW variables like "%table_size%";
+-----------------------+----------------------+
| Variable_name         | Value                |
+-----------------------+----------------------+
| max_heap_table_size   | 33554432             |
| tmp_disk_table_size   | 18446744073709551615 |
| tmp_memory_table_size | 33554432             |
| tmp_table_size        | 33554432             |
+-----------------------+----------------------+
4 rows in set (0.001 sec)

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@faaaaabi commented on GitHub (Jan 16, 2021):

Did you gather any new inside, yet @mnalis ?

@faaaaabi commented on GitHub (Jan 16, 2021): Did you gather any new inside, yet @mnalis ?

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@marcelklehr commented on GitHub (Jan 16, 2021):

@faaaaabi Could you open a new issue for this, please? It looks like some indices for oc_bookmarks are broken or don't get updated or are stuck updating.

@marcelklehr commented on GitHub (Jan 16, 2021): @faaaaabi Could you open a new issue for this, please? It looks like some indices for oc_bookmarks are broken or don't get updated or are stuck updating.

kerem commented

2026-02-25 22:36:54 +03:00

Author

Owner

@github-actions[bot] commented on GitHub (Mar 21, 2023):

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions[bot] commented on GitHub (Mar 21, 2023): This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Rows
Columns

[GH-ISSUE #476] High Load in Nextcloud at sync #350