[PR #1460] Fix nginx-ui issue 1455 #5725

New issue

Closed

opened 2026-03-01 15:42:24 +03:00 by kerem · 0 comments

kerem commented

2026-03-01 15:42:24 +03:00

Owner

Original Pull Request: https://github.com/0xJacky/nginx-ui/pull/1460

State: closed
Merged: Yes

Fixes #1455 by using per-file persisted metadata for incremental indexing decisions to prevent unnecessary re-indexing caused by aggregated log sizes.

The previous incremental indexing logic could incorrectly trigger re-indexing due to log.LastSize being an aggregated value across rotated log files (e.g., access.log and access.log.1). This led to false positives where the system believed a file had shrunk or grown, causing infinite rebuilds. This PR injects the actual PersistenceManager to use accurate per-file metadata for LastModified and LastSize, ensuring indexing only occurs when truly needed. A fallback mechanism is also included for cases where persistence data isn't available, clamping the LastSize to prevent similar issues.

Note

Switch incremental indexing to per-file persisted metadata with a safe fallback, add rotation-aware doc count updates, and include unit tests.

Incremental indexing logic:

Use persistence (logIndexProvider) to decide indexing via per-file metadata (GetLogIndex/NeedsIndexing).

Fallback path clamps aggregated LastSize, handles never-indexed files, and refines change detection.

Pass persistence from LogFileManager into needsIncrementalIndexing and job loop.

Indexing execution (queueIncrementalIndexing):

Detect log rotation by size decrease; reset document count on rotation, otherwise accumulate.

Persist indexing metadata (SaveIndexMetadata) and manage statuses via persistence-backed setFileIndexStatus with simple queue ordering.

Tests:

Add internal/cron/incremental_indexing_test.go with stub provider covering unchanged file skip and growth detection.

^{Written by Cursor Bugbot for commit 8e9fa7d0e0. This will update automatically on new commits. Configure here.}

**Original Pull Request:** https://github.com/0xJacky/nginx-ui/pull/1460 **State:** closed **Merged:** Yes --- Fixes #1455 by using per-file persisted metadata for incremental indexing decisions to prevent unnecessary re-indexing caused by aggregated log sizes. The previous incremental indexing logic could incorrectly trigger re-indexing due to `log.LastSize` being an aggregated value across rotated log files (e.g., `access.log` and `access.log.1`). This led to false positives where the system believed a file had shrunk or grown, causing infinite rebuilds. This PR injects the actual `PersistenceManager` to use accurate per-file metadata for `LastModified` and `LastSize`, ensuring indexing only occurs when truly needed. A fallback mechanism is also included for cases where persistence data isn't available, clamping the `LastSize` to prevent similar issues. --- <a href="https://cursor.com/background-agent?bcId=bc-98a269d0-ef88-4297-bfa0-5c088ce6e234"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/open-in-cursor-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/open-in-cursor-light.svg"><img alt="Open in Cursor" src="https://cursor.com/open-in-cursor.svg"></picture></a> <a href="https://cursor.com/agents?id=bc-98a269d0-ef88-4297-bfa0-5c088ce6e234"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/open-in-web-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/open-in-web-light.svg"><img alt="Open in Web" src="https://cursor.com/open-in-web.svg"></picture></a>  --- > [!NOTE] > Switch incremental indexing to per-file persisted metadata with a safe fallback, add rotation-aware doc count updates, and include unit tests. > > - **Incremental indexing logic**: > - Use persistence (`logIndexProvider`) to decide indexing via per-file metadata (`GetLogIndex`/`NeedsIndexing`). > - Fallback path clamps aggregated `LastSize`, handles never-indexed files, and refines change detection. > - Pass persistence from `LogFileManager` into `needsIncrementalIndexing` and job loop. > - **Indexing execution** (`queueIncrementalIndexing`): > - Detect log rotation by size decrease; reset document count on rotation, otherwise accumulate. > - Persist indexing metadata (`SaveIndexMetadata`) and manage statuses via persistence-backed `setFileIndexStatus` with simple queue ordering. > - **Tests**: > - Add `internal/cron/incremental_indexing_test.go` with stub provider covering unchanged file skip and growth detection. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 8e9fa7d0e0cc776648f225cdec097d4cfe7ea00b. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>