[PR #15] [MERGED] Implement parallel processing #102

New issue

Closed

opened 2026-02-27 15:55:09 +03:00 by kerem · 0 comments

kerem commented

2026-02-27 15:55:09 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/RayLabsHQ/gitea-mirror/pull/15
Author: @arunavo4
Created: 5/22/2025
Status: ✅ Merged
Merged: 5/22/2025
Merged by: @arunavo4

Base: main ← Head: parallel-jobs

📝 Commits (4)

f4bc28e Implement parallel processing with retry logic for repository mirroring and syncing operations
abe3113 feat: enhance job resilience with new database schema and recovery mechanisms
6ab7f0a fix: add missing --bun flag to vitest and astro build commands
894be88 feat: migrate testing framework to Bun and update test configurations

📊 Changes

27 files changed (+2625 additions, -172 deletions)

View changed files

📝 .github/workflows/astro-build-test.yml (+2 -2)
📝 docker-entrypoint.sh (+32 -2)
➕ docs/testing.md (+127 -0)
📝 package.json (+7 -5)
📝 scripts/manage-db.ts (+22 -0)
➕ scripts/update-mirror-jobs-table.ts (+133 -0)
➕ src/lib/db/index.test.ts (+42 -0)
📝 src/lib/db/index.ts (+12 -0)
📝 src/lib/db/schema.ts (+12 -0)
➕ src/lib/gitea.test.ts (+120 -0)
📝 src/lib/gitea.ts (+148 -93)
📝 src/lib/helpers.ts (+205 -1)
➕ src/lib/recovery.ts (+224 -0)
➕ src/lib/utils.test.ts (+110 -0)
➕ src/lib/utils/concurrency.test.ts (+167 -0)
➕ src/lib/utils/concurrency.ts (+292 -0)
➕ src/middleware.ts (+22 -0)
➕ src/pages/api/gitea/test-connection.test.ts (+187 -0)
➕ src/pages/api/github/test-connection.test.ts (+133 -0)
➕ src/pages/api/health.test.ts (+154 -0)

...and 7 more files

📄 Description

This pull request introduces significant improvements to mirroring operations between GitHub and Gitea, focusing on concurrency control, retry mechanisms, and logging enhancements. The most notable changes include the implementation of a new utility for parallel processing with retries, integration of this utility into mirroring and retry workflows, and enhanced logging for better tracking of operations.

Concurrency and Retry Utilities:

Added processInParallel and processWithRetry utilities in src/lib/utils/concurrency.ts to enable parallel processing with configurable concurrency limits, automatic retries, and progress tracking. These utilities are used across multiple mirroring and retry workflows.

Mirroring Enhancements:

Updated mirrorGitHubOrgToGitea in src/lib/gitea.ts to process repositories in parallel using processWithRetry, with a concurrency limit of 3 and retry functionality. Added logs for progress, retries, and completion. [1] [2]
Enhanced mirrorGitRepoIssuesToGitea in src/lib/gitea.ts to filter out pull requests, process issues and comments in parallel, and log progress and retries. Concurrency limits for issues and comments are set to 3 and 5, respectively. [1] [2] [3]

API Workflow Updates:

Refactored POST handlers in src/pages/api/job/mirror-org.ts, src/pages/api/job/mirror-repo.ts, and src/pages/api/job/retry-repo.ts to use processWithRetry for parallel processing of organizations and repositories. Added concurrency limits, retry mechanisms, and detailed logging for mirroring and retry operations. [1] [2] [3] [4] [5] [6]

Logging Improvements:

Introduced detailed logs for tracking the start, progress, retries, and completion of mirroring and retry tasks across organizations, repositories, issues, and comments. This enhances visibility into the operations and aids debugging. [1] [2] [3] [4]

These changes collectively improve the scalability, resilience, and observability of the mirroring system.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/RayLabsHQ/gitea-mirror/pull/15 **Author:** [@arunavo4](https://github.com/arunavo4) **Created:** 5/22/2025 **Status:** ✅ Merged **Merged:** 5/22/2025 **Merged by:** [@arunavo4](https://github.com/arunavo4) **Base:** `main` ← **Head:** `parallel-jobs` --- ### 📝 Commits (4) - [`f4bc28e`](https://github.com/RayLabsHQ/gitea-mirror/commit/f4bc28e6c2845f50106f4a51515aafb9ef7f8237) Implement parallel processing with retry logic for repository mirroring and syncing operations - [`abe3113`](https://github.com/RayLabsHQ/gitea-mirror/commit/abe31137551f2b170c7a28fd9e74c69e99a4cd1f) feat: enhance job resilience with new database schema and recovery mechanisms - [`6ab7f0a`](https://github.com/RayLabsHQ/gitea-mirror/commit/6ab7f0a5a0b1f11285047b630328bd0a38393bf4) fix: add missing --bun flag to vitest and astro build commands - [`894be88`](https://github.com/RayLabsHQ/gitea-mirror/commit/894be88a28a058d3e9f068f666afda10144d2cdd) feat: migrate testing framework to Bun and update test configurations ### 📊 Changes **27 files changed** (+2625 additions, -172 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/astro-build-test.yml` (+2 -2) 📝 `docker-entrypoint.sh` (+32 -2) ➕ `docs/testing.md` (+127 -0) 📝 `package.json` (+7 -5) 📝 `scripts/manage-db.ts` (+22 -0) ➕ `scripts/update-mirror-jobs-table.ts` (+133 -0) ➕ `src/lib/db/index.test.ts` (+42 -0) 📝 `src/lib/db/index.ts` (+12 -0) 📝 `src/lib/db/schema.ts` (+12 -0) ➕ `src/lib/gitea.test.ts` (+120 -0) 📝 `src/lib/gitea.ts` (+148 -93) 📝 `src/lib/helpers.ts` (+205 -1) ➕ `src/lib/recovery.ts` (+224 -0) ➕ `src/lib/utils.test.ts` (+110 -0) ➕ `src/lib/utils/concurrency.test.ts` (+167 -0) ➕ `src/lib/utils/concurrency.ts` (+292 -0) ➕ `src/middleware.ts` (+22 -0) ➕ `src/pages/api/gitea/test-connection.test.ts` (+187 -0) ➕ `src/pages/api/github/test-connection.test.ts` (+133 -0) ➕ `src/pages/api/health.test.ts` (+154 -0) _...and 7 more files_ </details> ### 📄 Description This pull request introduces significant improvements to mirroring operations between GitHub and Gitea, focusing on concurrency control, retry mechanisms, and logging enhancements. The most notable changes include the implementation of a new utility for parallel processing with retries, integration of this utility into mirroring and retry workflows, and enhanced logging for better tracking of operations. ### Concurrency and Retry Utilities: * Added `processInParallel` and `processWithRetry` utilities in `src/lib/utils/concurrency.ts` to enable parallel processing with configurable concurrency limits, automatic retries, and progress tracking. These utilities are used across multiple mirroring and retry workflows. ### Mirroring Enhancements: * Updated `mirrorGitHubOrgToGitea` in `src/lib/gitea.ts` to process repositories in parallel using `processWithRetry`, with a concurrency limit of 3 and retry functionality. Added logs for progress, retries, and completion. [[1]](diffhunk://#diff-0a581dfeb72efcc268fad951dd0ec01365a88e8eb31274bf2bcdab4bf8d6de26L604-R619) [[2]](diffhunk://#diff-0a581dfeb72efcc268fad951dd0ec01365a88e8eb31274bf2bcdab4bf8d6de26L617-R658) * Enhanced `mirrorGitRepoIssuesToGitea` in `src/lib/gitea.ts` to filter out pull requests, process issues and comments in parallel, and log progress and retries. Concurrency limits for issues and comments are set to 3 and 5, respectively. [[1]](diffhunk://#diff-0a581dfeb72efcc268fad951dd0ec01365a88e8eb31274bf2bcdab4bf8d6de26L840-R885) [[2]](diffhunk://#diff-0a581dfeb72efcc268fad951dd0ec01365a88e8eb31274bf2bcdab4bf8d6de26L854-R905) [[3]](diffhunk://#diff-0a581dfeb72efcc268fad951dd0ec01365a88e8eb31274bf2bcdab4bf8d6de26L935-R1016) ### API Workflow Updates: * Refactored `POST` handlers in `src/pages/api/job/mirror-org.ts`, `src/pages/api/job/mirror-repo.ts`, and `src/pages/api/job/retry-repo.ts` to use `processWithRetry` for parallel processing of organizations and repositories. Added concurrency limits, retry mechanisms, and detailed logging for mirroring and retry operations. [[1]](diffhunk://#diff-3d9b8b867c2a9354edb8cef0067dfee8f059ea9a989c304951f90dda40040c4eR9-R10) [[2]](diffhunk://#diff-3d9b8b867c2a9354edb8cef0067dfee8f059ea9a989c304951f90dda40040c4eL64-R132) [[3]](diffhunk://#diff-36cecc4f66a820e8c9afa1284d904baffbe6be484a3f451c6b17888177b7a8eeR11-R12) [[4]](diffhunk://#diff-36cecc4f66a820e8c9afa1284d904baffbe6be484a3f451c6b17888177b7a8eeL66-R85) [[5]](diffhunk://#diff-6c5a8e72408e63047df0f17fed8231345cf72e7d073be3c851ad52f4b387d976R13-R14) [[6]](diffhunk://#diff-6c5a8e72408e63047df0f17fed8231345cf72e7d073be3c851ad52f4b387d976L68-R84) ### Logging Improvements: * Introduced detailed logs for tracking the start, progress, retries, and completion of mirroring and retry tasks across organizations, repositories, issues, and comments. This enhances visibility into the operations and aids debugging. [[1]](diffhunk://#diff-0a581dfeb72efcc268fad951dd0ec01365a88e8eb31274bf2bcdab4bf8d6de26L617-R658) [[2]](diffhunk://#diff-0a581dfeb72efcc268fad951dd0ec01365a88e8eb31274bf2bcdab4bf8d6de26L935-R1016) [[3]](diffhunk://#diff-3d9b8b867c2a9354edb8cef0067dfee8f059ea9a989c304951f90dda40040c4eL64-R132) [[4]](diffhunk://#diff-6c5a8e72408e63047df0f17fed8231345cf72e7d073be3c851ad52f4b387d976R97-R110) These changes collectively improve the scalability, resilience, and observability of the mirroring system. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>