[PR #482] Fix git core compatibility harness baseline handling #503

New issue

Closed

opened 2026-03-02 04:13:49 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 04:13:49 +03:00

Owner

Original Pull Request: https://github.com/git-ai-project/git-ai/pull/482

State: closed
Merged: Yes

Motivation

Ensure the Git core compatibility harness is runnable in CI and locally by building the upstream Git checkout before running tests.
Avoid reporting normal test non-zero exits as harness errors and instead only flag true harness/parsing issues.
Record the current observed failures as a baseline whitelist so CI only fails on new regressions.

Description

Add ensure_git_build() to tests/git-compat/run-core-tests.py to run make in the cloned Git repo when GIT-BUILD-OPTIONS is missing.
Set a default test hash via GIT_TEST_DEFAULT_HASH=sha1 in the harness environment so tests that expect SHA-1 run correctly.
Parse and report only TAP/parse errors as harness issues (refined regex and parse_summary_issues()), while continuing to parse test failures for whitelist comparison.
Update tests/git-compat/whitelist.csv with the current baseline of failing tests so the harness only fails for new, unexpected regressions.

Testing

Built the project with cargo build --release --bin git-ai which completed successfully.
Ran the harness with python3 tests/git-compat/run-core-tests.py, which cloned and built upstream Git, ran the selected prove test subset, and returned success because all remaining failures were applied to the updated whitelist.
The harness now exits 0 when tests are either passing or explicitly whitelisted, and returns non-zero when parse/harness errors are detected.

Codex Task

**Original Pull Request:** https://github.com/git-ai-project/git-ai/pull/482 **State:** closed **Merged:** Yes --- ### Motivation - Ensure the Git core compatibility harness is runnable in CI and locally by building the upstream Git checkout before running tests. - Avoid reporting normal test non-zero exits as harness errors and instead only flag true harness/parsing issues. - Record the current observed failures as a baseline whitelist so CI only fails on new regressions. ### Description - Add `ensure_git_build()` to `tests/git-compat/run-core-tests.py` to run `make` in the cloned Git repo when `GIT-BUILD-OPTIONS` is missing. - Set a default test hash via `GIT_TEST_DEFAULT_HASH=sha1` in the harness environment so tests that expect SHA-1 run correctly. - Parse and report only TAP/parse errors as harness issues (refined regex and `parse_summary_issues()`), while continuing to parse test failures for whitelist comparison. - Update `tests/git-compat/whitelist.csv` with the current baseline of failing tests so the harness only fails for new, unexpected regressions. ### Testing - Built the project with `cargo build --release --bin git-ai` which completed successfully. - Ran the harness with `python3 tests/git-compat/run-core-tests.py`, which cloned and built upstream Git, ran the selected `prove` test subset, and returned success because all remaining failures were applied to the updated whitelist. - The harness now exits `0` when tests are either passing or explicitly whitelisted, and returns non-zero when parse/harness errors are detected. ------ [Codex Task](https://chatgpt.com/codex/tasks/task_e_6988c215bd74832c90d2741cac5e93df)  --- <a href="https://app.devin.ai/review/git-ai-project/git-ai/pull/482" target="_blank"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1"> <img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open with Devin"> </picture> </a>