[PR #602] Add nightly agent CLI integration tests #597

New issue

Open

opened 2026-03-02 04:14:05 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 04:14:05 +03:00

Owner

Original Pull Request: https://github.com/git-ai-project/git-ai/pull/602

State: open
Merged: No

Summary

Adds .github/workflows/nightly-agent-integration.yml — a two-tier nightly workflow that installs real agent CLI binaries and verifies git-ai hook wiring and attribution end-to-end
Adds scripts/nightly/ with four helper scripts implementing the test logic
Adds NIGHTLY_INTEGRATION_PLAN.md documenting the full design rationale and open questions

Test Architecture

Tier 1 — Hook Wiring (no API keys, free)

Builds git-ai from source, installs each agent CLI (Claude Code, Codex, Gemini, Droid, OpenCode) at both stable and latest versions via a dynamic matrix, then:

Runs git-ai install and verifies the correct checkpoint commands appear in each agent's config file
Exercises the full attribution pipeline with synthetic checkpoint data (via the agent-v1 preset)

Tier 2 — Live Integration (requires API key secrets)

Runs each agent with a minimal deterministic prompt ("create hello.txt, commit it"), then verifies the file was created, a commit landed, and authorship notes are present in refs/notes/ai. Pre-release failures are non-blocking (continue-on-error: true).

Hook config paths (verified against `src/mdm/agents/*.rs`)

Agent	Config file
Claude Code	`~/.claude/settings.json`
Codex	`~/.codex/config.toml`
Gemini CLI	`~/.gemini/settings.json`
Droid	`~/.factory/settings.json`
OpenCode	`~/.config/opencode/plugin/git-ai.ts`

Secrets required (Tier 2 only)

ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, FACTORY_API_KEY, SLACK_BOT_TOKEN, SLACK_CHANNEL_ID

Tier 1 runs without any secrets.

Cost estimate

~$0.05–0.25/night (weekdays only). See NIGHTLY_INTEGRATION_PLAN.md §6 for cost management strategies.

Test plan

Verify workflow YAML parses correctly in Actions UI
Trigger workflow_dispatch with tier: tier1 to validate hook-wiring jobs (no API keys needed)
Add ANTHROPIC_API_KEY secret and trigger tier: both to validate Claude Code Tier 2 end-to-end
Review open questions in NIGHTLY_INTEGRATION_PLAN.md §13 before enabling the nightly schedule

🤖 Generated with Claude Code

**Original Pull Request:** https://github.com/git-ai-project/git-ai/pull/602 **State:** open **Merged:** No --- ## Summary - Adds `.github/workflows/nightly-agent-integration.yml` — a two-tier nightly workflow that installs real agent CLI binaries and verifies git-ai hook wiring and attribution end-to-end - Adds `scripts/nightly/` with four helper scripts implementing the test logic - Adds `NIGHTLY_INTEGRATION_PLAN.md` documenting the full design rationale and open questions ## Test Architecture **Tier 1 — Hook Wiring (no API keys, free)** Builds `git-ai` from source, installs each agent CLI (Claude Code, Codex, Gemini, Droid, OpenCode) at both `stable` and `latest` versions via a dynamic matrix, then: 1. Runs `git-ai install` and verifies the correct checkpoint commands appear in each agent's config file 2. Exercises the full attribution pipeline with synthetic checkpoint data (via the `agent-v1` preset) **Tier 2 — Live Integration (requires API key secrets)** Runs each agent with a minimal deterministic prompt ("create hello.txt, commit it"), then verifies the file was created, a commit landed, and authorship notes are present in `refs/notes/ai`. Pre-release failures are non-blocking (`continue-on-error: true`). ## Hook config paths (verified against `src/mdm/agents/*.rs`) | Agent | Config file | |-------|-------------| | Claude Code | `~/.claude/settings.json` | | Codex | `~/.codex/config.toml` | | Gemini CLI | `~/.gemini/settings.json` | | Droid | `~/.factory/settings.json` | | OpenCode | `~/.config/opencode/plugin/git-ai.ts` | ## Secrets required (Tier 2 only) `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, `FACTORY_API_KEY`, `SLACK_BOT_TOKEN`, `SLACK_CHANNEL_ID` Tier 1 runs without any secrets. ## Cost estimate ~$0.05–0.25/night (weekdays only). See `NIGHTLY_INTEGRATION_PLAN.md` §6 for cost management strategies. ## Test plan - [ ] Verify workflow YAML parses correctly in Actions UI - [ ] Trigger `workflow_dispatch` with `tier: tier1` to validate hook-wiring jobs (no API keys needed) - [ ] Add `ANTHROPIC_API_KEY` secret and trigger `tier: both` to validate Claude Code Tier 2 end-to-end - [ ] Review open questions in `NIGHTLY_INTEGRATION_PLAN.md` §13 before enabling the nightly schedule 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- <a href="https://app.devin.ai/review/git-ai-project/git-ai/pull/602" target="_blank"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1"> <img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open with Devin"> </picture> </a>