[PR #9] [MERGED] feat(telemetry): Add per-iteration telemetry capture #85

New issue

Closed

opened 2026-02-27 10:22:10 +03:00 by kerem · 0 comments

kerem commented

2026-02-27 10:22:10 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/mikeyobrien/ralph-orchestrator/pull/9
Author: @sachafaust
Created: 12/28/2025
Status: ✅ Merged
Merged: 12/30/2025
Merged by: @mikeyobrien

Base: main ← Head: feature/per-iteration-telemetry

📝 Commits (10+)

8dec3ef feat(metrics): Add TriggerReason enum for per-iteration telemetry
335e0b4 docs: Update PROMPT.md with iteration 1 progress
20005a0 Extend IterationStats.record_iteration() with telemetry fields
5bb2c5e Update scratchpad: record_iteration() extension complete
e589ce0 test(metrics): Add unit tests for TriggerReason enum and telemetry fields
5396d09 Ralph checkpoint 3
2262a80 feat(orchestrator): add IterationStats alongside Metrics
77d66ec docs: update scratchpad with Phase 2 Step 1 progress
819754b Add _determine_trigger_reason() method to orchestrator
6135c16 feat(orchestrator): record per-iteration telemetry in arun() loop

📊 Changes

5 files changed (+569 additions, -20 deletions)

View changed files

📝 ralph.yml (+4 -0)
📝 src/ralph_orchestrator/metrics.py (+37 -1)
📝 src/ralph_orchestrator/orchestrator.py (+125 -19)
📝 tests/test_metrics.py (+241 -0)
📝 tests/test_orchestrator.py (+162 -0)

📄 Description

Summary

Adds per-iteration telemetry capture to ralph-orchestrator, enabling detailed analysis of orchestration patterns.

Changes

TriggerReason enum - Categorizes why each iteration was triggered (initial, task_incomplete, recovery, etc.)
Extended IterationStats.record_iteration() - Now captures trigger_reason, output_preview (truncated), tokens_used, cost, and tools_used
New _determine_trigger_reason() method - Analyzes orchestrator state to classify iteration triggers
Enhanced _print_summary() - Saves full iteration list with per-iteration details to metrics JSON

Output Format

{
  "summary": { /* backward compatible summary stats */ },
  "iterations": [
    {
      "iteration": 1,
      "duration": 5.2,
      "success": true,
      "trigger_reason": "initial",
      "output_preview": "Task started...",
      "tokens_used": 1500,
      "cost": 0.025
    }
  ],
  "cost": { /* cost breakdown */ },
  "analysis": { "avg_iteration_duration": 4.1, "success_rate": 0.95 }
}

Test Plan

All 68 tests pass (including 23 new telemetry tests)
Backward compatibility verified (old record_iteration() calls still work)
Output preview truncation at 500 chars verified
TriggerReason enum values tested

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/mikeyobrien/ralph-orchestrator/pull/9 **Author:** [@sachafaust](https://github.com/sachafaust) **Created:** 12/28/2025 **Status:** ✅ Merged **Merged:** 12/30/2025 **Merged by:** [@mikeyobrien](https://github.com/mikeyobrien) **Base:** `main` ← **Head:** `feature/per-iteration-telemetry` --- ### 📝 Commits (10+) - [`8dec3ef`](https://github.com/mikeyobrien/ralph-orchestrator/commit/8dec3ef74c2495a95bf5bad132f1de228ee35811) feat(metrics): Add TriggerReason enum for per-iteration telemetry - [`335e0b4`](https://github.com/mikeyobrien/ralph-orchestrator/commit/335e0b40e75082184a0a4d03840a1ce47bd4fae0) docs: Update PROMPT.md with iteration 1 progress - [`20005a0`](https://github.com/mikeyobrien/ralph-orchestrator/commit/20005a08a0e01562f2aeb279d12bb198000e5c39) Extend IterationStats.record_iteration() with telemetry fields - [`5bb2c5e`](https://github.com/mikeyobrien/ralph-orchestrator/commit/5bb2c5e07df802ddc65f593652f79a4afd0ff6b0) Update scratchpad: record_iteration() extension complete - [`e589ce0`](https://github.com/mikeyobrien/ralph-orchestrator/commit/e589ce0a42f14a7734d01bb3a346f8f976ed23a1) test(metrics): Add unit tests for TriggerReason enum and telemetry fields - [`5396d09`](https://github.com/mikeyobrien/ralph-orchestrator/commit/5396d09bae34a0454ccb9d341834ebdf1552a5c6) Ralph checkpoint 3 - [`2262a80`](https://github.com/mikeyobrien/ralph-orchestrator/commit/2262a8058ab68cde527190d56c875b03fed30821) feat(orchestrator): add IterationStats alongside Metrics - [`77d66ec`](https://github.com/mikeyobrien/ralph-orchestrator/commit/77d66ec36fc3bfd6cdbfc95842b3ce2430bd945e) docs: update scratchpad with Phase 2 Step 1 progress - [`819754b`](https://github.com/mikeyobrien/ralph-orchestrator/commit/819754b0684e7e91a6b98588b4de56ec186a0c70) Add _determine_trigger_reason() method to orchestrator - [`6135c16`](https://github.com/mikeyobrien/ralph-orchestrator/commit/6135c16dd16f83119fc15a49f7dd9f9e008c91b7) feat(orchestrator): record per-iteration telemetry in arun() loop ### 📊 Changes **5 files changed** (+569 additions, -20 deletions) <details> <summary>View changed files</summary> 📝 `ralph.yml` (+4 -0) 📝 `src/ralph_orchestrator/metrics.py` (+37 -1) 📝 `src/ralph_orchestrator/orchestrator.py` (+125 -19) 📝 `tests/test_metrics.py` (+241 -0) 📝 `tests/test_orchestrator.py` (+162 -0) </details> ### 📄 Description ## Summary Adds per-iteration telemetry capture to ralph-orchestrator, enabling detailed analysis of orchestration patterns. ### Changes - **`TriggerReason` enum** - Categorizes why each iteration was triggered (initial, task_incomplete, recovery, etc.) - **Extended `IterationStats.record_iteration()`** - Now captures trigger_reason, output_preview (truncated), tokens_used, cost, and tools_used - **New `_determine_trigger_reason()` method** - Analyzes orchestrator state to classify iteration triggers - **Enhanced `_print_summary()`** - Saves full iteration list with per-iteration details to metrics JSON ### Output Format ```json { "summary": { /* backward compatible summary stats */ }, "iterations": [ { "iteration": 1, "duration": 5.2, "success": true, "trigger_reason": "initial", "output_preview": "Task started...", "tokens_used": 1500, "cost": 0.025 } ], "cost": { /* cost breakdown */ }, "analysis": { "avg_iteration_duration": 4.1, "success_rate": 0.95 } } ``` ## Test Plan - [x] All 68 tests pass (including 23 new telemetry tests) - [x] Backward compatibility verified (old `record_iteration()` calls still work) - [x] Output preview truncation at 500 chars verified - [x] TriggerReason enum values tested 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>