starred/ralph-orchestrator

Fork 0

mirror of https://github.com/mikeyobrien/ralph-orchestrator.git synced 2026-04-25 23:25:57 +03:00

[GH-ISSUE #47] completion_promise not detected - orchestrator continues iterating after agent outputs completion string #14

New issue

Closed

opened 2026-02-27 10:21:46 +03:00 by kerem · 4 comments

kerem commented

2026-02-27 10:21:46 +03:00

Owner

Originally created by @ramarivera on GitHub (Jan 11, 2026).
Original GitHub issue: https://github.com/mikeyobrien/ralph-orchestrator/issues/47

Bug Description

The completion_promise configuration option is not being detected in agent output. The orchestrator continues iterating indefinitely even when the agent clearly outputs the configured completion string.

Expected Behavior

When the agent outputs the string specified in completion_promise (e.g., LOOP_COMPLETE), the orchestrator should detect it and stop the loop.

Actual Behavior

The orchestrator:

Receives agent output containing the completion string
Logs "Iteration X completed successfully"
Immediately starts the next iteration
Repeats indefinitely until manually killed or max_iterations reached

Configuration

completion_promise: "LOOP_COMPLETE"
max_iterations: 100
agent: claude

Evidence

Agent Output (Iteration 52)

## 📊 Summary - Task Complete

The orchestrated task has reached **LOOP_COMPLETE** status...

No further action is needed from this orchestration loop.

    1 LOOP_COMPLETE

Orchestrator Log (immediately after)

2026-01-11 12:48:30,160 - ralph-orchestrator - INFO - Estimated cost: $0.0084 (total: $0.6246)
✓ Iteration 52 completed successfully

2026-01-11 12:48:32,165 - ralph-orchestrator.safety - WARNING - High iteration count: 52

=== RALPH ITERATION 53 ===

The string LOOP_COMPLETE is clearly visible in the agent output, but the orchestrator doesn't detect it and starts iteration 53.

Metrics

From .agent/metrics/metrics_20260111_124854.json:

{
  "summary": {
    "iterations": 53,
    "successful": 51,
    "failed": 2
  },
  "cost": {
    "total": 0.6246
  }
}

All 53 iterations reported the task was already complete. The first iteration saw LOOP_COMPLETE in PROMPT.md and every subsequent iteration just verified "yep, still complete" - wasting $0.62 and ~60 minutes.

Reproduction Steps

Create a task that outputs LOOP_COMPLETE when finished
Configure completion_promise: "LOOP_COMPLETE" in ralph.yml
Run ralph run
Observe that the loop continues after the agent outputs LOOP_COMPLETE

Environment

ralph-orchestrator version: 1.2.2 (installed via mise/pipx)
Agent: Claude (claude-opus-4-5-20251101)
OS: macOS Darwin 25.3.0

Suggested Fix

The completion detection logic should scan the agent's output text for the completion_promise string before deciding whether to continue. The current code path appears to mark iterations as "successful" without checking for the completion promise.

🤖 This content was generated with AI assistance using Claude Opus 4.5.

Originally created by @ramarivera on GitHub (Jan 11, 2026). Original GitHub issue: https://github.com/mikeyobrien/ralph-orchestrator/issues/47 ## Bug Description The `completion_promise` configuration option is not being detected in agent output. The orchestrator continues iterating indefinitely even when the agent clearly outputs the configured completion string. ## Expected Behavior When the agent outputs the string specified in `completion_promise` (e.g., `LOOP_COMPLETE`), the orchestrator should detect it and stop the loop. ## Actual Behavior The orchestrator: 1. Receives agent output containing the completion string 2. Logs "Iteration X completed successfully" 3. Immediately starts the next iteration 4. Repeats indefinitely until manually killed or max_iterations reached ## Configuration ```yaml completion_promise: "LOOP_COMPLETE" max_iterations: 100 agent: claude ``` ## Evidence ### Agent Output (Iteration 52) ``` ## 📊 Summary - Task Complete The orchestrated task has reached **LOOP_COMPLETE** status... No further action is needed from this orchestration loop. 1 LOOP_COMPLETE ``` ### Orchestrator Log (immediately after) ``` 2026-01-11 12:48:30,160 - ralph-orchestrator - INFO - Estimated cost: $0.0084 (total: $0.6246) ✓ Iteration 52 completed successfully 2026-01-11 12:48:32,165 - ralph-orchestrator.safety - WARNING - High iteration count: 52 === RALPH ITERATION 53 === ``` The string `LOOP_COMPLETE` is clearly visible in the agent output, but the orchestrator doesn't detect it and starts iteration 53. ## Metrics From `.agent/metrics/metrics_20260111_124854.json`: ```json { "summary": { "iterations": 53, "successful": 51, "failed": 2 }, "cost": { "total": 0.6246 } } ``` **All 53 iterations** reported the task was already complete. The first iteration saw `LOOP_COMPLETE` in PROMPT.md and every subsequent iteration just verified "yep, still complete" - wasting $0.62 and ~60 minutes. ## Reproduction Steps 1. Create a task that outputs `LOOP_COMPLETE` when finished 2. Configure `completion_promise: "LOOP_COMPLETE"` in ralph.yml 3. Run `ralph run` 4. Observe that the loop continues after the agent outputs `LOOP_COMPLETE` ## Environment - ralph-orchestrator version: 1.2.2 (installed via mise/pipx) - Agent: Claude (claude-opus-4-5-20251101) - OS: macOS Darwin 25.3.0 ## Suggested Fix The completion detection logic should scan the agent's output text for the `completion_promise` string before deciding whether to continue. The current code path appears to mark iterations as "successful" without checking for the completion promise. 🤖 *This content was generated with AI assistance using Claude Opus 4.5.*

kerem closed this issue

2026-02-27 10:21:46 +03:00

kerem commented

2026-02-27 10:21:47 +03:00

Author

Owner

@michael-wise commented on GitHub (Jan 12, 2026):

Also experiencing this on both Ubuntu and Windows w/ Claude agent

@michael-wise commented on GitHub (Jan 12, 2026): Also experiencing this on both Ubuntu and Windows w/ Claude agent

kerem commented

2026-02-27 10:21:47 +03:00

Author

Owner

@mikeyobrien commented on GitHub (Jan 12, 2026):

Fixed by https://github.com/mikeyobrien/ralph-orchestrator/pull/49

@mikeyobrien commented on GitHub (Jan 12, 2026): Fixed by https://github.com/mikeyobrien/ralph-orchestrator/pull/49

kerem commented

2026-02-27 10:21:47 +03:00

Author

Owner

@michael-wise commented on GitHub (Jan 12, 2026):

Sorry to bother @mikeyobrien, I updated via uv to 1.2.3 (ubuntu) and did a fresh ralph init. Here is my task + otherwise default PROMPT.md

# Task: Your goal is to do nothing, as this is a Ralph Orchestrator test. Mark both Requirements as complete.

## Requirements
- [ ] Requirement 1
- [ ] Requirement 2

## Success Criteria
- All requirements met
- Tests pass
- Code is clean

## Completion Promise
- When all success criteria are met, output this exact line:
  LOOP_COMPLETE

The orchestrator instantly auto closes upon ralph run (before any agents even run).

A modified Completion Promise in PROMPT.md allows normal execution of Ralph Orchestrator:

## Completion Promise
- When all success criteria are met, output this exact line, but replacing the space with an _:
  LOOP COMPLETE

@michael-wise commented on GitHub (Jan 12, 2026): Sorry to bother @mikeyobrien, I updated via uv to 1.2.3 (ubuntu) and did a fresh ralph init. Here is my task + otherwise default PROMPT.md ``` # Task: Your goal is to do nothing, as this is a Ralph Orchestrator test. Mark both Requirements as complete. ## Requirements - [ ] Requirement 1 - [ ] Requirement 2 ## Success Criteria - All requirements met - Tests pass - Code is clean ## Completion Promise - When all success criteria are met, output this exact line: LOOP_COMPLETE ``` The orchestrator instantly auto closes upon ralph run (before any agents even run). A modified Completion Promise in PROMPT.md allows normal execution of Ralph Orchestrator: ``` ## Completion Promise - When all success criteria are met, output this exact line, but replacing the space with an _: LOOP COMPLETE ```

kerem commented

2026-02-27 10:21:47 +03:00

Author

Owner

@mikeyobrien commented on GitHub (Jan 12, 2026):

Oops... looks like we need to prevent the ralph prompt command from adding those instructions.

@mikeyobrien commented on GitHub (Jan 12, 2026): Oops... looks like we need to prevent the `ralph prompt` command from adding those instructions.

kerem referenced this issue

2026-02-27 10:22:11 +03:00

[PR #14] [CLOSED] feat: Add Real-Time TUI prompt and configuration #92