starred/ralph-orchestrator

Fork 0

mirror of https://github.com/mikeyobrien/ralph-orchestrator.git synced 2026-04-25 15:15:57 +03:00

[GH-ISSUE #66] Feature/question: Per-hat `max_activations` limit #26

New issue

Closed

opened 2026-02-27 10:21:50 +03:00 by kerem · 2 comments

kerem commented

2026-02-27 10:21:50 +03:00

Owner

Originally created by @cdriscol on GitHub (Jan 17, 2026).
Original GitHub issue: https://github.com/mikeyobrien/ralph-orchestrator/issues/66

Love what you've built here. I've been running my own custom Ralph implementation and was about to build out review/change loops when I found this library. The hat-based event system is exactly what I need — just ran into one edge case I wanted to raise (and I hope this makes sense).

Problem

Feedback loops between hats can cycle indefinitely. For example, a code reviewer requests changes, the executor fixes them but introduces new issues, the reviewer requests more changes, and so on.

The global max_iterations doesn't help here — it limits total cycles but can't distinguish between productive work and a stuck review loop consuming the entire budget.

How the loop works

The executor completes implementation and publishes implementation.done. The code reviewer triggers, reviews the changes, and publishes either review.approved or review.changes_requested with specific feedback (file paths, line numbers, issues found). The executor triggers on review.changes_requested, reads the feedback from the event payload, fixes the issues, and publishes implementation.done again. This continues until the reviewer approves — or indefinitely if it never does.

Example

hats:
  executor:
    triggers: ["task.start", "review.changes_requested"]
    publishes: ["implementation.done"]

  code_reviewer:
    triggers: ["implementation.done"]
    publishes: ["review.approved", "review.changes_requested"]
    max_activations: 3  # Proposed new field

When code_reviewer reaches 3 activations, the orchestrator publishes code_reviewer.exhausted instead of activating it again. An escalator hat can trigger on this event to log unresolved issues and terminate gracefully.

Originally created by @cdriscol on GitHub (Jan 17, 2026). Original GitHub issue: https://github.com/mikeyobrien/ralph-orchestrator/issues/66 Love what you've built here. I've been running my own custom Ralph implementation and was about to build out review/change loops when I found this library. The hat-based event system is exactly what I need — just ran into one edge case I wanted to raise (and I hope this makes sense). ## Problem Feedback loops between hats can cycle indefinitely. For example, a code reviewer requests changes, the executor fixes them but introduces new issues, the reviewer requests more changes, and so on. The global `max_iterations` doesn't help here — it limits total cycles but can't distinguish between productive work and a stuck review loop consuming the entire budget. ## How the loop works The executor completes implementation and publishes `implementation.done`. The code reviewer triggers, reviews the changes, and publishes either `review.approved` or `review.changes_requested` with specific feedback (file paths, line numbers, issues found). The executor triggers on `review.changes_requested`, reads the feedback from the event payload, fixes the issues, and publishes `implementation.done` again. This continues until the reviewer approves — or indefinitely if it never does. ## Example ```yaml hats: executor: triggers: ["task.start", "review.changes_requested"] publishes: ["implementation.done"] code_reviewer: triggers: ["implementation.done"] publishes: ["review.approved", "review.changes_requested"] max_activations: 3 # Proposed new field ``` When `code_reviewer` reaches 3 activations, the orchestrator publishes `code_reviewer.exhausted` instead of activating it again. An escalator hat can trigger on this event to log unresolved issues and terminate gracefully.

kerem closed this issue

2026-02-27 10:21:50 +03:00

kerem commented

2026-02-27 10:21:50 +03:00

Author

Owner

@mikeyobrien commented on GitHub (Jan 18, 2026):

I like this idea. As a stop-gap, you could instruct Ralph to use ralph events to inspect if stuck and exit, or add to the guardrails section:

# Core behaviors (always injected into prompts)
core:
  scratchpad: ".agent/scratchpad.md"    # Shared memory across iterations
  specs_dir: "./specs/"                 # Directory for specifications
  guardrails:                           # Rules injected into every prompt
    - "Fresh context each iteration - scratchpad is memory"
    - "Don't assume 'not implemented' - search first"
    - "Backpressure is law - tests/typecheck/lint must pass"
    - "Review code a maximum of 3 times, use `ralph events` to count `implementation.done` events"

I also plan to backport the similarity circuit breaker from v1. Somewhat related, but I don't think it will solve what's described here. I have this task locally:

---
status: pending
created: 2026-01-16
started: null
completed: null
---
# Task: Implement Output Similarity Detection for Loop Prevention

## Description
Add fuzzy output similarity detection as an additional safety mechanism to prevent agents from getting stuck in loops where they produce repetitive outputs without realizing it. This complements the existing event-driven `build.blocked` detection by watching what agents actually output rather than what they explicitly signal.

## Background
The current loop detection in ralph-orchestrator-2.0 relies on explicit `build.blocked` events to detect when agents are stuck. However, agents can get into implicit loops where they repeatedly produce similar outputs without emitting blocked signals. The Python v1.2.2 implementation (PR #6) solved this using RapidFuzz for fuzzy string matching.

This task ports that concept to Rust using the `similar` crate, which provides line-based text diffing with a built-in `ratio()` method that returns a normalized similarity score (0.0 to 1.0). Line-based comparison is more semantically appropriate for multi-line agent outputs than character-by-character matching.

**Design principle**: This implements defense-in-depth — the event-driven detection catches *explicit* stuck signals, while similarity detection catches *implicit* loops where the agent doesn't realize it's repeating itself.

## Technical Requirements
1. Add the `similar` crate as a dependency to `ralph-core/Cargo.toml`
2. Create an `OutputSimilarityDetector` struct that maintains a sliding window of recent outputs
3. Use `similar::TextDiff::from_lines()` for line-based comparison with `ratio()` for scoring
4. Make window size and similarity threshold configurable via `EventLoopConfig`
5. Add a new `TerminationReason::OutputSimilarity` variant for this termination type
6. Integrate detection into the event loop after successful iteration output capture
7. Emit tracing logs when similarity is detected (warn level with percentage)
8. Ensure the detector can be reset (for session restart scenarios)

## Dependencies
- `similar` crate (latest stable version) — provides `TextDiff::from_lines().ratio()`
- Existing `EventLoopConfig` struct in `crates/ralph-core/src/config.rs`
- Existing `LoopState` struct in `crates/ralph-core/src/event_loop.rs`
- Existing `TerminationReason` enum in `crates/ralph-core/src/event_loop.rs`

## Implementation Approach

### 1. Add Dependency
Add to `crates/ralph-core/Cargo.toml`:
```toml
similar = "2"

2. Create OutputSimilarityDetector

Create a new module or add to event_loop.rs:

use similar::TextDiff;
use std::collections::VecDeque;

pub struct OutputSimilarityDetector {
    recent_outputs: VecDeque<String>,
    window_size: usize,
    threshold: f64,
}

impl OutputSimilarityDetector {
    pub fn new(window_size: usize, threshold: f64) -> Self { ... }

    /// Returns true if loop detected (output too similar to recent ones)
    pub fn detect_loop(&mut self, current_output: &str) -> bool { ... }

    pub fn reset(&mut self) { ... }
}

3. Extend EventLoopConfig

Add new fields with serde defaults:

/// Number of recent outputs to compare against (default: 5)
#[serde(default = "default_similarity_window")]
pub similarity_window_size: usize,

/// Similarity threshold for loop detection, 0.0-1.0 (default: 0.9)
#[serde(default = "default_similarity_threshold")]
pub similarity_threshold: f64,

/// Enable output similarity detection (default: true)
#[serde(default = "default_true")]
pub enable_similarity_detection: bool,

4. Extend TerminationReason

Add variant and implement methods:

pub enum TerminationReason {
    // ... existing variants ...
    /// Output too similar to recent outputs (implicit loop).
    OutputSimilarity,
}

// exit_code: 1 (failure)
// as_str: "output_similarity"

5. Integrate into Event Loop

In EventLoop:

Initialize detector from config in constructor
Call detect_loop() after successful iteration output capture
Check for OutputSimilarity termination in check_termination()

6. Add Tracing

When similarity detected:

tracing::warn!(
    similarity = %format!("{:.1}%", ratio * 100.0),
    "Loop detected: output too similar to previous iteration"
);

Acceptance Criteria

Dependency Added
- Given the ralph-core crate
- When building the project
- Then the similar crate is available and compiles successfully
Detector Detects Similar Outputs
- Given an OutputSimilarityDetector with threshold 0.9 and window size 5
- When two outputs with ≥90% line-based similarity are compared
- Then detect_loop() returns true
Detector Allows Dissimilar Outputs
- Given an OutputSimilarityDetector with threshold 0.9
- When outputs with <90% similarity are compared
- Then detect_loop() returns false and outputs are added to history
Sliding Window Behavior
- Given an OutputSimilarityDetector with window size 5
- When 6 different outputs are processed
- Then only the 5 most recent are retained for comparison
Configuration Integration
- Given a ralph.yml with similarity_threshold: 0.85 and similarity_window_size: 3
- When the event loop is initialized
- Then the detector uses those values instead of defaults
Configuration Defaults
- Given a ralph.yml without similarity settings
- When the event loop is initialized
- Then defaults are used: window_size=5, threshold=0.9, enabled=true
Disable Feature
- Given a ralph.yml with enable_similarity_detection: false
- When outputs are processed
- Then similarity detection is skipped entirely
Termination Reason
- Given an event loop where similarity detection triggers
- When the loop terminates
- Then termination reason is OutputSimilarity with exit code 1
Tracing Output
- Given similarity detection triggering
- When the detection occurs
- Then a warning log is emitted with the similarity percentage
Reset Functionality
- Given a detector with outputs in history
- When reset() is called
- Then the history is cleared
Unit Test Coverage
- Given the OutputSimilarityDetector implementation
- When running cargo test
- Then all detector behaviors have corresponding unit tests
Smoke Test Passes
- Given the implementation is complete
- When running cargo test -p ralph-core smoke_runner
- Then existing smoke tests continue to pass

Metadata

Complexity: Medium
Labels: Safety, Loop Detection, Event Loop, Configuration
Required Skills: Rust, text diffing algorithms, serde configuration, tracing

@mikeyobrien commented on GitHub (Jan 18, 2026): I like this idea. As a stop-gap, you could instruct Ralph to use `ralph events` to inspect if stuck and exit, or add to the guardrails section: ``` # Core behaviors (always injected into prompts) core: scratchpad: ".agent/scratchpad.md" # Shared memory across iterations specs_dir: "./specs/" # Directory for specifications guardrails: # Rules injected into every prompt - "Fresh context each iteration - scratchpad is memory" - "Don't assume 'not implemented' - search first" - "Backpressure is law - tests/typecheck/lint must pass" - "Review code a maximum of 3 times, use `ralph events` to count `implementation.done` events" ``` --- I also plan to backport the similarity circuit breaker from v1. Somewhat related, but I don't think it will solve what's described here. I have this task locally: ``` --- status: pending created: 2026-01-16 started: null completed: null --- # Task: Implement Output Similarity Detection for Loop Prevention ## Description Add fuzzy output similarity detection as an additional safety mechanism to prevent agents from getting stuck in loops where they produce repetitive outputs without realizing it. This complements the existing event-driven `build.blocked` detection by watching what agents actually output rather than what they explicitly signal. ## Background The current loop detection in ralph-orchestrator-2.0 relies on explicit `build.blocked` events to detect when agents are stuck. However, agents can get into implicit loops where they repeatedly produce similar outputs without emitting blocked signals. The Python v1.2.2 implementation (PR #6) solved this using RapidFuzz for fuzzy string matching. This task ports that concept to Rust using the `similar` crate, which provides line-based text diffing with a built-in `ratio()` method that returns a normalized similarity score (0.0 to 1.0). Line-based comparison is more semantically appropriate for multi-line agent outputs than character-by-character matching. **Design principle**: This implements defense-in-depth — the event-driven detection catches *explicit* stuck signals, while similarity detection catches *implicit* loops where the agent doesn't realize it's repeating itself. ## Technical Requirements 1. Add the `similar` crate as a dependency to `ralph-core/Cargo.toml` 2. Create an `OutputSimilarityDetector` struct that maintains a sliding window of recent outputs 3. Use `similar::TextDiff::from_lines()` for line-based comparison with `ratio()` for scoring 4. Make window size and similarity threshold configurable via `EventLoopConfig` 5. Add a new `TerminationReason::OutputSimilarity` variant for this termination type 6. Integrate detection into the event loop after successful iteration output capture 7. Emit tracing logs when similarity is detected (warn level with percentage) 8. Ensure the detector can be reset (for session restart scenarios) ## Dependencies - `similar` crate (latest stable version) — provides `TextDiff::from_lines().ratio()` - Existing `EventLoopConfig` struct in `crates/ralph-core/src/config.rs` - Existing `LoopState` struct in `crates/ralph-core/src/event_loop.rs` - Existing `TerminationReason` enum in `crates/ralph-core/src/event_loop.rs` ## Implementation Approach ### 1. Add Dependency Add to `crates/ralph-core/Cargo.toml`: ```toml similar = "2" ``` ### 2. Create OutputSimilarityDetector Create a new module or add to `event_loop.rs`: ```rust use similar::TextDiff; use std::collections::VecDeque; pub struct OutputSimilarityDetector { recent_outputs: VecDeque<String>, window_size: usize, threshold: f64, } impl OutputSimilarityDetector { pub fn new(window_size: usize, threshold: f64) -> Self { ... } /// Returns true if loop detected (output too similar to recent ones) pub fn detect_loop(&mut self, current_output: &str) -> bool { ... } pub fn reset(&mut self) { ... } } ``` ### 3. Extend EventLoopConfig Add new fields with serde defaults: ```rust /// Number of recent outputs to compare against (default: 5) #[serde(default = "default_similarity_window")] pub similarity_window_size: usize, /// Similarity threshold for loop detection, 0.0-1.0 (default: 0.9) #[serde(default = "default_similarity_threshold")] pub similarity_threshold: f64, /// Enable output similarity detection (default: true) #[serde(default = "default_true")] pub enable_similarity_detection: bool, ``` ### 4. Extend TerminationReason Add variant and implement methods: ```rust pub enum TerminationReason { // ... existing variants ... /// Output too similar to recent outputs (implicit loop). OutputSimilarity, } // exit_code: 1 (failure) // as_str: "output_similarity" ``` ### 5. Integrate into Event Loop In `EventLoop`: - Initialize detector from config in constructor - Call `detect_loop()` after successful iteration output capture - Check for `OutputSimilarity` termination in `check_termination()` ### 6. Add Tracing When similarity detected: ```rust tracing::warn!( similarity = %format!("{:.1}%", ratio * 100.0), "Loop detected: output too similar to previous iteration" ); ``` ## Acceptance Criteria 1. **Dependency Added** - Given the `ralph-core` crate - When building the project - Then the `similar` crate is available and compiles successfully 2. **Detector Detects Similar Outputs** - Given an OutputSimilarityDetector with threshold 0.9 and window size 5 - When two outputs with ≥90% line-based similarity are compared - Then `detect_loop()` returns true 3. **Detector Allows Dissimilar Outputs** - Given an OutputSimilarityDetector with threshold 0.9 - When outputs with <90% similarity are compared - Then `detect_loop()` returns false and outputs are added to history 4. **Sliding Window Behavior** - Given an OutputSimilarityDetector with window size 5 - When 6 different outputs are processed - Then only the 5 most recent are retained for comparison 5. **Configuration Integration** - Given a ralph.yml with `similarity_threshold: 0.85` and `similarity_window_size: 3` - When the event loop is initialized - Then the detector uses those values instead of defaults 6. **Configuration Defaults** - Given a ralph.yml without similarity settings - When the event loop is initialized - Then defaults are used: window_size=5, threshold=0.9, enabled=true 7. **Disable Feature** - Given a ralph.yml with `enable_similarity_detection: false` - When outputs are processed - Then similarity detection is skipped entirely 8. **Termination Reason** - Given an event loop where similarity detection triggers - When the loop terminates - Then termination reason is `OutputSimilarity` with exit code 1 9. **Tracing Output** - Given similarity detection triggering - When the detection occurs - Then a warning log is emitted with the similarity percentage 10. **Reset Functionality** - Given a detector with outputs in history - When `reset()` is called - Then the history is cleared 11. **Unit Test Coverage** - Given the OutputSimilarityDetector implementation - When running `cargo test` - Then all detector behaviors have corresponding unit tests 12. **Smoke Test Passes** - Given the implementation is complete - When running `cargo test -p ralph-core smoke_runner` - Then existing smoke tests continue to pass ## Metadata - **Complexity**: Medium - **Labels**: Safety, Loop Detection, Event Loop, Configuration - **Required Skills**: Rust, text diffing algorithms, serde configuration, tracing ```

kerem commented

2026-02-27 10:21:50 +03:00

Author

Owner

@cdriscol commented on GitHub (Jan 18, 2026):

Nice - I'll play with that later tonight and provide additional feedback when I do. Thanks for the quick reply and putting together a sweet package here.

@cdriscol commented on GitHub (Jan 18, 2026): Nice - I'll play with that later tonight and provide additional feedback when I do. Thanks for the quick reply and putting together a sweet package here.