[PR #392] [MERGED] Add contextuality validation and low-context quality status to daily workflow plan generation #697

New issue

Closed

opened 2026-03-13 21:04:32 +03:00 by kerem · 0 comments

kerem commented

2026-03-13 21:04:32 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/AJaySi/ALwrity/pull/392
Author: @AJaySi
Created: 3/6/2026
Status: ✅ Merged
Merged: 3/8/2026
Merged by: @AJaySi

Base: main ← Head: codex/add-plan-validation-step-before-db-persistence

📝 Commits (1)

7096f03 Add contextuality validation and low-context workflow status

📊 Changes

3 files changed (+201 additions, -6 deletions)

View changed files

📝 backend/api/today_workflow.py (+2 -0)
📝 backend/services/today_workflow_service.py (+152 -5)
📝 backend/sif_release_readiness_checks.py (+47 -1)

📄 Description

Motivation

Ensure generated daily workflow plans are explicitly grounded in user onboarding data or active alerts before being persisted.
Provide a single regeneration attempt with stricter constraints when a plan lacks contextual anchors to improve plan quality.
Surface contextuality signals to analytics and UX by persisting a plan-level quality_status and validation payload.

Description

Added MIN_TASK_EVIDENCE_LINKS and PLAN_CONTEXT_THRESHOLD and implemented validate_plan_contextuality(plan, grounding) in backend/services/today_workflow_service.py to score per-task contextual anchors and compute a plan-level contextuality score.
Added helper functions _derive_onboarding_evidence_links and _valid_evidence_links and enhanced grounding to include alert_id entries so tasks can reference concrete alert anchors.
Extended generate_agent_enhanced_plan signature to accept grounding and strict_contextuality, added strict prompt constraints when strict_contextuality=True, and prevented agent-committee short-circuiting in strict mode so regeneration enforces evidence requirements.
In get_or_create_daily_workflow_plan perform validation after generation, trigger one strict regeneration if below threshold, set plan_data["quality_status"] to either "contextual" or "low_context", attach contextuality_validation to plan_json, and persist.
Exposed quality_status and contextuality_validation in the Today Workflow API response at backend/api/today_workflow.py so product analytics and UX can consume low-context signals.
Added unit tests for contextual validation success/failure to backend/sif_release_readiness_checks.py that exercise evidence-link handling and threshold behavior.

Testing

Compiled modified modules with python -m py_compile backend/services/today_workflow_service.py backend/api/today_workflow.py backend/sif_release_readiness_checks.py which succeeded.
Ran the new contextuality unit tests with python -m unittest sif_release_readiness_checks.SIFReleaseReadinessTests.test_validate_plan_contextuality_passes_with_evidence_links sif_release_readiness_checks.SIFReleaseReadinessTests.test_validate_plan_contextuality_flags_missing_evidence_links (from backend/), and both tests passed.
Note: running the full sif_release_readiness_checks suite surfaced pre-existing unrelated test failures in the repository, but the newly added contextuality tests succeeded and the code changes compile.

Codex Task

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/AJaySi/ALwrity/pull/392 **Author:** [@AJaySi](https://github.com/AJaySi) **Created:** 3/6/2026 **Status:** ✅ Merged **Merged:** 3/8/2026 **Merged by:** [@AJaySi](https://github.com/AJaySi) **Base:** `main` ← **Head:** `codex/add-plan-validation-step-before-db-persistence` --- ### 📝 Commits (1) - [`7096f03`](https://github.com/AJaySi/ALwrity/commit/7096f036230e410dd4b174bb8678183215ef6b6c) Add contextuality validation and low-context workflow status ### 📊 Changes **3 files changed** (+201 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `backend/api/today_workflow.py` (+2 -0) 📝 `backend/services/today_workflow_service.py` (+152 -5) 📝 `backend/sif_release_readiness_checks.py` (+47 -1) </details> ### 📄 Description ### Motivation - Ensure generated daily workflow plans are explicitly grounded in user onboarding data or active alerts before being persisted. - Provide a single regeneration attempt with stricter constraints when a plan lacks contextual anchors to improve plan quality. - Surface contextuality signals to analytics and UX by persisting a plan-level `quality_status` and validation payload. ### Description - Added `MIN_TASK_EVIDENCE_LINKS` and `PLAN_CONTEXT_THRESHOLD` and implemented `validate_plan_contextuality(plan, grounding)` in `backend/services/today_workflow_service.py` to score per-task contextual anchors and compute a plan-level contextuality score. - Added helper functions `_derive_onboarding_evidence_links` and `_valid_evidence_links` and enhanced grounding to include `alert_id` entries so tasks can reference concrete alert anchors. - Extended `generate_agent_enhanced_plan` signature to accept `grounding` and `strict_contextuality`, added strict prompt constraints when `strict_contextuality=True`, and prevented agent-committee short-circuiting in strict mode so regeneration enforces evidence requirements. - In `get_or_create_daily_workflow_plan` perform validation after generation, trigger one strict regeneration if below threshold, set `plan_data["quality_status"]` to either `"contextual"` or `"low_context"`, attach `contextuality_validation` to `plan_json`, and persist. - Exposed `quality_status` and `contextuality_validation` in the Today Workflow API response at `backend/api/today_workflow.py` so product analytics and UX can consume low-context signals. - Added unit tests for contextual validation success/failure to `backend/sif_release_readiness_checks.py` that exercise evidence-link handling and threshold behavior. ### Testing - Compiled modified modules with `python -m py_compile backend/services/today_workflow_service.py backend/api/today_workflow.py backend/sif_release_readiness_checks.py` which succeeded. - Ran the new contextuality unit tests with `python -m unittest sif_release_readiness_checks.SIFReleaseReadinessTests.test_validate_plan_contextuality_passes_with_evidence_links sif_release_readiness_checks.SIFReleaseReadinessTests.test_validate_plan_contextuality_flags_missing_evidence_links` (from `backend/`), and both tests passed. - Note: running the full `sif_release_readiness_checks` suite surfaced pre-existing unrelated test failures in the repository, but the newly added contextuality tests succeeded and the code changes compile. ------ [Codex Task](https://chatgpt.com/codex/tasks/task_e_69aaf8e785ac832890ca826628f01a53) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-13 21:04:32 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ALwrity#697

No description provided.

Rows
Columns