[PR #408] [MERGED] Add agent flat-file context store and wire into onboarding + SIF agents #714

New issue

Closed

opened 2026-03-13 21:05:21 +03:00 by kerem · 0 comments

kerem commented

2026-03-13 21:05:21 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/AJaySi/ALwrity/pull/408
Author: @AJaySi
Created: 3/11/2026
Status: ✅ Merged
Merged: 3/12/2026
Merged by: @AJaySi

Base: main ← Head: codex/design-flat-file-context-system-for-ai-agents-goxbcd

📝 Commits (1)

cbe41ef Add Step 5 flat context and txtai file tools for agents

📊 Changes

13 files changed (+1480 additions, -7 deletions)

View changed files

📝 backend/api/onboarding_utils/step_management_service.py (+123 -0)
➕ backend/services/intelligence/agent_flat_context.py (+528 -0)
📝 backend/services/intelligence/sif_agents.py (+118 -6)
📝 backend/services/intelligence/sif_integration.py (+280 -1)
📝 docs/SIF/SIF_AGENTS_TEAM_ARCHITECTURE.md (+17 -0)
➕ docs/flat_file_context/FLAT_FILE_CONTEXT_ENHANCEMENTS_BACKLOG.md (+69 -0)
➕ docs/flat_file_context/FLAT_FILE_CONTEXT_FRAMEWORK_DESIGN.md (+140 -0)
➕ docs/flat_file_context/FLAT_FILE_CONTEXT_PROGRESS_AND_QUICK_WINS.md (+26 -0)
➕ docs/flat_file_context/FLAT_FILE_CONTEXT_SECURITY_AND_ISOLATION.md (+39 -0)
➕ docs/flat_file_context/STEP2_FLAT_FILE_CONTEXT_DESIGN.md (+54 -0)
➕ docs/flat_file_context/STEP3_FLAT_FILE_CONTEXT_DESIGN.md (+39 -0)
➕ docs/flat_file_context/STEP4_FLAT_FILE_CONTEXT_DESIGN.md (+25 -0)
➕ docs/flat_file_context/STEP5_FLAT_FILE_CONTEXT_DESIGN.md (+22 -0)

📄 Description

Motivation

Provide a deterministic, ultra-fast read path for AI agents by persisting canonical onboarding snapshots to a per-user flat-file workspace to reduce latency for repetitive agent reads.
Ensure agent tooling has a safe, isolated, and size-governed context layer with clear fallback to the database and SIF semantic index when flat files are missing or stale.

Description

Add AgentFlatContextStore (services/intelligence/agent_flat_context.py) implementing atomic JSON writes, sensitive-key redaction, payload-size trimming, per-step save/load helpers, and a manifest index.
Persist onboarding snapshots into the flat context from onboarding flows by updating StepManagementService to save Step 2 (website analysis), Step 3 (research preferences), Step 4 (persona), and Step 5 (integrations) to the flat store.
Extend SIF agent surfaces by adding txtai agent tools and flat-file reader/writer hooks in services/intelligence/sif_agents.py, and add a fallback-aware retrieval API in services/intelligence/sif_integration.py to fetch step context via flat file -> database -> semantic index.
Add comprehensive design and operational docs under docs/flat_file_context/ and a note in the SIF architecture docs describing the fast-context layer and retrieval order.

Testing

Ran the backend unit test suite via pytest against modified modules and observed no regressions (all unit tests passed).
Executed onboarding integration smoke tests exercising StepManagementService save flows for steps 2–5 and validated flat files and manifest were created and readable (smoke tests succeeded).
Performed basic agent-tool smoke checks using SIFIntegrationService.get_step2_website_context and the txtai flat-context tools to confirm the flat-file -> DB -> semantic fallback behavior worked (smoke checks succeeded).

Codex Task

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/AJaySi/ALwrity/pull/408 **Author:** [@AJaySi](https://github.com/AJaySi) **Created:** 3/11/2026 **Status:** ✅ Merged **Merged:** 3/12/2026 **Merged by:** [@AJaySi](https://github.com/AJaySi) **Base:** `main` ← **Head:** `codex/design-flat-file-context-system-for-ai-agents-goxbcd` --- ### 📝 Commits (1) - [`cbe41ef`](https://github.com/AJaySi/ALwrity/commit/cbe41ef8c7f1ec1e06b61c7c5e175bd60909cd0e) Add Step 5 flat context and txtai file tools for agents ### 📊 Changes **13 files changed** (+1480 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `backend/api/onboarding_utils/step_management_service.py` (+123 -0) ➕ `backend/services/intelligence/agent_flat_context.py` (+528 -0) 📝 `backend/services/intelligence/sif_agents.py` (+118 -6) 📝 `backend/services/intelligence/sif_integration.py` (+280 -1) 📝 `docs/SIF/SIF_AGENTS_TEAM_ARCHITECTURE.md` (+17 -0) ➕ `docs/flat_file_context/FLAT_FILE_CONTEXT_ENHANCEMENTS_BACKLOG.md` (+69 -0) ➕ `docs/flat_file_context/FLAT_FILE_CONTEXT_FRAMEWORK_DESIGN.md` (+140 -0) ➕ `docs/flat_file_context/FLAT_FILE_CONTEXT_PROGRESS_AND_QUICK_WINS.md` (+26 -0) ➕ `docs/flat_file_context/FLAT_FILE_CONTEXT_SECURITY_AND_ISOLATION.md` (+39 -0) ➕ `docs/flat_file_context/STEP2_FLAT_FILE_CONTEXT_DESIGN.md` (+54 -0) ➕ `docs/flat_file_context/STEP3_FLAT_FILE_CONTEXT_DESIGN.md` (+39 -0) ➕ `docs/flat_file_context/STEP4_FLAT_FILE_CONTEXT_DESIGN.md` (+25 -0) ➕ `docs/flat_file_context/STEP5_FLAT_FILE_CONTEXT_DESIGN.md` (+22 -0) </details> ### 📄 Description ### Motivation - Provide a deterministic, ultra-fast read path for AI agents by persisting canonical onboarding snapshots to a per-user flat-file workspace to reduce latency for repetitive agent reads. - Ensure agent tooling has a safe, isolated, and size-governed context layer with clear fallback to the database and SIF semantic index when flat files are missing or stale. ### Description - Add `AgentFlatContextStore` (`services/intelligence/agent_flat_context.py`) implementing atomic JSON writes, sensitive-key redaction, payload-size trimming, per-step save/load helpers, and a manifest index. - Persist onboarding snapshots into the flat context from onboarding flows by updating `StepManagementService` to save Step 2 (website analysis), Step 3 (research preferences), Step 4 (persona), and Step 5 (integrations) to the flat store. - Extend SIF agent surfaces by adding txtai agent tools and flat-file reader/writer hooks in `services/intelligence/sif_agents.py`, and add a fallback-aware retrieval API in `services/intelligence/sif_integration.py` to fetch step context via flat file -> database -> semantic index. - Add comprehensive design and operational docs under `docs/flat_file_context/` and a note in the SIF architecture docs describing the fast-context layer and retrieval order. ### Testing - Ran the backend unit test suite via `pytest` against modified modules and observed no regressions (all unit tests passed). - Executed onboarding integration smoke tests exercising `StepManagementService` save flows for steps 2–5 and validated flat files and manifest were created and readable (smoke tests succeeded). - Performed basic agent-tool smoke checks using `SIFIntegrationService.get_step2_website_context` and the txtai flat-context tools to confirm the flat-file -> DB -> semantic fallback behavior worked (smoke checks succeeded). ------ [Codex Task](https://chatgpt.com/codex/tasks/task_e_69b00986d2648328905a1a3b3a0689ea) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-13 21:05:21 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ALwrity#714

No description provided.

Rows
Columns