diff --git a/benchmark/gdpval/README.md b/benchmark/gdpval/README.md index e8906a44c..25ab12b59 100644 --- a/benchmark/gdpval/README.md +++ b/benchmark/gdpval/README.md @@ -1,4 +1,26 @@ -# GDPVal Comparison Benchmark +# /benchmark/gdpval +Description: GDPVal cold/warm/Reflexio comparison across host-agent adapters. + +## Main Entry Points + + +- **`run_benchmark.py`** — phase orchestration CLI +- **`config.py`** — paths and run settings +- **`adapters/`** — OpenSpace and Hermes execution +- **`memory/reflexio_bridge.py`** — publish/retrieve bridge +- **`memory/injection.py`** — learning injection +- **`evaluation.py`** — shared artifact evaluator +- **`report.py`** — per-task and aggregate comparison + +## Purpose + + +Separate a host's native learning from Reflexio's marginal effect using matching warm-state snapshots. + +## Architecture Pattern + + +P1 produces the shared host snapshot; P2 and P3 fork it, and P3 additionally retrieves Reflexio learning. Both arms use the same evaluator and task set. Run the GDPVal dataset through two host agents (OpenSpace, Hermes) in a three-phase cold → warm → warm+reflexio protocol to measure: @@ -11,6 +33,7 @@ total (2 hosts × 3 phases). The headline we care about is `mean(P2 − P3)`. ## Prerequisites + 1. Clone the dependency repos. By default `config.py` looks under `~/repos/`: ```bash mkdir -p ~/repos @@ -46,6 +69,7 @@ total (2 hosts × 3 phases). The headline we care about is `mean(P2 − P3)`. ## Running + ```bash uv run python -m benchmark.gdpval.run_benchmark \ --hosts openspace,hermes \ @@ -66,6 +90,7 @@ Verification ladder: ## Output + ``` output// config.json @@ -87,7 +112,8 @@ output// comparison.md # headline deltas ``` -## Design notes +## Requirements / Problems to Avoid + - **Host isolation.** OpenSpace writes SkillStore state to `$OPENSPACE_ROOT/.openspace/`; Hermes writes MEMORY.md/skills to `$HERMES_HOME` (default `~/.hermes`). The diff --git a/docs/README.md b/docs/README.md index 39098899f..4e962dde2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,17 +1,35 @@ -# Reflexio API Documentation Site +# /docs +Description: Interactive Next.js API explorer for the OSS server and Python SDK. -Interactive API reference and documentation for the Reflexio platform. Built with Next.js. +## Main Entry Points -**Hosted docs:** https://www.reflexio.ai/docs -## What's Inside +- **Method pages**: `app/[group]/[method]/page.tsx` — grouped API/SDK method reference. +- **Registry**: `lib/methods/registry.ts`, `lib/methods/` — method definitions and examples by domain. +- **Execution**: `lib/execution/api-executor.ts`, `code-generator.ts`, `code-parser.ts` — request execution and example conversion (all under `lib/execution/`). +- **Configuration UI**: `app/configure/page.tsx`, `lib/config-schema.ts`. +- **Shared rendering**: `components/`, `app/layout.tsx`, `app/providers.tsx`. +- **Backend routing**: `next.config.ts`, `lib/constants.ts`. -- API reference for all REST endpoints (interactions, profiles, playbooks, config, search) -- Interactive API explorer for testing requests against a running server -- Schema documentation generated from the Reflexio backend +## Purpose + + +Help developers inspect method contracts and run examples against a configured Reflexio backend. The public authored documentation is hosted at [Reflexio docs](https://www.reflexio.ai/docs). + +## Architecture Pattern + + +App Router pages render the method registry; execution helpers translate examples into backend requests. Keep registry definitions, generated code, and the shared SDK/API schemas aligned when changing a method. + +## Requirements / Problems to Avoid + + +- **API execution uses the configured server**; examples can mutate connected data. +- **Use service-start output for ports** when running the full stack; standalone Next.js development has a different default. ## Development Setup + ```bash cd docs npm install @@ -22,6 +40,7 @@ The site runs on **port 3000** by default. When started via `run_services.sh` fr ## Build + ```bash npm run build ``` diff --git a/reflexio/README.md b/reflexio/README.md index 3adef0e9e..019b70af7 100644 --- a/reflexio/README.md +++ b/reflexio/README.md @@ -1,215 +1,76 @@ -# Reflexio Code Map -Describe the code structure and component dependencies for source code of reflexio - -## Table of Contents - -- [Overview](#overview) -- [models and client](#models-and-client) -- [cli](#cli) -- [reflexio_lib](#reflexio_lib) -- [mem0](#mem0) -- [server](#server) -- [data](#data) -- [See Also](#see-also) - -## Overview -Reflexio is a user profiling and agent playbook system with three main access patterns: - -1. **Remote API Access** (`client`) - Applications use Python SDK to call REST API -2. **Local Library Access** (`reflexio_lib`) - Direct synchronous access without HTTP layer -3. **CLI Access** (`cli`) - Local command-line workflows for services, publishing, search, auth, config, and diagnostics - -**Core Flow**: User Interactions → Server Processing → Profile/Playbook/Evaluation → Storage +# /reflexio +Description: Shared Python package for learning user profiles and agent playbooks from interactions, with SDK, CLI, local-library, and HTTP entry points. + +Paths below are relative to this package. The repository's [public README](../README.md) covers installation and examples. + +## Main Entry Points + + +| Path | When to modify it | Component map | +|------|-------------------|---------------| +| `__init__.py` | Public Python exports | | +| `client/client.py` | Typed HTTP SDK, Bearer authentication, sync calls and explicit async methods | [SDK reference](../client_dist/README.md) | +| `models/api_schema/` | Shared API and storage-facing contracts; domain entities live in `models/api_schema/domain/` | | +| `models/config_schema.py` | Shared `Config` and extractor/storage configuration | | +| `models/profile_id.py` | Canonical UUIDv4 profile identities | | +| `lib/reflexio_lib.py` | `Reflexio` local-library facade; focused `_*.py` mixins own domain operations | | +| `lib/generation_client.py` | Generation client protocol used by callers | | +| `cli/app.py`, `cli/commands/` | Typer command registration and handlers | [CLI](cli/README.md) | +| `server/api.py`, `server/routes/` | FastAPI composition and domain routes | [Server](server/README.md) | +| `server/api_endpoints/` | Request context, shared handlers, and clarification routes | [Endpoint helpers](server/api_endpoints/README.md) | +| `server/services/` | Extraction, aggregation, evaluation, retrieval, and storage | [Services](server/services/README.md) | +| `server/extensions.py` | Optional capability, hook, and typed runtime-service registration | | +| `server/prompt/prompt_bank/` | Versioned LLM prompts | [Prompts](server/prompt/prompt_bank/README.md) | +| `server/site_var/` | Model settings, retrieval defaults, and feature flags | [Site variables](server/site_var/README.md) | +| `mem0/` | Optional hosted mem0 wrappers and scoped cleanup | [mem0](mem0/README.md) | +| `integrations/` | External agent integrations | [OpenClaw](integrations/openclaw/README.md), [embedded OpenClaw](integrations/openclaw-embedded/README.md) | +| `benchmarks/retrieval_latency/` | Storage/library retrieval timing | [Benchmark](benchmarks/retrieval_latency/README.md) | +| `test_support/` | Shared test fixtures and helpers | | + +## Purpose + + +1. **Remote access** — `ReflexioClient` sends typed API requests; the CLI reuses it for publishing, search, and configuration. +2. **Local access** — `Reflexio` runs services directly without HTTP; LLM/provider calls can still use the network. +3. **Learning and retrieval** — Shared services produce profiles and playbooks, evaluate sessions, and retrieve relevant learning. +4. **Deployment reuse** — Optional capabilities extend the same server without importing enterprise implementation code. + +## Architecture Pattern + + +```text +CLI / external application -> client/client.py -> server/routes/ +Local application ----------------------------> lib/reflexio_lib.py +server/routes/ -> RequestContext + get_reflexio() -> lib/reflexio_lib.py + -> services/generation_service.py + -> durable_learning/ (admission -> user lease -> frozen windows -> fenced commit) + -> profile/ and playbook/ -> BaseStorage + -> agent_success_evaluation/ (deferred session evaluation) + -> unified_search_service.py -> pre_retrieval/ + storage/ + retrieval/ +``` -**Shared Components**: -- `models` - API and internal schemas shared by client, CLI, and server -- `server` - FastAPI backend with LLM-based processing services -- `data` - Bundled configs and local fixtures -- `docs` - Next.js API documentation site -- `mem0` - Optional mem0 hosted-client compatibility wrapper that mirrors learning into Reflexio - -## models and client -Description: Shared data contracts and the Python SDK used by external applications, the CLI, and server endpoint helpers - -### models -**Path**: `models/` - -#### Main Entry Points -- **API Schemas**: `models/api_schema/` - Pydantic request/response models for public API surfaces -- **Internal Schemas**: `models/api_schema/internal_schema.py` - Storage-facing profile, playbook, request, evaluation, and agent-run models -- **Profile IDs**: `models/profile_id.py` - Canonical UUIDv4 generation for Reflexio-created profiles -- **Validators**: `models/api_schema/validators.py` - Cross-schema validation helpers - -#### Purpose -Provides type-safe data contracts between client and server: -1. **Service Schemas** - Interactions, requests, profiles, user playbooks, agent playbooks, evaluations, and stall-state records -2. **Retriever Schemas** - Search/get/set requests and responses -3. **Login/Auth Schemas** - Credentials, API tokens, feature flags, and organization/account responses -4. **Config Schema** - YAML/API configuration structure (`tool_can_use` at root `Config` level, shared across services) - -### client -**Path**: `client/` - -Description: Python SDK for interacting with Reflexio API remotely - -#### Main Entry Point -- **Client**: `client.py` - `ReflexioClient` class - -#### Purpose -Remote API client for applications to: -1. **Publish interactions** - Send user interactions to server for processing -2. **Search/retrieve data** - Query profiles, interactions, playbooks, evaluations, and context -3. **Track deferred learning** - Poll `get_learning_status(request_id)` after `publish_interaction(..., wait_for_response=False)` queues extraction -4. **Manage profiles/playbooks** - Delete, regenerate, and update status where supported by API endpoints -5. **Configure** - Set/get organization configuration - -#### Architecture Pattern -Async HTTP client wrapping typed models from `models/api_schema/`. Automatically handles authentication via Bearer tokens. - -## cli -Description: Command-line entry point for operating Reflexio locally and against a running server - -### Main Entry Points -- **CLI app**: `cli/` - Typer command groups for services, publish/search/context, auth, config, status, and diagnostics -- **Reference**: `cli/README.md` - Command map and common workflows - -### Purpose -Local operator interface to: -1. **Run services** - Start/stop backend, docs, and optional embedding service -2. **Publish interactions** - Send JSON, JSONL, stdin, or quick single-turn payloads -3. **Search context** - Query profiles, user playbooks, and agent playbooks -4. **Inspect/manage data** - List/delete/regenerate profiles and playbooks -5. **Configure/authenticate** - Manage API keys, server URL, and configuration - -### Architecture Pattern -Thin Typer layer over the Python client and local service manager. Use `uv run reflexio --help` to inspect command groups. - -## reflexio_lib -Description: Local Python library interface for direct (non-API) access to Reflexio functionality - -### Main Entry Point -- **Library**: `reflexio_lib.py` - `Reflexio` class - -### Purpose -Direct programmatic access without HTTP/API layer: -1. **Same interface as client** - Mirror of `ReflexioClient` but synchronous -2. **Local execution** - Runs services directly (no network calls) -3. **Testing/debugging** - Useful for local development and testing - -### Architecture Pattern -Creates `RequestContext` and directly calls `GenerationService` - bypasses FastAPI layer. Methods are **synchronous** unlike `ReflexioClient`. - -## mem0 -Description: Optional compatibility layer for hosted mem0 clients that preserves mem0 behavior while adding Reflexio learning. - -**Detailed Documentation**: See [`reflexio/mem0/README.md`](mem0/README.md) for wrapper internals, identity scoping, and failure-mode contracts. - -### Main Entry Points -- **Public exports**: `mem0/__init__.py` - `MemoryClient`, `AsyncMemoryClient`, local `Memory`/`AsyncMemory` re-exports, and Reflexio helper classes. -- **Hosted wrappers**: `mem0/_wrapper.py` - Sync/async mem0 client subclasses that mirror `add()` calls and optionally enrich `search()`. -- **Lifecycle facade**: `mem0/_facade.py` - Explicit `client.reflexio` cleanup/delete operations scoped to mem0 identities. - -### Purpose -1. **One-import migration** - mem0 users can install `reflexio-ai[mem0]` and switch imports to `reflexio.mem0`. -2. **Best-effort learning mirror** - mem0 writes remain primary; Reflexio publish/search side effects are optional and fail-open. -3. **Scoped operations** - `user_id`, `app_id`, `agent_id`, and `run_id` are mapped into deterministic Reflexio user/session scopes. +- **Generation actors** load configuration, run extractors/evaluators, and persist results through `BaseStorage`. +- **Automatic extraction** uses durable streams with independent project/kind cursors under one user lease. Legacy `learning_jobs` remains only for reconciliation. +- **Paused extraction** uses `services/extraction/` to persist human clarification and resume/finalize idempotently. +- **Aggregation** is fenced per agent version; agent-playbook successors retain the approval workflow. +- **Storage selection** belongs to the configurator. OSS defaults to SQLite; deployment extensions supply other factories. +- **Runtime data** is outside this package (normally under `~/.reflexio`, resolved through `defaults.py` and `cli/paths.py`); there is no checked-in `reflexio/data/` directory. + +## Key Endpoints / Commands / Contracts + + +- **HTTP routes**: the [server route map](server/README.md#api-endpoints) groups the complete publish, search, lifecycle, evaluation, experiment, and clarification surface. +- **CLI commands**: the [CLI command map](cli/README.md) covers services, publish, search/context, data management, config, auth, and diagnostics. +- **Deferred learning**: `publish_interaction(..., wait_for_response=False)` returns after durable admission; `get_learning_status(request_id)` reports progress. +- **Shared schemas**: client, CLI, and server use `models/api_schema/`; enterprise-only schemas belong in the consuming extension. +- **Documentation and tests**: [interactive docs](../docs/README.md), [notebooks](../notebooks/README.md), and `../tests/` sit at repository level. + +## Requirements / Problems to Avoid -### Architecture Pattern -Pass-through wrapper around mem0-hosted clients. Reflexio is configured via environment, inline kwargs, or injected `ReflexioClient`; absent Reflexio configuration leaves mem0 behavior unchanged. `search()` is mem0-only unless `include_reflexio=True`, which reserves a `reflexio` namespace on returned results. - -## server -Description: FastAPI backend server that processes user interactions to generate profiles, extract playbooks, and evaluate agent success - -**Detailed Documentation**: See [`reflexio/server/README.md`](server/README.md) for component details, including the [Prompt Bank](server/prompt/prompt_bank/README.md), [Playbook Service](server/services/playbook/README.md), and [Site Variables](server/site_var/README.md) - -### Main Entry Points -- **API composer**: `api.py` - `create_app()` factory, middleware/capability wiring, and `core_router` aggregation -- **Domain routes**: `server/routes/` - FastAPI route modules grouped by system, interactions, profiles, playbooks, search, provenance, evaluation, Braintrust, and config -- **Extension Registry**: `extensions.py` - optional capability/service registration for OSS and enterprise integrations -- **Endpoint Helpers**: `api_endpoints/` - Shared handler/helper functions and `RequestContext` used by route modules -- **Core Service**: `services/generation_service.py` - Main orchestrator - -### Purpose -Receives user interactions from clients and processes them to: -1. **Generate user profiles** - Extract and maintain user preferences/traits from behavior -2. **Extract playbooks** - Identify issues and improvement opportunities for developers -3. **Evaluate agent success** - Determine if agent successfully fulfilled user's needs - -### Component Relationships -``` -client (Python SDK) - -> api.py (create_app + core_router) - -> routes/ (domain FastAPI routers) - -> api_endpoints/ (shared handlers + RequestContext) - -> reflexio_lib.Reflexio (main entry) - -> services/generation_service.py (orchestrator) - ├─> services/profile/ -> storage (BaseStorage) - ├─> services/playbook/ (playbook extraction) -> storage (BaseStorage) - ├─> services/durable_learning/ -> learning_jobs queue -> deferred extraction - └─> services/agent_success_evaluation/ -> storage (BaseStorage) -``` -### Key Components -- **`api_endpoints/`**: Request handling, `RequestContext` (bundles storage/config/prompts), auth -- **`routes/`**: Domain route modules (`system.py`, `interactions.py`, `profiles.py`, `playbooks.py`, `search.py`, `provenance.py`, `evaluation.py`, `braintrust.py`, `config.py`) included into `api.py`'s `core_router` -- **`db/`**: Auth & config storage only (SQLite) - NOT for profiles/interactions -- **`llm/`**: Unified LiteLLM client, provider adapters, local rerank helpers, structured-output repair, and fail-open per-provider concurrency caps -- **`prompt/`**: Versioned prompt templates in `prompt_bank/` -- **`services/`**: Core business logic - - `generation_service.py` - Orchestrator (runs profile/playbook/success services) - - `base_generation_service.py` + `base_generation/` - Abstract base plus mixins for parallel actor execution, batch progress, should-run prechecks, status transitions, and usage billing - - `profile/` - Profile extraction & updates - - `playbook/` - Playbook extraction, consolidation, and aggregation - - `agent_success_evaluation/` - Success evaluation - - `durable_learning/` - Claim/drain durable `learning_jobs` for deferred extraction when `REFLEXIO_DURABLE_LEARNING_QUEUE` is enabled - - `extraction/` - Resumable async extraction agent infrastructure - - `shadow_comparison/` - Per-turn regular vs shadow verdict judge - - `evaluation_overview/` - Evaluation-page aggregates and hero metrics - - `playbook_optimizer/` - Scenario-based playbook optimization experiments - - `braintrust/` - Braintrust eval export/sync support - - `lineage/` - Resolve current records and schedule tombstone garbage collection for superseded profile/playbook rows - - `governance/` - Subject-reference contracts and retention/barrier helpers used by storage and lineage - - `storage/` - Abstract layer (SQLite prod, LocalJSON test) with governance-aware write validation and durable `learning_jobs` contracts - - `pre_retrieval/` - Query rewriting and document expansion helpers - - `configurator/` - YAML config loader -- **`billing_meter.py`**: OSS usage-event facade for learning/search metering; keep imports function-local at call sites so enterprise emitters remain optional -- **`site_var/`**: Global settings singleton - -### Architecture Patterns - -**Service Pattern** (BaseGenerationService): -1. Load configs from YAML -> 2. Create actors from configs -> 3. Run actors in parallel (ThreadPoolExecutor) -> 4. Save results to storage - -**Actor Pattern**: Multiple actors (extractors/evaluators) run in parallel, each processing interactions independently, results aggregated - -**Storage Abstraction**: All access via `BaseStorage` interface, implementation selected by configurator, supports vector similarity search - -**Data Flow**: `User Interaction -> Storage (save) -> Services (parallel: LLM + Prompts) -> Results -> Storage (save)` - - -## data -Description: Local storage directory for configuration files and SQLite databases - -### Main Entry Points -- **Configs**: `configs/` - YAML configuration files for extractors and evaluators -- **Database**: `sql_app.db` - SQLite database for auth and config storage -- **JSON Storage**: `user_profiles_*.json` - Local JSON files for testing - -### Purpose -Local data storage for: -1. **Configuration files** - YAML configs defining extraction/evaluation behavior -2. **Authentication database** - User credentials and API tokens (SQLite/Postgres) -3. **Test data** - LocalJsonStorage files for development/testing - -### Architecture Pattern -Referenced by `SimpleConfigurator` for loading configs and by database operations for auth/config persistence. Not directly accessed by application code. - -## See Also - -- [Server README](server/README.md) -- detailed component documentation for the FastAPI backend -- [Prompt Bank README](server/prompt/prompt_bank/README.md) -- versioned prompt template system -- [Playbook Service README](server/services/playbook/README.md) -- playbook extraction, aggregation, and deduplication pipeline -- [Site Variables README](server/site_var/README.md) -- global configuration and feature flags -- [Retrieval Latency Benchmarks](benchmarks/retrieval_latency/README.md) -- search performance benchmarking -- [OpenClaw Integration](integrations/openclaw/README.md) -- federated OpenClaw plugin setup and behavior -- [mem0 Wrapper](mem0/README.md) -- hosted mem0 compatibility wrapper and Reflexio mirroring contracts +- **API handlers use `get_reflexio()`**, never fresh `Reflexio()` instances. +- **Use `request_context.storage`**, not concrete storage imports in business logic. +- **Use `LiteLLMClient` and `prompt_manager.render_prompt()`** for model calls and prompts. +- **Keep `tool_can_use` at root `Config` level**; extraction and success evaluation share it. +- **Preserve governance, lineage, and fencing contracts** when changing persistence; see the [service requirements](server/services/README.md#requirements--problems-to-avoid). +- **Keep OSS independent**; register optional providers through `server/extensions.py`. diff --git a/reflexio/benchmarks/retrieval_latency/README.md b/reflexio/benchmarks/retrieval_latency/README.md index 9f38cbd17..6f56f80b1 100644 --- a/reflexio/benchmarks/retrieval_latency/README.md +++ b/reflexio/benchmarks/retrieval_latency/README.md @@ -1,4 +1,25 @@ -# Retrieval Latency Benchmark +# /reflexio/benchmarks/retrieval_latency +Description: Retrieval timing across corpus sizes, storage backends, and in-process service/ASGI layers. + +## Main Entry Points + + +- **`bench.py`** — CLI and timing loop +- **`backends.py`** — storage setup +- **`seed.py`** — corpus generation +- **`scenarios.py`** — query scenarios +- **`embed_cache.py`** — query-vector cache +- **`report.py`** — statistics and baseline comparison + +## Purpose + + +Measure retrieval cost while separating storage/library work from framework overhead. + +## Architecture Pattern + + +Seed deterministic document vectors, cache real query embeddings, warm each cell, and record repeated timings for matched backend/layer/entity combinations. > Part of the [Reflexio Code Map](../../README.md). Related to the server's [Unified Search Service](../../server/README.md#unified-search-service) and [Storage](../../server/README.md#storage) components. @@ -7,6 +28,7 @@ and unified cross-entity search across storage backends and corpus sizes. ## What it measures + Four retrieval types × two layers × N storage backends × K corpus sizes. **Layers:** @@ -29,6 +51,7 @@ gracefully otherwise). ## Controlling embedder cost + Query embeddings are pre-cached on disk at `~/.cache/reflexio-benchmarks/embeddings-.json`. First run populates the cache via the real embedding API (requires `OPENAI_API_KEY` or the @@ -45,6 +68,7 @@ retrieval. ## Usage + ```bash # Default sweep: sizes 100/1000/10000, sqlite + supabase if available, # both layers, all four retrieval types, 50 trials + 5 warmup per cell. @@ -62,6 +86,7 @@ uv run python -m reflexio.benchmarks.retrieval_latency.bench \ ## Output + Each run writes `results.json` and `report.md` to `reflexio/benchmarks/retrieval_latency/results//` — next to the script itself, so reports travel with the code that produced them (override @@ -74,7 +99,11 @@ one per retrieval type, rows grouped by `(backend, layer)`. Cell format: `p50 / p95 (mean)` in milliseconds. When `--baseline` is passed, a ΔP95 column flags cells where p95 has grown by 20% or more (`⚠`). -## Interpreting the numbers +## Requirements / Problems to Avoid + + +### Interpreting the numbers + Sanity checks to run on any report: @@ -87,6 +116,7 @@ Sanity checks to run on any report: ## Pytest smoke test + `tests/benchmarks/test_retrieval_latency_smoke.py` runs a tiny version of this benchmark at `N=50, trials=10, sqlite + service only` and asserts that p95 has not regressed past 3× the committed baseline. It's marked @@ -105,5 +135,6 @@ Then commit the new `baseline.json`. ## See Also + - [Code Map (root README)](../../README.md) -- high-level overview of all Reflexio components - [Server README](../../server/README.md) -- backend architecture including search and storage diff --git a/reflexio/cli/README.md b/reflexio/cli/README.md index eb420b8b8..edd64a7e9 100644 --- a/reflexio/cli/README.md +++ b/reflexio/cli/README.md @@ -1,9 +1,33 @@ -# Reflexio CLI +# /reflexio/cli +Description: Typer command layer for local service management and typed calls to a Reflexio server. -`reflexio` is a first-class CLI for running the Reflexio service, publishing interactions, exploring extracted profiles and playbooks, and wiring results into your own agent. Every command is also runnable as `python -m reflexio ...`. +`reflexio` is a first-class CLI for running the Reflexio service, publishing interactions, exploring extracted profiles and playbooks, and wiring results into your own agent. Commands are also runnable as `uv run python -m reflexio.cli ...`. + +## Main Entry Points + + +- **`app.py`** — root app, aliases, global flags, and command-group registration +- **`commands/`** — domain command handlers +- **`_client.py`** — configured SDK client construction +- **`run_services.py`** — backend/docs/inference startup +- **`stop_services.py`** — shutdown +- **`state.py`** — persisted CLI settings +- **`output.py`** — human and JSON output +- **`paths.py`** — runtime paths + +## Purpose + + +Expose the SDK and local service lifecycle through consistent commands and output envelopes. + +## Architecture Pattern + + +Global options and persisted state configure a shared client; domain commands validate input and call the SDK. Service commands manage local processes. Enterprise reuses the CLI and overrides the services group. ## Table of Contents + - [Install & invoke](#install--invoke) - [Global flags](#global-flags) - [Quick Reference](#quick-reference) @@ -23,6 +47,7 @@ ## Quick Reference + Most common commands at a glance: | Task | Command | @@ -37,6 +62,7 @@ Most common commands at a glance: ## Install & invoke + The CLI ships with the `reflexio` package. After `uv sync`: ```shell @@ -46,6 +72,7 @@ uv run reflexio --version ## Global flags + Available on every command (set on the root, before the subcommand): | Flag | Env var | Purpose | @@ -59,11 +86,12 @@ Example: `uv run reflexio --json search "refund policy"`. ## Services + Start and stop the backend, docs server, and colocated inference service. ```shell uv run reflexio services start # backend :8061, docs :8062 -uv run reflexio services start --storage sqlite # sqlite (default) | supabase | postgres +uv run reflexio services start --storage sqlite # OSS default; remote storage factories require a deployment extension uv run reflexio services start --backend-port 9000 --docs-port 9001 uv run reflexio services start --only backend --no-reload uv run reflexio embeddings serve --port 8072 # OpenAI-compatible local embeddings @@ -86,12 +114,14 @@ failures never fall back to in-process inference. ## Publishing interactions + The top-level `publish` shortcut is the fastest way to get a conversation into Reflexio. It forwards to `interactions publish` and supports three input modes: single-turn flags, inline JSON, and file/stdin payloads. Reflexio learns most from interactions that contain a **signal** — a user correction, a stated preference, or an explicit choice between alternatives. Examples throughout this doc use that kind of content rather than trivial Q&A. ### Mode 1 — Single-turn (shortcut flags) + `--user-message` / `--agent-response` wrap a single user turn and a single assistant turn. Use this for quick preference captures or smoke-tests. It is **hard-coded to exactly 2 turns** — use Mode 2 or 3 below for anything longer. ```shell @@ -105,6 +135,7 @@ uv run reflexio publish \ ### Mode 2 — Multi-turn via inline JSON (`--data`) + For real dialogues (3+ turns, tool calls, corrections mid-conversation), pass a JSON object whose `interactions` field is a list of `{role, content}` items. Each item becomes one turn, in order. Roles are `user` and `assistant`; `system` and `tool` turns are also accepted. ```shell @@ -126,6 +157,7 @@ You can also load inline JSON from a file with `@`: `--data @conversation.json`. ### Mode 3 — JSON / JSONL file or stdin + For bulk publishing or conversations you already have on disk, point at a file. A `.json` file should be a single object (or a list of objects); a `.jsonl` file is one JSON object per line, one conversation per line. ```shell @@ -137,6 +169,7 @@ Each payload object accepts the same fields as Mode 2 (`interactions`, `session_ ### Common flags + Apply to all three modes: | Flag | Purpose | @@ -153,6 +186,7 @@ Full options via `uv run reflexio interactions publish --help`. ## Search & context + Unified semantic search across profiles and playbooks: ```shell @@ -169,6 +203,7 @@ uv run reflexio context --user-id alice --agent-version v1 --query "deploy workf ## Interactions + ```shell uv run reflexio interactions list --user-id alice uv run reflexio interactions search "deployment" @@ -178,6 +213,7 @@ uv run reflexio interactions delete-all ## User profiles + ```shell uv run reflexio user-profiles list --user-id alice uv run reflexio user-profiles search "preferences" @@ -189,6 +225,7 @@ uv run reflexio user-profiles delete-all ## Agent playbooks + ```shell uv run reflexio agent-playbooks list --agent-version v1 uv run reflexio agent-playbooks search "error handling" @@ -200,6 +237,7 @@ uv run reflexio agent-playbooks delete ## User playbooks + ```shell uv run reflexio user-playbooks list --user-id alice uv run reflexio user-playbooks search "preferences" --user-id alice @@ -210,6 +248,7 @@ uv run reflexio user-playbooks delete ## Config + ```shell uv run reflexio config show uv run reflexio config set --data '{"api_key_config": {"openai": "sk-..."}}' @@ -220,6 +259,7 @@ uv run reflexio config pull # pull server config to ## Auth + ```shell uv run reflexio auth login --api-key $REFLEXIO_API_KEY --server-url http://localhost:8061 uv run reflexio auth status @@ -228,6 +268,7 @@ uv run reflexio auth logout ## Diagnostics + ```shell uv run reflexio status check # server health uv run reflexio status whoami # resolved identity @@ -237,6 +278,7 @@ uv run reflexio setup init # interactive setup wiz ## Raw API access + Escape hatch for calling any endpoint directly. Supports `GET`, `POST`, `DELETE`: ```shell @@ -247,18 +289,21 @@ uv run reflexio api POST /api/set_config --data @config.json ## Common Workflows + A typical end-to-end workflow: publish a conversation, verify that Reflexio extracted the right data, then browse the results. ### 1. Start services + ```shell uv run reflexio services start ``` ### 2. Publish a conversation + ```shell -uv run reflexio publish --user-id alice --wait --data '{ +uv run reflexio publish --user-id alice --session-id dark-mode-demo --wait --data '{ "interactions": [ {"role": "user", "content": "Always use dark mode in the dashboard."}, {"role": "assistant", "content": "Noted — I will default to dark mode for you."} @@ -268,6 +313,7 @@ uv run reflexio publish --user-id alice --wait --data '{ ### 3. Search to verify extraction + ```shell uv run reflexio search "dark mode" ``` @@ -276,14 +322,24 @@ You should see a profile entry reflecting the user's preference. ### 4. Browse profiles and playbooks + ```shell uv run reflexio user-profiles list --user-id alice uv run reflexio user-playbooks list --user-id alice uv run reflexio agent-playbooks list ``` +## Requirements / Problems to Avoid + + +- **Keep `session_id` non-empty for publishes**; the server groups learning/evaluation by session. +- **Keep business logic in shared services**; command handlers delegate through the SDK. +- **Do not write `.env` for port overrides**; use flags or exported environment variables. +- **Treat `--wait` as response waiting**, not client-side asynchronous execution; durable admission and extraction happen on the server. + ## Getting help + Every command and subcommand supports `--help`: ```shell diff --git a/reflexio/integrations/openclaw-embedded/plugin/prompts/README.md b/reflexio/integrations/openclaw-embedded/plugin/prompts/README.md index 0a17a230c..a81afa39b 100644 --- a/reflexio/integrations/openclaw-embedded/plugin/prompts/README.md +++ b/reflexio/integrations/openclaw-embedded/plugin/prompts/README.md @@ -1,16 +1,30 @@ -# Openclaw-Embedded Prompts +# /reflexio/integrations/openclaw-embedded/plugin/prompts +Description: LLM prompt templates used by Flow C sub-agents and consolidation. -LLM prompt templates used by Flow C sub-agents and consolidation. +## Main Entry Points -## Files - `profile_extraction.md` — extract durable user facts from a transcript - `playbook_extraction.md` — extract procedural rules from correction+confirmation patterns - `full_consolidation.md` — consolidate a cluster of similar items into individual facts -## Format +## Purpose -Each file is a `.prompt.md` with YAML frontmatter (matches Reflexio's + +Supply extraction and consolidation instructions to the embedded plugin without a Reflexio server dependency. + +## Architecture Pattern + + +The plugin ships one active version of each prompt as a flat Markdown asset, with upstream-style YAML metadata and variable substitution. + +## Key Endpoints / Commands / Contracts + + +### Format + + +Each file is a `.md` asset with YAML frontmatter (matches Reflexio's `server/prompt/prompt_bank/` convention). Unlike upstream's versioned layout (`/v.prompt.md`), we store prompts flat (`.md`) since the plugin ships atomically with one active version at a time. @@ -28,9 +42,13 @@ variables: prompt body, with {var1} and {var2} substitution points ``` -## Upstream sync +## Requirements / Problems to Avoid + + +### Upstream sync + `profile_extraction.md` and `playbook_extraction.md` are ports of Reflexio's prompt_bank entries. On upstream bumps, review the prompt diff against our -adapted versions. Porting notes will be maintained in -`../references/porting-notes.md` (added in a later phase). +adapted versions. The corresponding upstream templates are under +`../../../../server/prompt/prompt_bank/`; preserve plugin-specific variables when syncing. diff --git a/reflexio/integrations/openclaw/README.md b/reflexio/integrations/openclaw/README.md index 1a1a93f33..b4d83c06b 100644 --- a/reflexio/integrations/openclaw/README.md +++ b/reflexio/integrations/openclaw/README.md @@ -1,5 +1,6 @@ # openclaw-smart + openclaw-smart is the openClaw plugin counterpart of [claude-smart](https://github.com/reflexio-ai/claude-smart): a thin TS shim plus a Python (`openclaw_smart`) package that wires openClaw's plugin hooks @@ -14,6 +15,7 @@ silently when the backend is unreachable. ## Quick install + Guided Reflexio setup: ```bash @@ -56,6 +58,7 @@ and `openclaw-smart repair`. ## How it works + * **session_start** — pushes openclaw-smart's preferred extraction window / stride and the shared-skill optimizer defaults to the reflexio backend. Emits a stall banner (`prependContext`) if learning has been idle. @@ -74,11 +77,11 @@ and `openclaw-smart repair`. `force_extraction=True` so the session's learnings are available before the next openClaw run. -The full design lives in -[`docs/superpowers/specs/2026-05-19-openclaw-smart-design.md`](../../../../docs/superpowers/specs/2026-05-19-openclaw-smart-design.md). +See the [plugin README](plugin/README.md) for the shipped installation and command surface. ## Skills + Six skill folders ship under `plugin/skills/` (`reflexio` is the always-on contract, the other five are user-invocable): | Skill | Purpose | @@ -92,6 +95,7 @@ Six skill folders ship under `plugin/skills/` (`reflexio` is the always-on contr ## Configuration + Plugin-side config lives in `plugin/openclaw.plugin.json` (no env vars required for normal use). The shell scripts honour these env knobs: @@ -116,6 +120,7 @@ required for normal use). The shell scripts honour these env knobs: ## Recursion guard + The reflexio backend uses `openclaw_provider` (a LiteLLM CustomLLM) to invoke the openclaw CLI for extraction. That CLI can in turn fire openClaw hooks back into openclaw-smart, which would re-publish the extractor's @@ -131,6 +136,7 @@ immediately — no buffer writes, no search, no publish. ## Multi-session limitation + The TS shim tracks one `activeSessionKey` per plugin instance to route the `reflexio_publish` tool to the right session. In concurrent multi-session use the last session-key seen wins. This matches @@ -138,6 +144,7 @@ claude-smart's behaviour and is intentional — see spec §10. ## Troubleshooting + * **Hook timing out / no inject** — check `~/.openclaw-smart/backend.log` for reflexio startup errors. The session-start backend autostart is best-effort; you can also start it diff --git a/reflexio/mem0/README.md b/reflexio/mem0/README.md index 4cd261d4e..06244e4ef 100644 --- a/reflexio/mem0/README.md +++ b/reflexio/mem0/README.md @@ -1,15 +1,17 @@ -# reflexio/mem0 +# /reflexio/mem0 Description: Drop-in mem0 hosted-client wrappers that keep mem0 behavior while mirroring learning events into Reflexio. ## Main Entry Points + - **Public exports**: `__init__.py` - Re-exports `MemoryClient`, `AsyncMemoryClient`, `Memory`, `AsyncMemory`, and Reflexio helper/failure classes for `from reflexio.mem0 import ...` imports. - **Hosted wrappers**: `_wrapper.py` - Subclasses mem0 hosted sync/async clients, mirrors `add()` conversations to Reflexio, optionally augments `search()`, and resolves mem0 identity scopes. - **Lifecycle facade**: `_facade.py` - Scope-aware Reflexio cleanup methods exposed as `client.reflexio` / async equivalent. -- **Packaging hooks**: `pyproject.toml` and `client_dist/pyproject.toml` - Declare the `mem0` optional extra and include this package in both full and lightweight client distributions. +- **Packaging hooks**: `../../pyproject.toml` and `../../client_dist/pyproject.toml` - Declare the `mem0` optional extra and include this package in both full and lightweight client distributions. ## Purpose + 1. **One-import migration** - Existing mem0 users switch from `mem0.MemoryClient` to `reflexio.mem0.MemoryClient` without changing normal mem0 calls. 2. **Best-effort Reflexio learning** - Hosted `add()` calls run mem0 first, then publish normalized user/assistant messages to Reflexio when `REFLEXIO_API_KEY`, `REFLEXIO_URL`, or an injected `ReflexioClient` is configured. 3. **Opt-in enriched retrieval** - `search()` remains mem0-only unless callers pass `include_reflexio=True`; Reflexio results are returned under a reserved `reflexio` namespace. @@ -17,11 +19,16 @@ Description: Drop-in mem0 hosted-client wrappers that keep mem0 behavior while m ## Architecture Pattern + The wrapper is pass-through by default: construction must not fail when Reflexio is absent, and Reflexio failures must not change a successful mem0 result. An injected `reflexio_client` cannot be combined with inline Reflexio settings (`reflexio_timeout`, `reflexio_api_key`, or `reflexio_url_endpoint`). `_wrapper.py` owns identity extraction from top-level args and simple one-level `AND` filters, message normalization, stable scope hashing, async/sync publish paths, and namespace-collision protection for opted-in search augmentation. `_facade.py` owns explicit Reflexio lifecycle calls and raises `ReflexioNotConfiguredError` only when the caller directly invokes a Reflexio operation without configuration. -## Key Contracts +## Key Endpoints / Commands / Contracts + - Install with `pip install 'reflexio-ai[mem0]'`; the optional extra tracks the certified `mem0ai>=2.0,<2.1` line. - `Memory` and `AsyncMemory` local classes are re-exported unchanged from mem0; only hosted `MemoryClient` and `AsyncMemoryClient` are wrapped. -- Do not let Reflexio publish/search side effects break mem0 compatibility: log or annotate failures while preserving mem0 return values unless the caller opted into a Reflexio-specific operation. +## Requirements / Problems to Avoid + + +- **Do not let Reflexio publish/search side effects break mem0 compatibility**: log or annotate failures while preserving mem0 return values unless the caller opted into a Reflexio-specific operation. - Keep the `reflexio` search-result namespace reserved and raise `ReflexioNamespaceCollisionError` when an opted-in mem0 result already owns that key. diff --git a/reflexio/server/README.md b/reflexio/server/README.md index 0664c8de0..9dbd6d938 100644 --- a/reflexio/server/README.md +++ b/reflexio/server/README.md @@ -1,8 +1,9 @@ -# Reflexio Server +# /reflexio/server Description: FastAPI backend server that processes user interactions to generate profiles, extract playbooks, and evaluate agent success ## Table of Contents + - [Main Entry Points](#main-entry-points) - [Cache](#cache) - [API Endpoints](#api-endpoints) @@ -25,14 +26,15 @@ Description: FastAPI backend server that processes user interactions to generate - [Unified Search Service](#unified-search-service) - [Storage](#storage) - [Configurator](#configurator) -- [Architecture Patterns](#architecture-patterns) +- [Architecture Pattern](#architecture-pattern) - [Request Flow](#request-flow) - [Service Pattern](#service-pattern) - - [Key Rules](#key-rules) + - [Requirements / Problems to Avoid](#requirements--problems-to-avoid) - [See Also](#see-also) ## Main Entry Points + - **API composer**: `api.py` - `create_app()` factory, middleware/capability wiring, OpenAPI auth decoration, and `core_router` aggregation - **Domain routes**: `routes/` - FastAPI route modules; add new public API surfaces here and include their routers in `api.py` - **Endpoint Helpers**: `api_endpoints/` - Shared handlers/helpers plus `RequestContext` used by route modules @@ -40,8 +42,14 @@ Description: FastAPI backend server that processes user interactions to generate - **Core Service**: `services/generation_service.py` - Main orchestrator - **Durable Learning**: `services/durable_learning/` - durable sliding-window admission, scheduling and extraction +## Purpose + + +Compose the shared HTTP API, resolve request dependencies, and coordinate durable learning, session evaluation, retrieval, and optional deployment capabilities. + ## Cache + **Directory**: `cache/` | File | Purpose | @@ -49,6 +57,7 @@ Description: FastAPI backend server that processes user interactions to generate | `reflexio_cache.py` | TTL-cached Reflexio instances (1 hour TTL, max 100 orgs) | **Key Functions**: + - `get_reflexio(org_id)` - Get or create cached instance - `invalidate_reflexio_cache(org_id)` - Invalidate after config changes - `clear_reflexio_cache()` - Clear entire cache (testing/admin) @@ -57,6 +66,7 @@ Description: FastAPI backend server that processes user interactions to generate ## API Endpoints + **Directory**: `api_endpoints/` **Route modules** live in `routes/` and are grouped by domain. `api.py` remains the composition root: it creates `core_router`, includes each domain router, and mounts the aggregate router into the FastAPI app. **Detailed handler documentation**: See [`api_endpoints/README.md`](api_endpoints/README.md) for the `RequestContext` contract and helper map. @@ -85,8 +95,10 @@ Description: FastAPI backend server that processes user interactions to generate | `precondition_checks.py` | Request validation | **Key Endpoints**: + - **Health/version**: `GET /`, `GET /health`, `GET /healthz`, `GET /healthz/eval`, `GET /meta/version` - **Identity/config**: `GET /api/whoami`, `GET /api/my_config`, `GET /api/get_config`, `POST /api/set_config`, `POST /api/update_config` +- **Provenance**: `POST /api/get_learning_provenance` - **Publish/direct writes**: `POST /api/publish_interaction`, `POST /api/add_user_profile`, `POST /api/add_user_playbook`, `POST /api/add_agent_playbook` - **Retrieval**: `POST /api/get_requests`, `POST /api/get_interactions`, `GET /api/get_all_interactions`, `GET /api/learning_status`, `POST /api/get_profiles`, `GET /api/get_all_profiles`, `POST /api/get_user_playbooks`, `POST /api/get_agent_playbooks`, `POST /api/get_agent_success_evaluation_results`, `POST /api/get_retrieved_learning_evaluation_results` - **Search/stats**: `POST /api/search`, `POST /api/search_profiles`, `POST /api/rerank_user_profiles`, `POST /api/search_interactions`, `POST /api/search_user_playbooks`, `POST /api/search_agent_playbooks`, `GET /api/storage_stats`, `GET /api/get_profile_statistics`, `POST /api/get_dashboard_stats`, `POST /api/get_playbook_application_stats` @@ -108,6 +120,7 @@ A batch carrying neither `request_id` nor `session_id` is refused before any rec ## Extension Registry + **File**: `extensions.py` `CapabilityRegistry` lets deployments register optional routers, startup/shutdown hooks, and cross-cutting services without hardcoding enterprise-only imports into the OSS app. `create_app()` builds the active registry, stores it on `app.state.capability_registry`, installs capability routers/startup/shutdown hooks, and exposes typed service lookup through `ServiceKey`. @@ -116,6 +129,7 @@ A batch carrying neither `request_id` nor `session_id` is refused before any rec ### Error reporting hook + `error_reporting.py` defines the vendor-neutral `ErrorReporter` protocol and the `configure_error_reporter`, `error_tags`, `set_error_tags`, and `capture_anomaly` facades. They are no-ops unless a deployment registers an implementation through @@ -125,19 +139,20 @@ diagnostics never change product control flow; exceptions raised by code inside ## LLM Client + **Directory**: `llm/` **Entry Point**: `litellm_client.py` - `LiteLLMClient` facade composed from focused mixins Key files: + - `litellm_client.py`: Stable import surface, client config/credential resolution, and `LiteLLMClient` facade - `_litellm_text_generation.py`, `_litellm_embedding.py`, `_litellm_structured_output.py`: Completion/tool-call, embedding, and structured-output mixins - `_litellm_json_extraction.py`, `_litellm_subprocess.py`, `_provider_concurrency.py`, `_litellm_types.py`: JSON parsing, hard-timeout subprocess snapshots/workers, per-provider concurrency caps (fail-open by default, fail-closed for configured providers), and shared public types/errors - `providers/`: Optional local/provider adapters (`claude-code/`, OpenClaw, local embedding, Nomic embedding, and GPU-only multilingual E5); registration is opt-in via environment/config -- `openai_client.py`: OpenAI implementation (legacy, do not use directly) -- `claude_client.py`: Claude implementation (legacy, do not use directly) - `llm_utils.py`: Helper functions for Pydantic model conversion **Features**: + - Uses LiteLLM for multi-provider support (OpenAI, Claude, Azure, OpenRouter, Gemini, custom endpoints, etc.) - **Custom endpoint support**: `CustomEndpointConfig` (model, api_key, api_base) takes priority over all other providers for LLM completion calls when configured with non-empty fields (but not embeddings) - **Gemini support**: Model names with `gemini/` prefix route through Google Gemini; API key from `api_key_config.gemini` @@ -171,23 +186,27 @@ response = client.generate_response("What is 2+2?", response_format=Answer) # R ``` **Rules**: + - **ALWAYS use `LiteLLMClient`**, never import `OpenAIClient` or `ClaudeClient` directly - **ALWAYS use Pydantic models** for structured outputs (dict-based schemas are not supported) ## Prompts + **Directory**: `prompt/` **Detailed Documentation**: See [`prompt/prompt_bank/README.md`](prompt/prompt_bank/README.md) for the versioned template system. Key components: + - `prompt_manager.py`: PromptManager for loading and rendering -- `prompt_bank/`: Templates by prompt_id (metadata.json + version.prompt files) +- `prompt_bank/`: Templates by prompt ID with versioned `.prompt.md` files and YAML frontmatter **Pattern**: Access via `request_context.prompt_manager.render_prompt(prompt_id, variables)` ## Site Variables + **Directory**: `site_var/` **Detailed Documentation**: See [`site_var/README.md`](site_var/README.md) for the full configuration and feature flag system. @@ -203,6 +222,7 @@ Access: `SiteVarManager().get_site_var(key)` for raw values, `feature_flags.is_f ## Services + **Directory**: `services/` **Detailed Documentation**: See [`services/README.md`](services/README.md) for the per-directory file index across generation, evaluation, async extraction, search, and persistence. @@ -210,6 +230,7 @@ Access: `SiteVarManager().get_site_var(key)` for raw values, `feature_flags.is_f **Service Boundary**: The service layer owns LLM orchestration, extraction, evaluation, optimization, search preparation, storage access, and long-running operation state. API endpoints should validate/authenticate requests, build `RequestContext`, and delegate into `Reflexio` or focused service helpers rather than embedding business logic. **Encapsulated Components**: + - **Publish pipeline**: `generation_service.py` coordinates interaction persistence, profile generation, playbook generation, and deferred evaluation scheduling. - **Profile memory**: `profile/` extracts, deduplicates, and applies user profile updates. - **Playbook memory**: `playbook/` extracts and consolidates user playbooks, durably schedules bounded same-version aggregation, and reconstructs aggregation change logs from lineage. @@ -226,6 +247,7 @@ Access: `SiteVarManager().get_site_var(key)` for raw values, `feature_flags.is_f ### Orchestrator + **File**: `generation_service.py` - GenerationService Automatic publish flow: @@ -240,8 +262,8 @@ existing extractor stack; HTTP waiting does not own an extraction slot. Called by API endpoints via `Reflexio` **Profile Timeout Troubleshooting**: -- Use `python -m reflexio.scripts.reproduce_profile_timeout --mode storage --org-id --user-id ` to reproduce with real interactions. -- Use `--mode log --log-path server_log.txt` to replay extraction prompts captured in logs. + +- Inspect `services/profile/components/extractor.py` and `llm/litellm_client.py` for generation and provider timeout handling. - Look for structured events in logs: - `event=profile_extract_llm_start` / `event=profile_extract_llm_end` - `event=llm_request_start` / `event=llm_request_end` @@ -250,6 +272,7 @@ Called by API endpoints via `Reflexio` ### Base Infrastructure + - `base_generation_service.py`: Stable `BaseGenerationService` import surface plus service-specific orchestration hooks (parallel extractor execution via ThreadPoolExecutor, `EXTRACTOR_TIMEOUT_SECONDS = 300` per-extractor safety timeout) - `base_generation/`: Mixins for batch progress, config filtering, extraction lifecycle, should-run prechecks, status transitions, and usage billing that keep `base_generation_service.py` navigable without changing caller imports - `extractor_config_utils.py`: Shared utility for filtering extractor configs by source, `allow_manual_trigger`, and extractor names @@ -259,23 +282,25 @@ Called by API endpoints via `Reflexio` - `service_utils.py`: Utilities (`construct_messages_from_interactions()`, `format_interactions_to_history_string()` (prepends tool usage info when `tools_used` is present), `extract_json_from_string()`, `log_model_response()` for colored LLM response logging) **Operation State Management** (via `OperationStateManager` in `operation_state_utils.py`): + - Centralized manager for all `_operation_state` table interactions with 6 use cases: 1. **Progress tracking**: Rerun + manual batch operations (key: `{service}::{org_id}::progress`) 2. **Concurrency lock**: Atomic lock with request queuing (key: `{service}::{org_id}[::scope_id]::lock`) 3. **Extractor bookmark**: Track last-processed interactions per extractor (key: `{service}::{org_id}[::scope_id]::{name}`) - 4. **Aggregator bookmark**: Track last-processed raw_feedback_id per aggregator + 4. **Aggregator bookmark**: Legacy aggregator bookmarks; active incremental aggregation uses `storage_base/playbook/_aggregation.py` 4b. **Cluster fingerprints**: Track cluster membership fingerprints for change detection (key: `{service}::{org_id}::{name}[::version]::clusters`) 5. **Simple lock**: Non-queuing lock for cleanup operations 6. **Cancellation**: Cooperative cancellation for batch operations (`request_cancellation()`, `is_cancellation_requested()`, `mark_cancelled()`). Uses separate DB row (key: `{service}::{org_id}::cancellation`) to avoid lost-update race conditions with progress updates. - Stale lock timeout: 5 minutes (assumes crashed if lock held longer) -- Lock scoping: Profile generation = per-user, Playbook generation = per-org -- Re-run mechanism: If new request arrives during generation, `pending_request_id` is set and generation re-runs after completion +- Automatic, manual, and resumed extraction share `durable_learning/user_lease.py` ownership. Operation-state locks/bookmarks remain for their scoped legacy and batch workflows; they are not the automatic stream scheduler. ### Profile Generation + **Directory**: `services/profile/` Key files: + - `service.py`: Service orchestrator and profile persistence/finalization - `components/extractor.py`: Extractor that generates profile updates - `components/consolidator.py`: Consolidates newly extracted profiles against existing DB profiles using LLM @@ -299,6 +324,7 @@ Key files: **Note**: All modes use `window_size` (per-extractor override or global). The key difference is that Regular checks stride_size before running, while Rerun/Manual always run. When no window is configured, rerun/manual falls back to `k=1000`. **Constructor Flags** (`ProfileGenerationService`): + - `allow_manual_trigger`: Include `manual_trigger=True` extractors (default: False) - `output_pending_status`: Set output profiles to PENDING status (default: False) @@ -330,17 +356,20 @@ Users can regenerate and manage profile versions using a four-state system: 3. Complete archiving: ARCHIVE_IN_PROGRESS → ARCHIVED **Use Cases**: + - Test prompt changes without affecting production profiles - Review AI-generated updates before deployment - Rollback to previous profile version if needed ### Playbook Extraction + **Directory**: `services/playbook/` **Detailed Documentation**: See [`services/playbook/README.md`](services/playbook/README.md) for detailed component documentation. Key files: + - `service.py`: Service orchestrator - `components/extractor.py`: Extractor that extracts user playbooks - `aggregation_trigger.py` / `aggregation_scheduler.py`: Durably signal, claim, lease, and retry bounded per-version aggregation work @@ -349,6 +378,7 @@ Key files: - `review_service.py`: Re-reviews current user playbooks selected by created-at bounds and commits each completed decision newest-first **Flow**: + - Interactions → PlaybookExtractor (extraction-only) → PlaybookConsolidator (consolidates new vs existing DB playbooks) → UserPlaybook (with optional `blocking_issue`) → Storage - UserPlaybook write → durable hourly-coalesced signal → PlaybookAggregationScheduler → fixed-page invalidation drain → same-version centroid match → one current-agent-plus-bounded-delta refresh per changed cluster → bounded residual clustering → AgentPlaybook → Storage - `POST /api/run_playbook_aggregation` → fenced, capped administrative full rerun @@ -389,6 +419,7 @@ for storage contracts and failure dispositions. **Note**: All modes use `window_size` (per-extractor override or global). The key difference is that Regular checks stride_size before running, while Rerun/Manual always run. When no window is configured, rerun/manual falls back to `k=1000`. **Constructor Flags** (`PlaybookGenerationService`): + - `allow_manual_trigger`: Include `manual_trigger=True` extractors (default: False) - `output_pending_status`: Set output user playbooks to PENDING status (default: False) @@ -410,9 +441,11 @@ Similar to profiles, user playbooks support versioning: ### Agent Success Evaluation + **Directory**: `services/agent_success_evaluation/` Key files: + - `service.py`: `AgentSuccessEvaluationService`, the request-path service orchestrator (tracks run outcome flags: `last_run_result_count`, `has_run_failures()`) - `components/evaluator.py`: `AgentSuccessEvaluator`, evaluates success at session level (all interactions as one group) - `agent_success_evaluation_constants.py`: Output schema (`AgentSuccessEvaluationOutput`) @@ -432,9 +465,11 @@ Key files: ### Durable Learning Queue + **Directory**: `services/durable_learning/` Key files: + - `admission.py`: Atomic publish admission and eligibility snapshots. - `scheduler.py` / `worker.py`: Always-on discovery, bounded worker turns and renewable user leases. - `window_executor.py` / `window_codec.py`: Model execution, saved outcomes and frozen window policies. @@ -452,9 +487,11 @@ effort. `GET /api/learning_status` reads required cursor coverage. Legacy ### Async Extraction + **Directory**: `services/extraction/` Key files: + - `extraction/resumable_agent.py`: Resumable extraction agent runtime - `extraction/resume_scheduler.py` and `extraction/resume_worker.py`: Background scheduling/worker loop for paused extraction runs - `extraction/pending_tool_call_dispatch.py` and `extraction/prior_answer_search.py`: Tool surface and prior-answer context for async extraction agents @@ -464,9 +501,11 @@ Key files: ### Shadow Comparison and Evaluation Overview + **Directories**: `services/shadow_comparison/`, `services/evaluation_overview/` Key files: + - `shadow_comparison/judge.py`: Per-turn regular-vs-shadow judge - `shadow_comparison/dispatcher.py` and `shadow_comparison/worker.py`: Publish-time dispatch and bounded background execution for shadow verdict writes - `shadow_comparison/outcome.py`: Verdict outcome model helpers @@ -478,9 +517,11 @@ Key files: ### Playbook Optimizer and Braintrust + **Directories**: `services/playbook_optimizer/`, `services/braintrust/` Key files: + - `playbook_optimizer/optimizer.py`: Scenario-based playbook optimization loop - `playbook_optimizer/scheduler.py` and `rollout.py`: Scheduling and rollout helpers - `playbook_optimizer/judge.py`, `models.py`, `scenario_resolver.py`: Evaluation and scenario resolution models @@ -491,6 +532,7 @@ Key files: ### Lineage + **Directory**: `services/lineage/` | File | Purpose | @@ -502,6 +544,7 @@ Key files: ### Query Reformulator + **File**: `services/pre_retrieval/_query_reformulator.py` - `QueryReformulator` Reformulates user search queries into clean, normalized natural language for improved search recall. Resolves conversation context, expands abbreviations, fixes grammar. Enabled per-request via `enable_reformulation` parameter. @@ -514,6 +557,7 @@ Reformulates user search queries into clean, normalized natural language for imp ### Unified Search Service + **File**: `services/unified_search_service.py` - `run_unified_search()` Searches across all entity types (profiles, agent_playbooks, user_playbooks) in parallel via a two-phase approach: @@ -525,6 +569,7 @@ Pre-computed embeddings passed to storage methods via `query_embedding` paramete ### Storage + **Directory**: `services/storage/` | File | Purpose | @@ -538,16 +583,17 @@ Pre-computed embeddings passed to storage methods via `query_embedding` paramete **Pattern**: **NEVER import storage implementations directly** - Always use `request_context.storage` **Key Methods**: + - CRUD: profiles, interactions, playbooks, results, requests, playbook aggregation change logs - `get_sessions(offset, top_k, session_id)` → `dict[str, list[RequestInteractionDataModel]]` (groups by session_id; paginates per-session — `top_k`/`offset` count sessions, and each returned session includes all of its requests) - `get_rerun_user_ids(user_id, start_time, end_time, source, agent_version)` → `list[str]` - Get distinct user IDs matching filters for rerun workflows (pushes filtering to storage layer) -- `get_feedbacks(status_filter, feedback_status_filter)` - Filter by playbook status and approval status -- `save_feedbacks()` → returns `list[Feedback]` with `feedback_id` populated (callers can ignore return) +- `get_agent_playbooks(status_filter=..., playbook_status_filter=...)` - Filter by playbook status and approval status +- `save_agent_playbooks()` → returns `list[AgentPlaybook]` with `agent_playbook_id` populated (callers can ignore return) - Selective playbook operations (used by cluster change detection): - - `archive_feedbacks_by_ids(feedback_ids)` - Archive specific agent playbooks by ID (skips APPROVED) - - `restore_archived_feedbacks_by_ids(feedback_ids)` - Restore archived agent playbooks by ID - - `delete_feedbacks_by_ids(feedback_ids)` - Delete agent playbooks by ID - - `delete_raw_feedbacks_by_ids(raw_feedback_ids)` - Delete user playbooks by ID + - `archive_agent_playbooks_by_ids(agent_playbook_ids)` - Archive specific agent playbooks by ID (skips APPROVED) + - `restore_archived_agent_playbooks_by_ids(agent_playbook_ids)` - Restore archived agent playbooks by ID + - `delete_agent_playbooks_by_ids(agent_playbook_ids)` - Delete agent playbooks by ID + - `delete_user_playbooks_by_ids(user_playbook_ids)` - Delete user playbooks by ID - Vector search via LiteLLMClient embeddings - Operation state: `get_operation_state()`, `upsert_operation_state()`, `get_operation_state_with_new_request_interaction()`, `try_acquire_in_progress_lock()` - All operation state interactions are managed through `OperationStateManager` (in `operation_state_utils.py`) @@ -555,9 +601,11 @@ Pre-computed embeddings passed to storage methods via `query_embedding` paramete ### Configurator + **Directory**: `services/configurator/` Key files: + - `configurator.py`: DefaultConfigurator - loads YAML config, creates storage - `local_file_config_storage.py`: Local file-based config storage **Config Storage Priority** (in `DefaultConfigurator`): @@ -568,75 +616,29 @@ Key files: Access: `request_context.configurator` -## Architecture Patterns +## Architecture Pattern + ### Request Flow -``` -API Request (api.py) - -> API Endpoint (api_endpoints/) - -> get_reflexio() (cache/) - -> Reflexio (reflexio_lib.py) - -> GenerationService - ├─> ProfileGenerationService → Storage - ├─> PlaybookGenerationService → Storage - └─> agent_success_evaluation/scheduler.py:GroupEvaluationScheduler (deferred 10 min) → agent_success_evaluation/runner.py:run_group_evaluation → agent_success_evaluation/service.py → Storage -``` -```mermaid -flowchart TB - subgraph API["API Layer"] - A[api.py] --> B[api_endpoints/] - end - - B --> C[get_reflexio] - C --> D[Reflexio] - D --> E[GenerationService] - - subgraph ProfileService["ProfileGenerationService"] - E --> F1[ProfileExtractor 1] - E --> F2[ProfileExtractor N] - F1 --> PC[ProfileConsolidator] - F2 --> PC - PC --> PU[ProfileUpdater] - end - - subgraph PlaybookService["PlaybookGenerationService"] - E --> G1[PlaybookExtractor 1] - E --> G2[PlaybookExtractor N] - G1 --> FD[PlaybookConsolidator] - G2 --> FD - end - - subgraph EvalService["AgentSuccessEvaluationService"] - E -.->|deferred 10 min| SCH[agent_success_evaluation/scheduler.py
GroupEvaluationScheduler] - SCH --> H1[AgentSuccessEvaluator 1] - SCH --> H2[AgentSuccessEvaluator N] - end - - PU --> I[(Storage)] - FD --> I - H1 --> I - H2 --> I - - subgraph Support["Supporting Components"] - J[LiteLLMClient] - K[PromptManager] - L[Configurator] - end - - J -.-> F1 - J -.-> G1 - J -.-> H1 - J -.-> PC - J -.-> FD - K -.-> F1 - K -.-> G1 - K -.-> H1 + +```text +routes/ -> RequestContext + get_reflexio() -> ../lib/reflexio_lib.py + -> generation_service.py -> durable_learning/admission.py + -> scheduler/worker -> frozen extraction window + -> profile/ or playbook/ -> fenced outputs + cursor + effect receipt + -> agent_success_evaluation/scheduler.py -> runner.py -> evaluation storage + -> unified_search_service.py -> pre_retrieval/ + storage/ + retrieval/ ``` +Provider calls occur outside storage transactions. Durable workers, manual +operations, and resumed extraction share the same user lease; HTTP response +waiting only observes committed cursor coverage. + ### Service Pattern -All services follow BaseGenerationService: + +Generation services follow `BaseGenerationService`; focused search and read-side services have their own entry points: 1. Load extractor configs from YAML 2. Load generation service config from request (runtime parameters) 3. Filter extractors by source, `allow_manual_trigger`, and extractor names (via `extractor_config_utils`) @@ -645,6 +647,7 @@ All services follow BaseGenerationService: 6. Process and save results to storage **Extractor Pattern**: Multiple extractors run in parallel, each handling its own data collection. Each extractor: + - Receives **ExtractorConfig** (from YAML): Static configuration like prompts and settings - Receives **GenerationServiceConfig** (from request): Runtime parameters like user_id, source - **Collects its own interactions** using `extractor_interaction_utils.py`: @@ -654,32 +657,39 @@ All services follow BaseGenerationService: - Updates per-extractor bookmark state after processing (via `OperationStateManager`) **Per-Extractor Window Overrides**: Each extractor config can override global window settings: + - `window_size_override`: Override global `window_size` for this extractor - `stride_size_override`: Override global `stride_size` for this extractor - Each extractor applies its own override or falls back to global values -### Key Rules +## Requirements / Problems to Avoid + **Reflexio Instances**: + - **NEVER instantiate `Reflexio()` directly** in API endpoints - **ALWAYS use**: `get_reflexio(org_id)` from `cache/reflexio_cache.py` - Cache invalidated automatically on config changes **Storage**: + - **NEVER import storage implementations directly** - **ALWAYS use**: `request_context.storage` (type: BaseStorage) **LLM**: + - **NEVER import OpenAIClient/ClaudeClient directly** - **ALWAYS use**: `LiteLLMClient` (uses LiteLLM for multi-provider support) **Prompts**: + - **NEVER hardcode prompts** - **ALWAYS use**: `request_context.prompt_manager.render_prompt(prompt_id, variables)` -- Prompts versioned in `prompt_bank/` +- Prompts versioned in `prompt/prompt_bank/` ## See Also + - [Code Map (root README)](../README.md) -- high-level overview of all Reflexio components - [API Endpoints README](api_endpoints/README.md) -- RequestContext contract and handler/helper map - [Services README](services/README.md) -- per-directory index of the business-logic layer diff --git a/reflexio/server/api_endpoints/README.md b/reflexio/server/api_endpoints/README.md index cf8eae9c8..99c59205c 100644 --- a/reflexio/server/api_endpoints/README.md +++ b/reflexio/server/api_endpoints/README.md @@ -1,9 +1,10 @@ -# server/api_endpoints +# /reflexio/server/api_endpoints Description: Shared helpers between FastAPI domain routes and business logic — builds `RequestContext`, validates requests, and delegates into `Reflexio`. Public route declarations live in `../routes/` and are aggregated by `core_router` in `../api.py`; the files here are reusable handlers/helpers those routes call. > For the complete endpoint list (publish, retrieval, search, profile/playbook lifecycle, evaluation, Braintrust, operations), see the parent [server README](../README.md#api-endpoints). -## Files +## Main Entry Points + | File | Purpose | |------|---------| @@ -15,13 +16,19 @@ Description: Shared helpers between FastAPI domain routes and business logic — | `stall_state_api.py` | `GET /api/stall_state`, `POST /api/stall_state/notified` — extraction-agent waiting state. | | `precondition_checks.py` | `precondition_checks()` — shared request validation. | +## Purpose + + +Build per-request dependencies, validate input, and share handler logic across the domain routers without duplicating business services. + ## Architecture Pattern + ``` api.py (create_app + core_router) -> routes/.py (FastAPI routers) -> Depends(get_request_context) -> RequestContext(org_id, storage, configurator, prompt_manager) - -> get_reflexio(org_id) -> Reflexio (reflexio_lib) -> services/ + -> get_reflexio(org_id) -> Reflexio (lib/reflexio_lib.py) -> services/ ``` - **`RequestContext` is the context-passing contract** — handlers receive it via `Depends` and never reach for storage/config/prompts globally. @@ -30,6 +37,7 @@ api.py (create_app + core_router) ## Requirements / Problems to Avoid + - **NEVER instantiate `Reflexio()` in a handler** — use `get_reflexio(org_id)` from `server/cache/`. - **Keep business logic in `services/`** — endpoints validate, build context, and delegate; they don't embed extraction/evaluation logic. - **Pending-tool-call writes can race** — use the migration-retry + HMAC-verify helpers already in `pending_tool_call_api.py` rather than writing raw. diff --git a/reflexio/server/prompt/prompt_bank/README.md b/reflexio/server/prompt/prompt_bank/README.md index c8cd0a176..a3025d6c9 100644 --- a/reflexio/server/prompt/prompt_bank/README.md +++ b/reflexio/server/prompt/prompt_bank/README.md @@ -1,15 +1,26 @@ -# prompt_bank - -File-based versioned prompt templates for LLM operations. +# /reflexio/server/prompt/prompt_bank +Description: File-based versioned prompt templates for LLM operations. > Part of the [Reflexio Server](../../README.md). See also the [Playbook Service](../../services/playbook/README.md) for prompt usage in playbook extraction. ## Main Entry Points + - **Manager**: `../prompt_manager.py` — `PromptManager` - **Templates**: Each subdirectory is a `prompt_id` -## Directory Structure +## Purpose + + +Keep model instructions versioned and shared by services, with declared variables validated during rendering. + +## Architecture Pattern + + +`PromptManager` discovers prompt-ID directories and reads version metadata from each `.prompt.md` file. Callers render the active or explicitly selected version through request context. + +### Directory Structure + ``` prompt_bank/ @@ -25,6 +36,7 @@ prompt_bank/ ## File Format + Each `.prompt.md` file is self-contained with YAML frontmatter: ```markdown @@ -42,6 +54,7 @@ Your prompt content with {var1} and {var2} placeholders. ### Frontmatter Fields + | Field | Type | Required | Purpose | |-------|------|----------|---------| | `active` | bool | No | `true` on the active version. Exactly one per prompt_id | @@ -51,6 +64,7 @@ Your prompt content with {var1} and {var2} placeholders. ## Usage + ```python # Access via request_context rendered = request_context.prompt_manager.render_prompt( @@ -61,12 +75,14 @@ rendered = request_context.prompt_manager.render_prompt( ## Adding a New Prompt + 1. Create directory: `mkdir prompt_bank/my_new_prompt/` 2. Create `v1.0.0.prompt.md` with frontmatter and `{variable}` placeholders 3. Set `active: true` in frontmatter ## Version Naming Convention + File names: `v{MAJOR}.{MINOR}.{PATCH}.prompt.md` - **MAJOR**: Breaking changes that introduce a different set of variables @@ -75,7 +91,8 @@ File names: `v{MAJOR}.{MINOR}.{PATCH}.prompt.md` ## Deactivating a Prompt Version -When creating a replacement version, deactivate the old version by **removing** the `active: true` line from its frontmatter. Do NOT add `active: false` — simply omit the field. Prompts without the `active` field default to `active: false` (see `prompt_manager.py` line 211: `meta.get("active", False)`). Only the new replacement version should have `active: true`. + +When creating a replacement version, deactivate the old version by **removing** the `active: true` line from its frontmatter. Do NOT add `active: false` — simply omit the field. Prompts without the `active` field default to `active: false` (see `../prompt_manager.py`: `meta.get("active", False)`). Only the new replacement version should have `active: true`. **Before** (old version `v1.0.0.prompt.md`): ```yaml @@ -107,7 +124,8 @@ variables: --- ``` -## Key Rules +## Requirements / Problems to Avoid + - **Prompt ID** = Directory name - **Variables** use `{variable_name}` syntax in prompt body @@ -117,5 +135,6 @@ variables: ## See Also + - [Server README](../../README.md) -- FastAPI backend component overview - [Playbook Service README](../../services/playbook/README.md) -- how prompts are used in playbook extraction and aggregation diff --git a/reflexio/server/services/README.md b/reflexio/server/services/README.md index 4eb7b4374..94627f38f 100644 --- a/reflexio/server/services/README.md +++ b/reflexio/server/services/README.md @@ -1,4 +1,4 @@ -# server/services +# /reflexio/server/services Description: Core business-logic layer — LLM orchestration, extraction, evaluation, optimization, search preparation, storage access, and long-running operation state. > This is a directory-local index. For the full request flow, workflow tables (versioning, generation modes, cluster change detection), and the `OperationStateManager` use cases, see the parent [server README](../README.md#services). @@ -7,6 +7,7 @@ Description: Core business-logic layer — LLM orchestration, extraction, evalua ## LLM Pipeline Module Contract + LLM/pipeline modules use a shared vocabulary across OSS and enterprise: `service.py` for request-path entry points, `runner.py` for manual/background workflow entry points, `scheduler.py` for periodic/deferred execution, @@ -18,19 +19,23 @@ Files are optional. Do not create empty files only to satisfy the vocabulary. Complete cutover migrations update consumers, tests, docs, and monkeypatch strings before deleting old import paths in the same PR. -## Orchestration & Base Infrastructure +## Main Entry Points + + +### Orchestration & Base Infrastructure + | File | Purpose | |------|---------| | `generation_service.py` | `GenerationService` — admits interactions through the durable stream engine, schedules deferred evaluation, and coordinates manual generation under the shared user lease. | - | `search_metering_worker.py` | Bounded process-local search-metering queue. Four daemon workers emit `search_request` and `learning_applied` after the response path, with fail-open drops, a five-second shutdown drain, and trace-linked `search.metering` transactions; persistence requires a registered usage recorder. | | `base_generation_service.py` + `base_generation/` | `BaseGenerationService` stable import surface plus mixins for batch progress, config filtering, extraction lifecycle, should-run prechecks, status transitions, and usage billing. Per-extractor timeout `EXTRACTOR_TIMEOUT_SECONDS = 300`. | | `operation_state_utils.py` | `OperationStateManager` — all `_operation_state` access (progress, concurrency locks, extractor/aggregator bookmarks, cluster fingerprints, cancellation). | | `extractor_config_utils.py`, `extractor_interaction_utils.py` | Filter extractors by source / `allow_manual_trigger` / names; per-extractor stride + window + bookmark handling. | | `deduplication_utils.py`, `service_utils.py`, `embedding_text.py` | LLM dedup helpers (used by `ProfileConsolidator` + `PlaybookConsolidator`), message construction / JSON extraction / response logging, embedding text builders. | -## Generation Services +### Generation Services + | Directory | Entry class | Key files | |-----------|-------------|-----------| @@ -38,14 +43,16 @@ strings before deleting old import paths in the same PR. | `playbook/` | `PlaybookGenerationService` | `aggregation_trigger.py` durably signals work; `aggregation_scheduler.py` claims fenced bounded units; `components/aggregator.py` performs same-version centroid matching and residual clustering — see [README](playbook/README.md) | | `agent_success_evaluation/` | `AgentSuccessEvaluationService` | `service.py` (session-level service), `runner.py` (`run_group_evaluation`), `scheduler.py` (`GroupEvaluationScheduler`, 10-min defer), `regen_jobs.py`, `components/evaluator.py` | -## Durable and Async Extraction +### Durable and Async Extraction + | Directory | Purpose | |-----------|---------| | `durable_learning/` | `admission.py` commits incoming streams; `scheduler.py` / `worker.py` run bounded user turns; `window_executor.py` / `window_codec.py` compute frozen windows; `user_lease.py` shares ownership with manual/resumed generation; `waiting.py` bounds HTTP waits; `local.py` recovers standalone-library work. | | `extraction/` | Shared async extraction runtime: `resumable_agent.py`, `resume_scheduler.py`, `resume_worker.py`, `pending_tool_call_dispatch.py` (`ask_human`), `prior_answer_search.py`, `agent_run_records.py`, and `outcome.py`. Long-horizon / tool-mediated extraction continues outside the request path. See [README](extraction/README.md). | -## Evaluation, Search & Integrations +### Evaluation, Search & Integrations + | Path | Purpose | |------|---------| @@ -61,16 +68,28 @@ strings before deleting old import paths in the same PR. | `search_exposure.py` | Optional synchronous recorder contract for final user-playbook result sets. Enterprise capability registration installs the recorder and makes authenticated unified/direct search fail closed before metering/response. Shared `create_app()` has no default recorder; other constructions persist exposures only when they register one. Direct search omits empty batches. | | `retrieval/` | `relevance_floor.py` — result relevance thresholding. `temporal.py` — temporal post-processing driven by reformulation signals: query time windows → per-arm SQL filters, near-duplicate freshness collapse for current-value questions, timestamp ordering for latest-value questions. `user_context_guard.py` — high-precision detection of explicit personalization opt-outs, including Simplified and Traditional Chinese, before user-context retrieval. (Superseded/expired rows are already excluded by storage search SQL.) | -## Persistence & Config +### Persistence & Config + | Path | Purpose | |------|---------| | `storage/` | `storage_base/` and `sqlite_storage/` keep legacy domain facades while focused subpackages own `profiles/`, `playbook/`, `agent_run/`, `governance`, durable extraction streams, and SQLite `base/` helpers. SQLite hybrid search preserves Porter FTS for ASCII and adds bounded Unicode substring candidates only when a query contains at least one non-ASCII alphanumeric character, including mixed-script queries; emoji-only and punctuation-only queries are ineligible. `storage_base/playbook/_aggregation.py` defines fenced aggregation state; SQLite implements it in the matching playbook package. Access via `request_context.storage` only. | | `configurator/` | `DefaultConfigurator` — loads YAML config and creates the storage backend. | -## Key Rules +## Purpose + + +Translate admitted interactions into profiles/playbooks, evaluate sessions, and retrieve learning through shared storage and configuration contracts. + +## Architecture Pattern + + +Generation uses configured actors and fenced persistence; deferred workers rebuild request context rather than retaining request objects. Search composes reformulation, parallel entity retrieval, filtering, optional exposure recording, and asynchronous metering. Keep service-owned implementations behind their public entry points. + +## Requirements / Problems to Avoid + -- **NEVER instantiate services bypassing the Service Pattern** — extend `BaseGenerationService`; load YAML configs, create actors, run in parallel, save to storage. +- **Generation pipelines extend `BaseGenerationService`** — load configs, create actors, run in parallel, and save through storage. Focused search/read-side services retain their own entry points. - **NEVER import storage implementations directly** — use `request_context.storage` (`BaseStorage`). - **ALWAYS use `LiteLLMClient`** for completions/embeddings and `request_context.prompt_manager.render_prompt(...)` for prompts — no hardcoded prompts, no direct OpenAI/Claude clients. - **All `_operation_state` writes go through `OperationStateManager`** — don't touch the table directly (it backs locks, bookmarks, progress, and cancellation). diff --git a/reflexio/server/services/agent_success_evaluation/README.md b/reflexio/server/services/agent_success_evaluation/README.md index eea2fd502..fc6903b9b 100644 --- a/reflexio/server/services/agent_success_evaluation/README.md +++ b/reflexio/server/services/agent_success_evaluation/README.md @@ -1,8 +1,8 @@ -# agent_success_evaluation +# /reflexio/server/services/agent_success_evaluation +Description: Session-level agent success and retrieved-learning evaluation. -Session-level agent success evaluation module. +## Main Entry Points -## Module Shape - `service.py`: `AgentSuccessEvaluationService`, the request-path service that runs configured evaluators and saves result rows. - `runner.py`: `run_group_evaluation(...)`, the background/manual workflow entry point that loads a session, runs the service, and marks operation state. After agent-success work it also runs the retrieved-learning evaluation (independent completion; generation + session-fingerprint fenced) and returns a `GroupEvaluationOutcome` carrying both statuses. @@ -14,7 +14,21 @@ Session-level agent success evaluation module. - `agent_success_evaluation_constants.py`: prompt/model output constants. - `agent_success_evaluation_utils.py`: request DTO and prompt-message construction helpers. -## Failure Classification +## Purpose + + +Evaluate completed sessions and feed persisted judgments to dashboard rollups and regeneration jobs. + +## Architecture Pattern + + +The inactivity scheduler or manual runner loads a session, runs configured evaluators, and persists agent-success and retrieved-learning outcomes independently under generation/session-fingerprint fencing. + +## Key Endpoints / Commands / Contracts + + +### Failure Classification + Unsuccessful sessions use `system_error`, `missing_tool`, `wrong_tool`, `insufficient_info_from_tool`, or `wrong_answer`. `system_error` is reserved for @@ -23,10 +37,14 @@ missing responses, persistent timeouts or 5xx responses, unavailable services, and quota or credit failures. The evaluation overview keeps these rows in task success while excluding them from the separate behavior-success denominator. -## Prompt IDs +### Prompt IDs + - Owns `agent_success_evaluation`, `retrieved_learning_relevance`, and `retrieved_learning_impact`. - Keeps historical/configured prompt ID `agent_success_evaluation_with_comparison` stable where prompt mapping tests require it. -Do not reintroduce the deleted service/evaluator/runner/scheduler legacy +## Requirements / Problems to Avoid + + +**Do not reintroduce** the deleted service/evaluator/runner/scheduler legacy module files. diff --git a/reflexio/server/services/evaluation_overview/README.md b/reflexio/server/services/evaluation_overview/README.md index 264da11ea..9ed85a16e 100644 --- a/reflexio/server/services/evaluation_overview/README.md +++ b/reflexio/server/services/evaluation_overview/README.md @@ -1,15 +1,33 @@ -# evaluation_overview +# /reflexio/server/services/evaluation_overview +Description: Read-side aggregation for `POST /api/get_evaluation_overview`. + +## Main Entry Points -Read-side aggregation module for `POST /api/get_evaluation_overview`. - `service.py` is the request-path entry point. It loads evaluation, citation, Braintrust, and optional shadow verdict data, then composes `GetEvaluationOverviewResponse`. - `components/` contains pure read-side aggregation helpers used by the service and focused tests. - `eval_sampler.py` stays at the package root because regenerate jobs also use it to sample evaluation sessions. +## Purpose + + +Combine evaluation, citation, Braintrust, and optional shadow data for the evaluation dashboard. + +## Architecture Pattern + + +The service loads persisted evidence and calls pure aggregation helpers; it does not mutate core state. + +## Key Endpoints / Commands / Contracts + + The overview reports task success across every evaluated session and a separate behavior-success metric that excludes `failure_type=system_error` rows from both numerator and denominator. A window with no behavior-evaluable rows returns a null behavior rate plus eligible/excluded counts for honest UI rendering. -This module mutates no core state. Keep response-shape changes in API schema tests and service integration tests. +## Requirements / Problems to Avoid + + +**This module mutates no core state.** Keep response-shape changes in API schema tests and service integration tests. diff --git a/reflexio/server/services/extraction/README.md b/reflexio/server/services/extraction/README.md index a0dfe2b5f..8155ef621 100644 --- a/reflexio/server/services/extraction/README.md +++ b/reflexio/server/services/extraction/README.md @@ -1,13 +1,16 @@ -# services/extraction +# /reflexio/server/services/extraction +Description: Shared async extraction runtime for profile and playbook pipelines. + +## Purpose -Shared async extraction runtime for profile and playbook pipelines. This package is intentionally domain-neutral. Profile and playbook modules own their prompts, schemas, extractor semantics, and storage decisions; this package owns the reusable runtime needed when extraction pauses, waits for more information, and resumes outside the request path. -## Files +## Main Entry Points + | File | Responsibility | |------|----------------| @@ -19,7 +22,13 @@ information, and resumes outside the request path. | `resume_worker.py` | Resumes paused runs, rebuilds request context, and records retry state. Finalization uses an immutable run-keyed receipt so learning writes and retry billing reuse the same persisted IDs. | | `outcome.py` | Provides the generic extraction outcome wrapper used by callers. | -## Boundary Rules +## Architecture Pattern + + +Extraction records durable agent state and pending human calls; the scheduler discovers resumable work and the worker reconstructs context, resumes the agent, and commits finalization with an immutable receipt. Domain services retain prompts and learning semantics. + +## Requirements / Problems to Avoid + - Keep profile-specific and playbook-specific extraction behavior in their own modules; call this package only for shared async runtime concerns. @@ -36,7 +45,11 @@ information, and resumes outside the request path. bindings remain readable for backward compatibility; playbook resume derives an in-memory owner only from complete, unanimous persisted source evidence. -## Resume discovery +## Key Endpoints / Commands / Contracts + + +### Resume discovery + When an `org_id_provider` is installed, the scheduler calls it on every tick and treats its actionable org list as authoritative for cross-ref discovery. Without diff --git a/reflexio/server/services/playbook/README.md b/reflexio/server/services/playbook/README.md index 7de6ae6cb..4ccd67bdb 100644 --- a/reflexio/server/services/playbook/README.md +++ b/reflexio/server/services/playbook/README.md @@ -1,10 +1,11 @@ -# Playbook Service +# /reflexio/server/services/playbook Description: Evidence-grounded playbook extraction, candidate review, aggregation, and consolidation pipeline > Part of the [Reflexio Server](../../README.md). See also the [Prompt Bank](../../prompt/prompt_bank/README.md) for prompt template details. ## Main Entry Points + - **Service Orchestrator**: `service.py` - Manages playbook extraction lifecycle (regular, rerun, manual modes) - **Playbook Extractor**: `components/extractor.py` - Extracts user playbooks from interactions via LLM - **Candidate Reviewer**: `components/reviewer.py` - Accepts, narrowly revises, or rejects validated normal-extraction candidates before consolidation @@ -15,6 +16,7 @@ Description: Evidence-grounded playbook extraction, candidate review, aggregatio ## Supporting Files + | File | Purpose | |------|---------| | `playbook_service_constants.py` | Prompt IDs for all playbook operations | @@ -26,10 +28,17 @@ Description: Evidence-grounded playbook extraction, candidate review, aggregatio | `aggregation_scheduler.py` | Polling, fleet claim/lease handling, retries, and structured aggregation progress telemetry | | `aggregation_prompt_processing.py` | Optional aggregation-boundary interfaces and helpers for prompt preprocessing, contextual prompt guidance, and output post-processing | -## Architecture +## Purpose + + +Turn evidenced corrections into user playbooks, review/consolidate them, and aggregate same-version user learning into agent playbooks while retaining provenance and lifecycle invariants. + +## Architecture Pattern + ### Data Flow + ``` Interactions -> PlaybookExtractor (per-extractor, extraction-only, parallel) @@ -44,6 +53,7 @@ Interactions ### Playbook Extraction (`components/extractor.py`) + Extends `BaseGenerationService` extractor pattern. Each extractor: 1. Checks stride_size threshold before running 2. Constructs messages from interactions (via `service_utils.py`) @@ -60,6 +70,7 @@ continue; an unresolved malformed response fails the extraction run. ### Candidate Review (`components/reviewer.py`) + Strict normal candidates enter a fresh same-model review call before consolidation. The reviewer receives the request-bounded chronology, validated referenced turns, artifact-availability context, and relevant existing playbooks. @@ -74,6 +85,7 @@ missing or mixed-owner evidence fails closed before review or persistence. ### Persisted Review (`review_service.py`) + `POST /api/review_user_playbooks` selects the newest current user playbooks in an inclusive creation-time window, capped by `top_k`. A row whose original generation window or cited evidence can no longer be reconstructed yields a @@ -104,6 +116,7 @@ earlier decisions. ### Playbook Aggregation (`components/aggregator.py`) + Normal generation durably schedules bounded incremental aggregation through `aggregation_trigger.py`; `aggregation_scheduler.py` claims due work across processes. A successful drained run waits at least @@ -157,8 +170,8 @@ a stale centroid or rejected canonical rule from becoming the base of a later incremental refresh. The portable state contract is -`storage/storage_base/playbook/_aggregation.py`; SQLite implements it in -`storage/sqlite_storage/playbook/_aggregation.py`. Claims are lease-fenced, and +`../storage/storage_base/playbook/_aggregation.py`; SQLite implements it in +`../storage/sqlite_storage/playbook/_aggregation.py`. Claims are lease-fenced, and all multi-write effects must remain inside `storage.commit_scope()`. **Optional prompt processing**: deployments can register an @@ -168,7 +181,7 @@ aggregation prompt boundary, carries an opaque per-cluster processing context, injects extra prompt guidance only when preprocessing changed prompt input, and post-processes generated outputs before storage or model-response logging. -**Change Log**: The legacy `playbook_aggregation_change_logs` table is retired (Track B, 2026-06-24) — the aggregator no longer writes it. The change-log view is reconstructed on demand from `lineage_event` via `reconstruct_playbook_aggregation_change_log` (`lib/_agent_playbook.py`): each run emits `op=aggregate` events (the "added" side) and `status_change→superseded` events from the supersede calls (the "removed" side), grouped by the run's `request_id`. Per-row `updated` pairing is not reconstructed (`updated_agent_playbooks=[]`, a tolerated parity delta). +**Change Log**: The legacy `playbook_aggregation_change_logs` table is retired (Track B, 2026-06-24) — the aggregator no longer writes it. The change-log view is reconstructed on demand from `lineage_event` via `reconstruct_playbook_aggregation_change_log` (`../../../lib/_agent_playbook.py`): each run emits `op=aggregate` events (the "added" side) and `status_change→superseded` events from the supersede calls (the "removed" side), grouped by the run's `request_id`. Per-row `updated` pairing is not reconstructed (`updated_agent_playbooks=[]`, a tolerated parity delta). **Requirements / Problems to Avoid**: @@ -185,6 +198,7 @@ post-processes generated outputs before storage or model-response logging. ### Playbook Consolidation (`components/consolidator.py`) + Consolidates newly extracted playbooks against existing playbooks in the database via LLM semantic matching. For each NEW vs EXISTING pair the LLM returns one of four decision kinds, and the consolidator applies the chosen kind: - `unify` — merge multiple rows into one, archiving members and emitting one merged row. @@ -201,7 +215,15 @@ source unions and reviewer revisions and prevent overlapping duplicate survivors Inline consolidation always runs during generation (the legacy `deduplicator` feature flag is retired). The enterprise repo additionally ships a **scheduled second-pass job** (`reflexio_ext/server/services/playbook_reconsolidation/`) that re-runs consolidation daily over already-persisted rows — user playbooks per `(user_id, agent_version)` and agent playbooks per `agent_version` — by rendering each duplicate group as NEW candidates against an empty EXISTING side via `_consolidation_decisions`, then tombstoning merged sources with `merge_records` lineage. -## Prompt IDs +## Key Endpoints / Commands / Contracts + + +- **Review**: `POST /api/review_user_playbooks`; report is inline, apply is background and returns a run ID. +- **Aggregate**: `POST /api/run_playbook_aggregation`; administrative reruns retain lease fencing and the input cap. +- **Lifecycle**: see the [complete route map](../../README.md#api-endpoints) for create/update/status/upgrade/downgrade/delete and change-log reads. + +### Prompt IDs + | Constant | Prompt ID | Used By | |----------|-----------|---------| @@ -211,7 +233,8 @@ Inline consolidation always runs during generation (the legacy `deduplicator` fe | `PLAYBOOK_CANDIDATE_REVIEW_PROMPT_ID` | `playbook_candidate_review` | PlaybookCandidateReviewer | | `PLAYBOOK_AGGREGATION_PROMPT_ID` | `playbook_aggregation` | PlaybookAggregator | -## Key Output Schemas (in `playbook_service_utils.py`) +### Key Output Schemas (in `playbook_service_utils.py`) + | Class | Purpose | |-------|---------| @@ -221,7 +244,15 @@ Inline consolidation always runs during generation (the legacy `deduplicator` fe | `PlaybookGenerationRequest` | Request dataclass for playbook extraction | | `PlaybookAggregatorRequest` | Request dataclass for playbook aggregation | +## Requirements / Problems to Avoid + + +- **Fail closed on unreconstructable or invalid evidence**; manual review skips an unreconstructable row instead of inventing chronology. +- **Preserve per-version aggregation, bounded discovery, and transaction rules** listed in the aggregation section above. +- **Keep normal candidate review separate from expert/legacy paths**; a revision cannot invent evidence or a missing lesson. + ## See Also + - [Server README](../../README.md) -- FastAPI backend component overview - [Prompt Bank README](../../prompt/prompt_bank/README.md) -- versioned prompt template system used by playbook prompts diff --git a/reflexio/server/services/playbook_optimizer/README.md b/reflexio/server/services/playbook_optimizer/README.md index 16754b0e2..694f967ed 100644 --- a/reflexio/server/services/playbook_optimizer/README.md +++ b/reflexio/server/services/playbook_optimizer/README.md @@ -1,10 +1,25 @@ -# playbook_optimizer +# /reflexio/server/services/playbook_optimizer +Description: GEPA-driven optimization of playbook content using assistant rollouts and judged candidates. + +## Main Entry Points -GEPA-driven playbook content optimizer. - `optimizer.py` orchestrates one optimization run and persists candidates, evaluations, events, and optional successor playbooks. - `scheduler.py` owns deferred optimization scheduling. - `models.py` owns optimizer-local data shapes. - `judge.py`, `rollout.py`, `gepa_adapter.py`, `assistant_webhook.py`, and `scenario_resolver.py` are mature implementation units and intentionally remain at the package root. +## Purpose + + +Generate and compare content candidates while retaining the user/agent playbook adoption rules. + +## Architecture Pattern + + +The scheduler invokes `optimizer.py`, which resolves source scenarios, runs assistant rollouts through the configured backend, judges candidates, and persists run artifacts and eligible successors. + +## Requirements / Problems to Avoid + + Do not introduce a `components/` package without a separate design that proves the dependency direction is clearer after the move. diff --git a/reflexio/server/services/pre_retrieval/README.md b/reflexio/server/services/pre_retrieval/README.md index 0925b6242..241842f67 100644 --- a/reflexio/server/services/pre_retrieval/README.md +++ b/reflexio/server/services/pre_retrieval/README.md @@ -1,9 +1,24 @@ -# pre_retrieval +# /reflexio/server/services/pre_retrieval +Description: Query reformulation and document expansion shared by search and storage indexing. + +## Main Entry Points -Compact search-preparation capability. - `__init__.py` is the public import surface for query reformulation and document expansion. - `_query_reformulator.py` rewrites user queries and can run a caller-provided search function. - `_document_expander.py` enriches stored documents with related terms before indexing. +## Purpose + + +Improve recall by preparing queries before retrieval and enriching documents before indexing. + +## Architecture Pattern + + +Callers import `QueryReformulator` and `DocumentExpander` from `__init__.py`; the two focused implementation files separate query-time and index-time work. + +## Requirements / Problems to Avoid + + This package intentionally does not use `components/`: both implementation files are already focused, and storage adapters import the package-level public surface. diff --git a/reflexio/server/services/shadow_comparison/README.md b/reflexio/server/services/shadow_comparison/README.md index 0badb41c5..2d5e79203 100644 --- a/reflexio/server/services/shadow_comparison/README.md +++ b/reflexio/server/services/shadow_comparison/README.md @@ -1,8 +1,25 @@ -# shadow_comparison +# /reflexio/server/services/shadow_comparison +Description: LLM judging of regular and shadow responses with Reflexio-relative outcomes. -Compact LLM-as-judge capability for Reflexio-vs-shadow response comparison. +## Main Entry Points + +- `dispatcher.py` schedules assistant turns carrying shadow content from publish. +- `worker.py` owns bounded background execution and verdict persistence. - `judge.py` owns prompt rendering and the LLM call for one interaction. - `outcome.py` owns pure position randomization and Reflexio-relative win/loss/tie derivation. -This package intentionally does not use `components/`: the current two-file split already separates LLM orchestration from pure outcome logic. +## Purpose + + +Produce independent comparison verdicts for evaluation reporting. + +## Architecture Pattern + + +`judge.py` renders the prompt and invokes the model; `outcome.py` randomizes answer positions and converts the result back to a Reflexio-relative win/loss/tie. + +## Requirements / Problems to Avoid + + +This package intentionally does not use `components/`: dispatch, worker execution, judging, and pure outcome logic already have focused files. diff --git a/reflexio/server/services/tagging/README.md b/reflexio/server/services/tagging/README.md index 04ddc20d1..f66134ecd 100644 --- a/reflexio/server/services/tagging/README.md +++ b/reflexio/server/services/tagging/README.md @@ -1,6 +1,8 @@ -# tagging +# /reflexio/server/services/tagging +Description: Deferred tagging of profiles, playbooks, and agent-success evaluation summaries. + +## Main Entry Points -Compact post-generation entity tagging capability. - `service.py` tags profiles, playbooks, and persisted agent-success evaluation summaries with their configured tagging prompts. Evaluation tagging receives @@ -8,4 +10,17 @@ Compact post-generation entity tagging capability. - `tagging_scheduler.py` coalesces tagging by organization, user, and agent version, then rebuilds request context for background execution. +## Purpose + + +Attach configured tags after generation so retrieval and evaluation views can use them. + +## Architecture Pattern + + +`tagging_scheduler.py` coalesces organization/user/version work, rebuilds request context, and invokes `service.py` against persisted entities. + +## Requirements / Problems to Avoid + + This package intentionally does not use `components/`: the service and scheduler are the only module responsibilities. diff --git a/reflexio/server/site_var/README.md b/reflexio/server/site_var/README.md index 5b3653441..dae124f89 100644 --- a/reflexio/server/site_var/README.md +++ b/reflexio/server/site_var/README.md @@ -1,16 +1,18 @@ -# Site Variables +# /reflexio/server/site_var Description: Global configuration manager for site-wide variables and per-org feature flags > Part of the [Reflexio Server](../README.md). ## Main Entry Points + - **Manager**: `site_var_manager.py` - `SiteVarManager` (singleton) - **Feature Flags**: `feature_flags.py` - Per-org feature gating helpers - **Sources**: `site_var_sources/` - JSON/TXT config files ## Purpose + 1. **Global settings** - Model names, embedding models, and retrieval defaults 2. **Feature flags** - Per-org feature gating (global enable or per-org allowlist) 3. **Dual storage** - File-based with optional Redis caching @@ -18,6 +20,7 @@ Description: Global configuration manager for site-wide variables and per-org fe ## Feature Flags + **File**: `feature_flags.py` **Config**: `site_var_sources/feature_flags.json` @@ -46,33 +49,42 @@ get_all_feature_flags(org_id) # All flags as dict[str, bool] ## Usage + ```python from reflexio.server.site_var.site_var_manager import SiteVarManager manager = SiteVarManager() -config = manager.get_site_var("app_config") # Returns dict or string +config = manager.get_site_var("llm_model_setting") # Returns dict or string ``` ## File Structure + ``` site_var/ ├── site_var_manager.py # SiteVarManager (singleton) ├── feature_flags.py # Per-org feature flag helpers └── site_var_sources/ - ├── app_config.json # JSON → parsed dict - ├── model_config.json + ├── llm_model_setting.json # Provider/model defaults ├── search_settings.json # Default retrieval mode and hybrid search weights └── feature_flags.json # Feature flag config (per-flag enable + org allowlist) ``` ## Architecture Pattern + - **JSON priority**: `.json` files take precedence over `.txt` - **Variable name**: Filename without extension - **Redis optional**: Enable via `SiteVarManager(enable_redis=True)` - **Feature flags**: Loaded via SiteVarManager, resolved per-org at request time +## Requirements / Problems to Avoid + + +- **Unknown flags are enabled by default**; do not use feature flags as an authorization boundary. +- **Keep source names aligned with lookup keys**; a JSON filename without its extension is the site-variable key. + ## See Also + - [Server README](../README.md) -- FastAPI backend component overview diff --git a/tests/eval/consolidation/README.md b/tests/eval/consolidation/README.md index 81b565e2f..8427b7593 100644 --- a/tests/eval/consolidation/README.md +++ b/tests/eval/consolidation/README.md @@ -1,8 +1,24 @@ -# Consolidator decision-eval harness (AI-judged) +# /tests/eval/consolidation +Description: Decision-quality harness for playbook consolidation. -Scaffolding to measure whether the **consolidator** makes the right decision -for a new playbook fragment against the existing rows, and to catch -regressions when the `playbook_consolidation` prompt changes. +## Main Entry Points + + +- **`case.py`** — case schema and discriminator +- **`runner.py`** — provider execution and metrics +- **`judge.py`** — panel verdicts +- **`providers.py`** — live consolidator adapter +- **`fixtures/`** — illustrative cases + +## Purpose + + +Detect consolidation prompt regressions without confusing illustrative fixtures with curated evidence. + +## Architecture Pattern + + +Cases feed precomputed decisions or a provider; deterministic kind scores and independent judge verdicts are aggregated separately. It is **AI-judged**: no human gold labels are required at scoring time — an LLM-judge labels each case and the headline metric is *agreement with the @@ -13,17 +29,9 @@ judge*. A cheap deterministic *kind accuracy* is tracked alongside. > `fixtures/illustrative_cases.json` exists only to exercise the harness > end-to-end. -## Layout - -| File | Purpose | -| --- | --- | -| `case.py` | `ConsolidationEvalCase` schema (`existing` rows + one `candidate`) + `kind_for_decision` | -| `judge.py` | `judge_consolidation_decision` AI-judge (panel of N, default 1, majority vote) returning `ConsolidationVerdict{correct, self_contradiction, reason}` | -| `runner.py` | `run_eval` runner + `EvalResults` metrics (kind accuracy, judge agreement, over/under-merge, self-contradiction) | -| `fixtures/` | `illustrative_cases.json` + loader (`load_illustrative_cases`) | - ## Kind mapping (explicit, not heuristic) + A consolidation decision carries an explicit discriminator `kind` (`unify` / `reject_new` / `differentiate` / `independent`). So `kind_for_decision(decision)` simply returns @@ -31,6 +39,7 @@ judge*. A cheap deterministic *kind accuracy* is tracked alongside. ## What the judge adds: the self-contradiction dimension + The verdict carries a second flag beyond `correct`: - **`self_contradiction`** — for a produced `unify`, did the merge fold together @@ -47,7 +56,9 @@ For non-`unify` decisions `self_contradiction` is not applicable (recorded as ## Metrics + Deterministic (no LLM): + - `kind_accuracy` — fraction of cases whose produced `kind` equals `gold_kind`. - `over_merge_rate` — of cases that should stay separate (`gold_kind` ∈ {`differentiate`, `independent`}), the fraction that produced a merge/drop @@ -56,6 +67,7 @@ Deterministic (no LLM): `reject_new`}), the fraction kept separate (`differentiate` / `independent`). AI-judged: + - `judge_accuracy` — fraction the judge marked `correct` (None if none judged). - `self_contradiction_rate` — among produced-`unify` judged cases, the fraction the judge flagged as self-contradicting (None when there is no judged @@ -66,6 +78,7 @@ Panel ties break **conservatively, in opposite directions**: `correct` ties → ## Fixture coverage + | Case id | gold_kind | | --- | --- | | `unify_same_trigger_duplicate` | `unify` (classic dedup) | @@ -74,7 +87,11 @@ Panel ties break **conservatively, in opposite directions**: `correct` ties → | `reject_new_redundant` | `reject_new` | | `independent_unrelated` | `independent` | -## Running +## Key Endpoints / Commands / Contracts + + +### Running + Tests mock the LLM judge and the produced decision — no real API calls: @@ -87,6 +104,7 @@ real-*judge* smoke test, `test_real_judge_smoke`). ## Running against the live consolidator + `run_eval` accepts either a parallel `decisions` list (precomputed, as the default tests use) or a `decision_provider` callable that maps a case to a produced decision. To evaluate the **live** consolidator end-to-end, use the @@ -109,6 +127,7 @@ covers the provider's construction in default CI. ## Comparing two prompt versions (deviation guard) + To gate a candidate `playbook_consolidation` version against a baseline (catch regressions when iterating the prompt across versions), use the shared CLI which runs this harness under both pinned versions and fails on regression: @@ -120,3 +139,9 @@ uv run python -m tests.eval.prompt_deviation_guard \ See `tests/eval/prompt_deviation_guard.py` (gate logic unit-tested in `tests/eval/test_prompt_deviation_guard.py`). + +## Requirements / Problems to Avoid + + +- **Keep default tests keyless**; real providers and judges require the documented opt-in. +- **Illustrative fixtures prove harness behavior**, not production extraction quality. diff --git a/tests/eval/extraction/README.md b/tests/eval/extraction/README.md index 9cee93453..f605c51bd 100644 --- a/tests/eval/extraction/README.md +++ b/tests/eval/extraction/README.md @@ -1,7 +1,24 @@ -# Extraction golden-set eval runner (AI-judged) +# /tests/eval/extraction +Description: Float-scored profile/playbook extraction evaluation. -Scaffolding to score the **extractor** (profiles + playbooks) against curated -gold cases, and to catch regressions when the extraction prompts change. +## Main Entry Points + + +- **`runner.py`** — case scoring and aggregation +- **`providers.py`** — real extractor adapter +- **`../judge.py`** — shared LLM judge +- **`../golden_set/extraction/`** — gold cases +- **`../conftest.py`** — case and judge fixtures + +## Purpose + + +Measure signal coverage and grounded evidence for extraction prompt changes. + +## Architecture Pattern + + +Load cases, obtain supplied/provider extractions, score with the shared judge, and aggregate continuous metrics. It is **AI-judged** and **float-scored** — unlike consolidation decision evals (which classify a decision into a discrete @@ -16,6 +33,7 @@ judge → aggregate). ## What it reuses (not re-implemented here) + | Piece | Lives in | Role | | --- | --- | --- | | `LLMJudge.score(*, expected, actual)` | `tests/eval/judge.py` | the shared judge — returns `JudgeScore{signal_f1, answer_correctness, grounded_rate, rationale}` | @@ -28,6 +46,7 @@ This package adds only `runner.py` (`run_eval` / `score_case` / `EvalResults` / ## Metrics + The judge scores each case on two dimensions in `[0, 1]`: - **`signal_f1`** — did the extraction capture the expected *signals*, including @@ -38,6 +57,7 @@ The judge scores each case on two dimensions in `[0, 1]`: (`answer_correctness` is a search-only dimension, pinned to 0 here and ignored.) `EvalResults` aggregates per-case scores into: + - `signal_f1_mean`, `grounded_rate_mean` (arithmetic means; 0.0 when empty), - `pass_rate(threshold=0.7)` — fraction of cases with `signal_f1 >= threshold` **and** `grounded_rate >= threshold`, @@ -46,7 +66,11 @@ The judge scores each case on two dimensions in `[0, 1]`: There is **no kind label or confusion matrix** — extraction is graded, not classified. -## Running +## Key Endpoints / Commands / Contracts + + +### Running + The default `extraction_judge` is stubbed, so the harness runs without credentials: @@ -61,6 +85,7 @@ judge. The `@skip_low_priority` `test_real_judge_smoke` (gated by ## Supplying extractions + `run_eval` takes either a parallel `extractions` list (precomputed `(profiles, playbooks)` per case — the default tests use the case's own gold items as a perfect-extraction baseline) or an `extraction_provider` callable. To @@ -82,3 +107,9 @@ the `@skip_low_priority` smoke `test_live_extraction_provider_real` (run with `litellm.completion`) covers the provider's construction in default CI. Produced items may be `UserProfile` / `UserPlaybook` entities or plain dicts (a `_to_dict` shim normalizes both for the judge). + +## Requirements / Problems to Avoid + + +- **Keep default tests keyless**; real providers and judges require the documented opt-in. +- **Illustrative fixtures prove harness behavior**, not production extraction quality. diff --git a/tests/eval/golden_set/search/README.md b/tests/eval/golden_set/search/README.md index 4f6b75953..9dc545333 100644 --- a/tests/eval/golden_set/search/README.md +++ b/tests/eval/golden_set/search/README.md @@ -1,10 +1,29 @@ -# Search golden set +# /tests/eval/golden_set/search +Description: YAML search cases with explicit gold candidates and relative-time fixtures. + +## Main Entry Points + + +- **`../../conftest.py`** — case loading +- **`../../search/runner.py`** — case execution and scoring +- **`../../search/providers.py`** — search and seeding adapter + +## Purpose + + +Encode retrieval edge cases without aging out fixed calendar timestamps. + +## Architecture Pattern + + +The provider maps case-local keys to stored identities, resolves relative ages at seed time, and drives unified search; the runner compares returned candidates against gold labels. Each YAML file is one search eval case, loaded by `tests/eval/conftest.py` (`_load("search")`) and driven by `tests/eval/search/runner.py`. ## Case schema + ```yaml id: # required, unique; also the pytest param id category: preference | supersession | temporal_window | temporal_current | recall @@ -56,6 +75,7 @@ requires both the stale and fresh items to be live. ## Categories + - `recall` — direct lexical/semantic match; a sanity floor every backend should pass. - `preference` — disambiguation / bridging (e.g. "DB" vs a dataframe lib). diff --git a/tests/eval/playbook_ask_human/README.md b/tests/eval/playbook_ask_human/README.md index 88c686808..a60edbbff 100644 --- a/tests/eval/playbook_ask_human/README.md +++ b/tests/eval/playbook_ask_human/README.md @@ -1,4 +1,24 @@ -# Playbook ask_human invocation eval +# /tests/eval/playbook_ask_human +Description: Binary evaluation of playbook human-clarification decisions. + +## Main Entry Points + + +- **`case.py`** — label schema +- **`providers.py`** — extractor invocation +- **`runner.py`** — precision/recall scoring +- **`run_benchmark.py`** — benchmark CLI +- **`../golden_set/playbook_ask_human/cases.yaml`** — golden trajectories + +## Purpose + + +Distinguish missing organizational context from extractable learning and trajectories requiring no playbook. + +## Architecture Pattern + + +The provider runs the resumable extractor; the runner compares whether it invoked ask_human against the case label. This eval scores the resumable playbook extractor's decision to call `ask_human`. It is a binary precision/recall eval over natural agent-user @@ -24,4 +44,3 @@ uv run pytest tests/eval/playbook_ask_human -o 'addopts=' -q uv run ruff check tests/eval/playbook_ask_human uv run pyright tests/eval/playbook_ask_human ``` - diff --git a/tests/eval/scenarios/README.md b/tests/eval/scenarios/README.md index 3e67fcefd..f8ac9dd9d 100644 --- a/tests/eval/scenarios/README.md +++ b/tests/eval/scenarios/README.md @@ -1,13 +1,23 @@ -# Multi-round learning-scenario harness (AI-judged) +# /tests/eval/scenarios +Description: Controlled multi-round extraction/consolidation evaluation. -This lightweight in-process harness measures how the extractor and playbook -consolidator evolve a playbook over controlled learning rounds. +## Main Entry Points -Each round feeds interactions to the extraction provider, routes produced -playbooks through the consolidation provider against the accumulated in-memory -book, judges the decision, and applies it through the test-only shim in -`book.py`. An optional end-state judge compares the final book with the -scenario's expected outcome. + +- **`case.py`** — scenario model +- **`runner.py`** — round orchestration +- **`book.py`** — test-only state application +- **`fixtures/scenarios.json`** — illustrative scenarios + +## Purpose + + +Inspect how learning evolves over multiple rounds and detect contradictory consolidation. + +## Architecture Pattern + + +Each round extracts, consolidates against the accumulated in-memory book, judges the decision, and applies it through the test shim. The fixtures cover: