From 446d575cb467c81b212f66c7baaf45daef70cbb9 Mon Sep 17 00:00:00 2001 From: Ashwani Kharwar Date: Mon, 14 Sep 2026 01:43:10 +0530 Subject: [PATCH 01/13] feat: add source-backed feature workflow explorer - Generate and persist feature/workflow manifests during tldrgraph init - Make AI enrichment and route-link inference explicit opt-ins - Load saved workflows in the visualizer with pending/ready states - Add workflow generation, loading, and visualizer test coverage --- .agents/skills/tldrgraph-init/SKILL.md | 164 +++--------- .claude/commands/tldrgraph-init.md | 164 +++--------- .cursor/commands/tldrgraph-init.md | 132 +++------- .tldrgraph/AGENT_CONTRACT.md | 87 ++++--- AGENTS.md | 40 +-- AGENT_CONTRACT.md | 88 ++++--- tests/test_agent_loop.py | 7 +- tests/test_auto_agent.py | 127 ++++++---- tests/test_dynamic_layers.py | 48 +--- tests/test_feature_workflows.py | 185 ++++++++++++++ tests/test_visualizer.py | 188 ++++++-------- tldrgraph/agent_commands.py | 122 +++------ tldrgraph/cli.py | 9 +- tldrgraph/cli_pipeline.py | 79 +++--- tldrgraph/feature_workflow_agent.py | 70 ++++++ tldrgraph/feature_workflow_loader.py | 177 +++++++++++++ tldrgraph/feature_workflows.py | 330 +++++++++++++++++++++++++ tldrgraph/installer_contract.py | 29 ++- tldrgraph/propose_layers.py | 6 +- tldrgraph/visualizer/assets/app.js | 36 ++- tldrgraph/visualizer/data.py | 9 +- 21 files changed, 1293 insertions(+), 804 deletions(-) create mode 100644 tests/test_feature_workflows.py create mode 100644 tldrgraph/feature_workflow_agent.py create mode 100644 tldrgraph/feature_workflow_loader.py create mode 100644 tldrgraph/feature_workflows.py diff --git a/.agents/skills/tldrgraph-init/SKILL.md b/.agents/skills/tldrgraph-init/SKILL.md index e7e58cf..2240a04 100644 --- a/.agents/skills/tldrgraph-init/SKILL.md +++ b/.agents/skills/tldrgraph-init/SKILL.md @@ -5,149 +5,56 @@ description: Build or continue this repository's TLDRGraph architecture graph (l # TLDRGraph: build this repository's architecture graph -In Claude Code or Cursor, invoke `/tldrgraph-init`. In Codex CLI, type `/skills` -and select `tldrgraph-init`, or mention `$tldrgraph-init` directly. +In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select +`tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles layer design, extraction, enrichment, LLM route links, and embeddings: +One command handles extraction, feature-file scaffolding, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph detects `claude`, `cursor-agent`, or `gemini`, processes every -eligible node in batches of 200, and downloads/builds the local embedding model. -In a detected coding-agent session, plain `tldrgraph init` auto-approves the full -enrichment campaign; normal terminal users and non-agent automation still get the -confirmation gate. +By default TLDRGraph does not ask another AI process to design architecture +layers, enrich every symbol, infer route links, or generate BPMN workflows. It +writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` +file per feature. The coding agent that ran `tldrgraph init` owns completing any +pending workflow files from source evidence. Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent `needs_confirmation` response explicitly asks for approval. `--batch 200` means -all nodes in 200-node batches; `--limit 200` means stop after only 200 nodes. -Never add `--limit` or `--embeddings off` unless the user explicitly requests a -partial or no-embedding run. +all nodes in chunks; `--limit 200` means stop after only 200 nodes. +Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless the user +explicitly requests it. -## Completion contract for agents +## Feature Workflow Explorer artifacts -Once a full enrichment campaign is approved or auto-approved, you are not done until one of -these terminal states occurs: +Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +Workflow Explorer reads only those files; missing or invalid workflows show pending states. +Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. +Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +When `init` reports pending feature workflows: -- `tldrgraph init` reports `status: done`. -- Dense embeddings finish, or `init` explicitly reports that embeddings are - unavailable while preserving a queryable graph. -- A real blocking error occurs that requires user action, and you report the - exact command/output that blocked continuation. - -These are **not** terminal states: - -- `status: needs_enrichment` -- `status: needs_llm_links` -- `NEXT ACTION` -- "nodes remaining" -- "batches remaining" -- "applied N enrichment entries" - -For every non-terminal enrichment state, immediately continue the loop: - -1. Read `.tldrgraph/enrichment_request.yaml`. -2. Open the source file for every requested node. -3. Write `.tldrgraph/enrichment_response.yaml`. -4. Run `tldrgraph init` again. -5. Repeat until a terminal state occurs. - -Do not end the task with a progress-only summary such as "I applied 400 entries -and 1,438 remain." That is an incomplete run, not a final answer. If nested-agent -protection prevents a CLI from launching another agent, you are the enrichment -agent and must process the batch yourself. - -If no supported agent is available or dense embeddings cannot be built, `init` -preserves the graph and prints a resumable status. It never guesses source intent, -route links, or architectural layers. +1. Open `.tldrgraph/features.yaml`. +2. Open every pending `.tldrgraph/workflows/.yaml`. +3. Open the source files referenced in `evidence_nodes`; do not guess. +4. Replace `steps: []` with source-backed steps and set `status: generated`. +5. Run `tldrgraph init` again. ## `status: needs_layers` -TLDRGraph ships **no layer templates** and will not invent an architecture. -Design one from this repository. - -1. Read `.tldrgraph/propose_layers_request.json`. It carries the symbols and - files extraction already found -- a starting point, not a substitute for - opening the code. -2. **Open real source files**: entry points first, then a representative file - from each cluster in the evidence. Work out what this codebase actually does - and where responsibility changes hands. -3. Write `.tldrgraph/propose_layers_response.json`: - -```json -{ - "utility_id": "", - "layers": [ - { - "id": "short_machine_id", - "name": "Layer 1: Human Friendly Name", - "order": 1, - "description": "One sentence on what lives here", - "rules": [ - {"file_contains": ["substring"], "exclude_file": ["optional"]}, - {"label_contains": ["SymbolNamePart"]} - ] - } - ] -} -``` - -4. Run `tldrgraph init` again; it continues with enrichment and embeddings. - -### Rules that hold for any answer - -- 3 to 6 layers, plus exactly one catch-all whose `id` equals `utility_id` and - whose `rules` are `[]`. -- Unique `id` and `name` per layer; sequential integer `order` from 1. -- Rule keys: `file_contains`, `exclude_file`, `path_regex`, `label_contains`, - `exclude_label`, `label_ends_with`, `type_in`, `id_prefix`. Values are lists of - strings. Rules are evaluated in `order` and the first match wins. -- Derive rules from paths and symbol names you actually saw. A rule matching - nothing is worse than no rule; a rule matching everything collapses the map. +This should only appear when the user explicitly opted into architecture AI with +`--agent-cli` or an older TLDRGraph build is running. Do not complete this +handoff unless the user asked for architecture layer design. ## `status: needs_confirmation` -Detected coding-agent sessions should not reach this state for a full run. If a -non-agent run does, the output shows how many nodes need enrichment and how many -agent round-trips that implies; ask the user whether to proceed. - -- They agree: `tldrgraph init --yes` saves approval for the full campaign -- Smaller first pass: `tldrgraph init --yes --limit 100` -- They decline: stop. The graph is already built and queryable. +This belongs to explicit `--agent-cli` enrichment. Ask before continuing. ## `status: needs_enrichment` -1. Read `.tldrgraph/enrichment_request.yaml`. -2. **Open the source file of every node in it.** This is the entire point: an - intent paraphrased from a symbol name poisons semantic search with - confident-sounding noise. -3. Write `.tldrgraph/enrichment_response.yaml` -- a *different* file from the - request, which is regenerated on every run: - -```yaml -- id: "" - intent: | - What this symbol does, why it exists, and its execution logic. - input_fields: [caseId, remarks] - output_fields: [status, disposition] - calls: [ApplicationsService, pension_cases] -``` - -Every `intent` must contain **2-3 complete sentences** covering what the symbol does, -why it exists, and its source-backed behavior. Markdown headings and list markers do not -count as sentences. - -4. Run `tldrgraph init` again. Approval is saved; process any next - `needs_enrichment` batch immediately without asking the user again until - init either reports `status: done` or advances to the required `needs_llm_links` - phase. - -Inside an existing Codex/Claude/Cursor session, nested-agent protection may stop -the CLI from launching a second agent. In that case **you are the enrichment -agent**: process every 200-node batch yourself. A `needs_enrichment` status is a -continuation instruction, not a reason to stop or request confirmation. +This should only appear when the user explicitly opted into architecture +enrichment with `--agent-cli` or an older TLDRGraph build is running. Do not +process enrichment batches unless the user asked for full graph enrichment. **Copy every `id` verbatim.** A constructed id matches nothing, is dropped, and gets reported back to you -- but the work is wasted. @@ -157,13 +64,12 @@ empty list is a correct answer, a wrong `calls` entry becomes a real wrong edge. ## `status: needs_llm_links` -1. Read `.tldrgraph/llm_links_request.yaml`. -2. Open the referenced frontend and backend source files. -3. Write `.tldrgraph/llm_links_response.yaml` as a YAML list of - `{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. - Only include source-backed links with file and line evidence. -4. Run `tldrgraph init` again. This is a required continuation state for a - complete init run unless the user explicitly requested `--no-llm-links`. +Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend +files, then write `.tldrgraph/llm_links_response.yaml` as a YAML list of +`{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. +Only include source-backed links with file and line evidence. Run `tldrgraph init` +again. This continuation state only appears when the user explicitly opted into +route-link inference with `--llm-links`. ## Once it says DONE diff --git a/.claude/commands/tldrgraph-init.md b/.claude/commands/tldrgraph-init.md index e7e58cf..2240a04 100644 --- a/.claude/commands/tldrgraph-init.md +++ b/.claude/commands/tldrgraph-init.md @@ -5,149 +5,56 @@ description: Build or continue this repository's TLDRGraph architecture graph (l # TLDRGraph: build this repository's architecture graph -In Claude Code or Cursor, invoke `/tldrgraph-init`. In Codex CLI, type `/skills` -and select `tldrgraph-init`, or mention `$tldrgraph-init` directly. +In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select +`tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles layer design, extraction, enrichment, LLM route links, and embeddings: +One command handles extraction, feature-file scaffolding, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph detects `claude`, `cursor-agent`, or `gemini`, processes every -eligible node in batches of 200, and downloads/builds the local embedding model. -In a detected coding-agent session, plain `tldrgraph init` auto-approves the full -enrichment campaign; normal terminal users and non-agent automation still get the -confirmation gate. +By default TLDRGraph does not ask another AI process to design architecture +layers, enrich every symbol, infer route links, or generate BPMN workflows. It +writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` +file per feature. The coding agent that ran `tldrgraph init` owns completing any +pending workflow files from source evidence. Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent `needs_confirmation` response explicitly asks for approval. `--batch 200` means -all nodes in 200-node batches; `--limit 200` means stop after only 200 nodes. -Never add `--limit` or `--embeddings off` unless the user explicitly requests a -partial or no-embedding run. +all nodes in chunks; `--limit 200` means stop after only 200 nodes. +Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless the user +explicitly requests it. -## Completion contract for agents +## Feature Workflow Explorer artifacts -Once a full enrichment campaign is approved or auto-approved, you are not done until one of -these terminal states occurs: +Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +Workflow Explorer reads only those files; missing or invalid workflows show pending states. +Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. +Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +When `init` reports pending feature workflows: -- `tldrgraph init` reports `status: done`. -- Dense embeddings finish, or `init` explicitly reports that embeddings are - unavailable while preserving a queryable graph. -- A real blocking error occurs that requires user action, and you report the - exact command/output that blocked continuation. - -These are **not** terminal states: - -- `status: needs_enrichment` -- `status: needs_llm_links` -- `NEXT ACTION` -- "nodes remaining" -- "batches remaining" -- "applied N enrichment entries" - -For every non-terminal enrichment state, immediately continue the loop: - -1. Read `.tldrgraph/enrichment_request.yaml`. -2. Open the source file for every requested node. -3. Write `.tldrgraph/enrichment_response.yaml`. -4. Run `tldrgraph init` again. -5. Repeat until a terminal state occurs. - -Do not end the task with a progress-only summary such as "I applied 400 entries -and 1,438 remain." That is an incomplete run, not a final answer. If nested-agent -protection prevents a CLI from launching another agent, you are the enrichment -agent and must process the batch yourself. - -If no supported agent is available or dense embeddings cannot be built, `init` -preserves the graph and prints a resumable status. It never guesses source intent, -route links, or architectural layers. +1. Open `.tldrgraph/features.yaml`. +2. Open every pending `.tldrgraph/workflows/.yaml`. +3. Open the source files referenced in `evidence_nodes`; do not guess. +4. Replace `steps: []` with source-backed steps and set `status: generated`. +5. Run `tldrgraph init` again. ## `status: needs_layers` -TLDRGraph ships **no layer templates** and will not invent an architecture. -Design one from this repository. - -1. Read `.tldrgraph/propose_layers_request.json`. It carries the symbols and - files extraction already found -- a starting point, not a substitute for - opening the code. -2. **Open real source files**: entry points first, then a representative file - from each cluster in the evidence. Work out what this codebase actually does - and where responsibility changes hands. -3. Write `.tldrgraph/propose_layers_response.json`: - -```json -{ - "utility_id": "", - "layers": [ - { - "id": "short_machine_id", - "name": "Layer 1: Human Friendly Name", - "order": 1, - "description": "One sentence on what lives here", - "rules": [ - {"file_contains": ["substring"], "exclude_file": ["optional"]}, - {"label_contains": ["SymbolNamePart"]} - ] - } - ] -} -``` - -4. Run `tldrgraph init` again; it continues with enrichment and embeddings. - -### Rules that hold for any answer - -- 3 to 6 layers, plus exactly one catch-all whose `id` equals `utility_id` and - whose `rules` are `[]`. -- Unique `id` and `name` per layer; sequential integer `order` from 1. -- Rule keys: `file_contains`, `exclude_file`, `path_regex`, `label_contains`, - `exclude_label`, `label_ends_with`, `type_in`, `id_prefix`. Values are lists of - strings. Rules are evaluated in `order` and the first match wins. -- Derive rules from paths and symbol names you actually saw. A rule matching - nothing is worse than no rule; a rule matching everything collapses the map. +This should only appear when the user explicitly opted into architecture AI with +`--agent-cli` or an older TLDRGraph build is running. Do not complete this +handoff unless the user asked for architecture layer design. ## `status: needs_confirmation` -Detected coding-agent sessions should not reach this state for a full run. If a -non-agent run does, the output shows how many nodes need enrichment and how many -agent round-trips that implies; ask the user whether to proceed. - -- They agree: `tldrgraph init --yes` saves approval for the full campaign -- Smaller first pass: `tldrgraph init --yes --limit 100` -- They decline: stop. The graph is already built and queryable. +This belongs to explicit `--agent-cli` enrichment. Ask before continuing. ## `status: needs_enrichment` -1. Read `.tldrgraph/enrichment_request.yaml`. -2. **Open the source file of every node in it.** This is the entire point: an - intent paraphrased from a symbol name poisons semantic search with - confident-sounding noise. -3. Write `.tldrgraph/enrichment_response.yaml` -- a *different* file from the - request, which is regenerated on every run: - -```yaml -- id: "" - intent: | - What this symbol does, why it exists, and its execution logic. - input_fields: [caseId, remarks] - output_fields: [status, disposition] - calls: [ApplicationsService, pension_cases] -``` - -Every `intent` must contain **2-3 complete sentences** covering what the symbol does, -why it exists, and its source-backed behavior. Markdown headings and list markers do not -count as sentences. - -4. Run `tldrgraph init` again. Approval is saved; process any next - `needs_enrichment` batch immediately without asking the user again until - init either reports `status: done` or advances to the required `needs_llm_links` - phase. - -Inside an existing Codex/Claude/Cursor session, nested-agent protection may stop -the CLI from launching a second agent. In that case **you are the enrichment -agent**: process every 200-node batch yourself. A `needs_enrichment` status is a -continuation instruction, not a reason to stop or request confirmation. +This should only appear when the user explicitly opted into architecture +enrichment with `--agent-cli` or an older TLDRGraph build is running. Do not +process enrichment batches unless the user asked for full graph enrichment. **Copy every `id` verbatim.** A constructed id matches nothing, is dropped, and gets reported back to you -- but the work is wasted. @@ -157,13 +64,12 @@ empty list is a correct answer, a wrong `calls` entry becomes a real wrong edge. ## `status: needs_llm_links` -1. Read `.tldrgraph/llm_links_request.yaml`. -2. Open the referenced frontend and backend source files. -3. Write `.tldrgraph/llm_links_response.yaml` as a YAML list of - `{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. - Only include source-backed links with file and line evidence. -4. Run `tldrgraph init` again. This is a required continuation state for a - complete init run unless the user explicitly requested `--no-llm-links`. +Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend +files, then write `.tldrgraph/llm_links_response.yaml` as a YAML list of +`{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. +Only include source-backed links with file and line evidence. Run `tldrgraph init` +again. This continuation state only appears when the user explicitly opted into +route-link inference with `--llm-links`. ## Once it says DONE diff --git a/.cursor/commands/tldrgraph-init.md b/.cursor/commands/tldrgraph-init.md index 9fa8c56..2240a04 100644 --- a/.cursor/commands/tldrgraph-init.md +++ b/.cursor/commands/tldrgraph-init.md @@ -5,115 +5,56 @@ description: Build or continue this repository's TLDRGraph architecture graph (l # TLDRGraph: build this repository's architecture graph -In Claude Code or Cursor, invoke `/tldrgraph-init`. In Codex CLI, type `/skills` -and select `tldrgraph-init`, or mention `$tldrgraph-init` directly. +In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select +`tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles layer design, extraction, enrichment, and embeddings: +One command handles extraction, feature-file scaffolding, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph detects `claude`, `cursor-agent`, or `gemini`, asks once before enrichment token spend, processes every -eligible node in batches of 200, and downloads/builds the local embedding model. -Use `--yes` for non-interactive approval, `--batch N` to override the batch size, -`--embeddings off|auto|on` to override embeddings, or `--no-agent-cli` for the -manual file handoff. +By default TLDRGraph does not ask another AI process to design architecture +layers, enrich every symbol, infer route links, or generate BPMN workflows. It +writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` +file per feature. The coding agent that ran `tldrgraph init` owns completing any +pending workflow files from source evidence. -After the user approves the full run, use exactly `tldrgraph init --yes`. The -approval is saved for the current candidate set, so later `tldrgraph init` calls -must continue without asking again. `--batch 200` means all nodes in 200-node -batches; `--limit 200` means stop after only 200 nodes. Never add `--limit` or -`--embeddings off` unless the user explicitly requests a partial or no-embedding run. +Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent +`needs_confirmation` response explicitly asks for approval. `--batch 200` means +all nodes in chunks; `--limit 200` means stop after only 200 nodes. +Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless the user +explicitly requests it. -If no supported agent is available or dense embeddings cannot be built, `init` -preserves the graph and prints a resumable status. It never guesses source intent -or architectural layers. +## Feature Workflow Explorer artifacts -## `status: needs_layers` - -TLDRGraph ships **no layer templates** and will not invent an architecture. -Design one from this repository. - -1. Read `.tldrgraph/propose_layers_request.json`. It carries the symbols and - files extraction already found -- a starting point, not a substitute for - opening the code. -2. **Open real source files**: entry points first, then a representative file - from each cluster in the evidence. Work out what this codebase actually does - and where responsibility changes hands. -3. Write `.tldrgraph/propose_layers_response.json`: - -```json -{ - "utility_id": "", - "layers": [ - { - "id": "short_machine_id", - "name": "Layer 1: Human Friendly Name", - "order": 1, - "description": "One sentence on what lives here", - "rules": [ - {"file_contains": ["substring"], "exclude_file": ["optional"]}, - {"label_contains": ["SymbolNamePart"]} - ] - } - ] -} -``` +Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +Workflow Explorer reads only those files; missing or invalid workflows show pending states. +Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. +Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +When `init` reports pending feature workflows: -4. Run `tldrgraph init` again; it continues with enrichment and embeddings. +1. Open `.tldrgraph/features.yaml`. +2. Open every pending `.tldrgraph/workflows/.yaml`. +3. Open the source files referenced in `evidence_nodes`; do not guess. +4. Replace `steps: []` with source-backed steps and set `status: generated`. +5. Run `tldrgraph init` again. -### Rules that hold for any answer +## `status: needs_layers` -- 3 to 6 layers, plus exactly one catch-all whose `id` equals `utility_id` and - whose `rules` are `[]`. -- Unique `id` and `name` per layer; sequential integer `order` from 1. -- Rule keys: `file_contains`, `exclude_file`, `path_regex`, `label_contains`, - `exclude_label`, `label_ends_with`, `type_in`, `id_prefix`. Values are lists of - strings. Rules are evaluated in `order` and the first match wins. -- Derive rules from paths and symbol names you actually saw. A rule matching - nothing is worse than no rule; a rule matching everything collapses the map. +This should only appear when the user explicitly opted into architecture AI with +`--agent-cli` or an older TLDRGraph build is running. Do not complete this +handoff unless the user asked for architecture layer design. ## `status: needs_confirmation` -The output shows how many nodes need enrichment and how many agent round-trips -that implies. **Ask the user whether to proceed, and show them that estimate.** -Do not decide for them. - -- They agree: `tldrgraph init --yes` saves approval for the full campaign -- Smaller first pass: `tldrgraph init --yes --limit 100` -- They decline: stop. The graph is already built and queryable. +This belongs to explicit `--agent-cli` enrichment. Ask before continuing. ## `status: needs_enrichment` -1. Read `.tldrgraph/enrichment_request.yaml`. -2. **Open the source file of every node in it.** This is the entire point: an - intent paraphrased from a symbol name poisons semantic search with - confident-sounding noise. -3. Write `.tldrgraph/enrichment_response.yaml` -- a *different* file from the - request, which is regenerated on every run: - -```yaml -- id: "" - intent: | - What this symbol does, why it exists, and its execution logic. - input_fields: [caseId, remarks] - output_fields: [status, disposition] - calls: [ApplicationsService, pension_cases] -``` - -Every `intent` must contain **2-3 complete sentences** covering what the symbol does, -why it exists, and its source-backed behavior. Markdown headings and list markers do not -count as sentences. - -4. Run `tldrgraph init` again. Approval is already saved. If another - `needs_enrichment` batch appears, process it immediately and repeat this loop - without asking the user again. Continue until `status: done`. - -Inside an existing Codex/Claude/Cursor session, nested-agent protection may stop -the CLI from launching a second agent. In that case **you are the enrichment -agent**: process every 200-node batch yourself. A `needs_enrichment` status is a -continuation instruction, not a reason to stop or request confirmation. +This should only appear when the user explicitly opted into architecture +enrichment with `--agent-cli` or an older TLDRGraph build is running. Do not +process enrichment batches unless the user asked for full graph enrichment. **Copy every `id` verbatim.** A constructed id matches nothing, is dropped, and gets reported back to you -- but the work is wasted. @@ -121,6 +62,15 @@ gets reported back to you -- but the work is wasted. **Never invent `fields` or `calls`.** Omit what you cannot verify in the code: an empty list is a correct answer, a wrong `calls` entry becomes a real wrong edge. +## `status: needs_llm_links` + +Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend +files, then write `.tldrgraph/llm_links_response.yaml` as a YAML list of +`{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. +Only include source-backed links with file and line evidence. Run `tldrgraph init` +again. This continuation state only appears when the user explicitly opted into +route-link inference with `--llm-links`. + ## Once it says DONE ```bash diff --git a/.tldrgraph/AGENT_CONTRACT.md b/.tldrgraph/AGENT_CONTRACT.md index b2655be..ee4ccde 100644 --- a/.tldrgraph/AGENT_CONTRACT.md +++ b/.tldrgraph/AGENT_CONTRACT.md @@ -2,37 +2,31 @@ **Audience: the coding agent with this repository open** (Codex, Claude Code, Cursor, Antigravity). -TLDRGraph builds an architectural graph from the graphify AST export. The layer set -itself is designed by you, reading this repository. TLDRGraph ships no layer templates. -Structure *within* a layer, and the high-volume deterministic seams between layers, are -extracted automatically. What cannot be extracted automatically is: - -- indirect dispatch, queue / event hops, dynamically-built routes; -- the natural-language **intent** that makes semantic search work at all. - -That is your job. You are not a fallback for a hosted model — you are the primary -enrichment path, because **you can open the files**. The API path only ever sees a label -and a path (`snippet` is never populated), so it guesses. You do not have to guess. +TLDRGraph builds an architectural graph from the graphify AST export. By default, +`tldrgraph init` does not ask AI to design architecture layers, enrich every +symbol, infer route links, or generate BPMN workflows. The only default AI-shaped +artifact is the saved Feature Workflow Explorer YAML, and it must stay backed by +real source evidence. --- ## Start here: `tldrgraph init` -One command handles layers, extraction, enrichment, and embeddings: +One command handles extraction, saved feature workflows, and embeddings: ```bash tldrgraph init ``` -It asks once before enrichment token spend. Full approval is persisted for the current -candidate set until enrichment finishes, so continuation runs must not ask again. By -default it uses 200-node batches and builds dense embeddings. +Use `--agent-cli` only when the user explicitly asks for AI-assisted architecture +layer design and all-symbol enrichment. Saved feature workflow generation is +independent and does not require `--agent-cli`. | status | what it wants | | --- | --- | -| `needs_layers` | Read the code and design this repository's architecture. **TLDRGraph ships no layer templates**; nothing will be applied for you. The request carries sketches of how other kinds of codebase divide — for shape only, never to copy. | -| `needs_confirmation` | Show the estimate and ask once. Approval via `tldrgraph init --yes` persists until the current campaign is done. | -| `needs_enrichment` | Open, read, and describe this batch, then continue immediately without asking again. | +| `needs_layers` | Only process if the user explicitly opted into architecture AI with `--agent-cli`. | +| `needs_confirmation` | Only belongs to explicit `--agent-cli` enrichment. Ask before continuing. | +| `needs_enrichment` | Only process if the user explicitly asked for full graph enrichment. | | `needs_embeddings` | Enrichment finished but the required dense model/index could not be built. Fix model access and rerun init. | | `done` | Nothing left. Use `query` / `trace` / `layers`. | @@ -45,22 +39,13 @@ dropped, and will be reported back to you — but the work is wasted. --- -## The loop +## Explicit Enrichment Loop -```bash -tldrgraph init --yes # 1. approve every current candidate; writes a 200-node request -# 2. read every requested source and write enrichment_response.yaml -tldrgraph init # 3. applies it and emits the next batch; approval is remembered -# 4. repeat steps 2-3 without asking until status: done -``` - -Inside an existing coding-agent session, nested-agent protection can prevent the CLI -from launching another agent. In that case **you are the enrichment agent** and must -process every batch yourself. Do not stop at `needs_enrichment`. - -`--batch 200` means process all candidates in chunks of 200. `--limit 200` means stop -after only 200 candidates and is only for an explicitly requested partial run. Never add -`--limit` or `--embeddings off` unless the user explicitly requests that behavior. +The legacy enrichment files below are for explicit `--agent-cli`, +`queue-enrichment`, or `apply-enrichment` work. Do not process them as part of +default Feature Workflow Explorer generation. Never add `--limit`, +`--agent-cli`, `--llm-links`, or `--embeddings off` unless the user explicitly +requests that behavior. Request and response are **separate files**. Never write your answer back into `enrichment_request.yaml`; it is regenerated on every run and your work would be lost. @@ -75,6 +60,42 @@ Request and response are **separate files**. Never write your answer back into --- +## Feature Workflow Explorer artifacts + +`tldrgraph init` also creates the saved workflow artifacts used by the visualizer: + +| File | Written by | Read by | +| --- | --- | --- | +| `.tldrgraph/features.yaml` | `tldrgraph init` | Workflow Explorer | +| `.tldrgraph/workflows/.yaml` | `tldrgraph init` | Workflow Explorer | + +The Workflow Explorer tab is intentionally file-driven. It must read only +`.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`; if those +files are missing, invalid, incomplete, or a feature has no generated workflow +yet, show an explicit empty or pending state. + +Do **not** reintroduce Workflow Explorer fallback discovery through +`discover_workflows()`, curated workflow blueprints, route-link workflow +discovery, `llm_http_route_link`, `http_route_link`, `calls_endpoint`, or +BPMN-derived workflow generation. Graph views elsewhere may still show route +links or BPMN data, but saved feature workflows must remain independent. + +Every saved workflow step must be simple enough for non-technical users and vibe +coders, and each step must carry source evidence: `node_id`, symbol, file, and +line/range. If the evidence is absent, mark the workflow pending instead of +guessing. + +A saved feature workflow should describe the complete flow when evidence exists: +the exact user button/menu/form action, event handler, validation, client +request code, request payload construction, API route/controller, +middleware/auth, service or use-case logic, persistence/database, background +job, external system, response payload creation, client response parsing, state +update, navigation/toast/rendered result, and visible success or error handling. +Do not stop at only the frontend or only the backend when the source proves the +handoff, and do not collapse multiple proven source hops into one vague step. + +--- + ## Request schema (`enrichment_request.yaml`) ```yaml diff --git a/AGENTS.md b/AGENTS.md index 5fdcbf7..fc74970 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,34 +23,18 @@ tldrgraph dead-code # review candidates, never a dele 2. Use `view_file` on the target file path returned by TLDRGraph to inspect the code. Those are read-only and never trigger enrichment. - -**To build or refresh the graph**, run `tldrgraph init`. In a detected coding-agent -session, plain `tldrgraph init` auto-approves the full enrichment campaign; normal -terminal users and non-agent automation still get confirmation. It handles layer -design, extraction, source-aware enrichment in 200-node batches, LLM route links, -and dense embeddings. If it prints a `NEXT ACTION` fallback, follow that handoff. - -### ✅ TLDRGraph Init Completion Contract -After a full TLDRGraph enrichment run is approved or auto-approved, the agent MUST -keep working until `tldrgraph init` reports `status: done` or a genuine blocking -error requires user action. - -- `needs_enrichment`, "nodes remaining", "batches remaining", and "NEXT ACTION" - are continuation states, not completion states. -- Do not end with a progress-only summary such as "I enriched 400 nodes and - 1,438 remain" unless a blocking error prevents continuation. -- After each enrichment response is written, immediately run `tldrgraph init` - again and repeat the loop. -- If `status: needs_llm_links` appears, read `.tldrgraph/llm_links_request.yaml`, - open the referenced frontend/backend source files, and write - `.tldrgraph/llm_links_response.yaml` with evidence-backed - `{source, target, confidence, frontend_evidence, backend_evidence, explanation}` - entries before running `tldrgraph init` again. -- If nested-agent protection prevents launching another CLI agent, the current - agent is responsible for processing the batch manually. -- Final responses may summarize progress only after `status: done`, embeddings - are completed or explicitly unavailable, or the exact blocker is reported. - +**To build or refresh the graph**, run `tldrgraph init`. It handles layer setup, +extraction, embeddings, and writes the file-backed Feature Workflow Explorer +artifacts: `.tldrgraph/features.yaml` plus `.tldrgraph/workflows/.yaml`. + +Feature workflows are owned by the agent running `tldrgraph init`. If a workflow +file is pending, open `.tldrgraph/features.yaml`, then complete each pending +`.tldrgraph/workflows/.yaml` from its evidence. Workflow Explorer +reads only saved YAML, never curated blueprints, route-link workflow discovery, +BPMN-derived generation, `discover_workflows()`, `llm_http_route_link`, +`http_route_link`, or `calls_endpoint`. When writing feature workflows, start at +the user's button/menu/form action and continue through client request, backend +work, response payload, client handling, and final UI update. Do not skip proven steps. Full workflow: `.claude/commands/tldrgraph-init.md` (identical copies live in every other agent directory). Schema: `.tldrgraph/AGENT_CONTRACT.md`. diff --git a/AGENT_CONTRACT.md b/AGENT_CONTRACT.md index 2c1aac6..ee4ccde 100644 --- a/AGENT_CONTRACT.md +++ b/AGENT_CONTRACT.md @@ -2,38 +2,31 @@ **Audience: the coding agent with this repository open** (Codex, Claude Code, Cursor, Antigravity). -TLDRGraph builds an architectural graph from the graphify AST export. The layer set -itself is designed by you, reading this repository. TLDRGraph ships no layer templates. -Structure *within* a layer, and the high-volume deterministic seams between layers, are -extracted automatically. What cannot be extracted automatically is: - -- indirect dispatch, queue / event hops, dynamically-built routes; -- the natural-language **intent** that makes semantic search work at all. - -That is your job. You are not a fallback for a hosted model — you are the primary -enrichment path, because **you can open the files**. The API path only ever sees a label -and a path (`snippet` is never populated), so it guesses. You do not have to guess. +TLDRGraph builds an architectural graph from the graphify AST export. By default, +`tldrgraph init` does not ask AI to design architecture layers, enrich every +symbol, infer route links, or generate BPMN workflows. The only default AI-shaped +artifact is the saved Feature Workflow Explorer YAML, and it must stay backed by +real source evidence. --- ## Start here: `tldrgraph init` -One command handles layers, extraction, enrichment, and embeddings: +One command handles extraction, saved feature workflows, and embeddings: ```bash tldrgraph init ``` -In a detected coding-agent session, plain `tldrgraph init` auto-approves the full -enrichment campaign. Normal terminal users and non-agent automation still get the -confirmation gate. Full approval is persisted for the current candidate set until -enrichment finishes, so continuation runs must not ask again. +Use `--agent-cli` only when the user explicitly asks for AI-assisted architecture +layer design and all-symbol enrichment. Saved feature workflow generation is +independent and does not require `--agent-cli`. | status | what it wants | | --- | --- | -| `needs_layers` | Read the code and design this repository's architecture. **TLDRGraph ships no layer templates**; nothing will be applied for you. The request carries sketches of how other kinds of codebase divide — for shape only, never to copy. | -| `needs_confirmation` | Non-agent runs only: show the estimate and ask once. Approval via `tldrgraph init --yes` persists until the current campaign is done. | -| `needs_enrichment` | Open, read, and describe this batch, then continue immediately without asking again. | +| `needs_layers` | Only process if the user explicitly opted into architecture AI with `--agent-cli`. | +| `needs_confirmation` | Only belongs to explicit `--agent-cli` enrichment. Ask before continuing. | +| `needs_enrichment` | Only process if the user explicitly asked for full graph enrichment. | | `needs_embeddings` | Enrichment finished but the required dense model/index could not be built. Fix model access and rerun init. | | `done` | Nothing left. Use `query` / `trace` / `layers`. | @@ -46,22 +39,13 @@ dropped, and will be reported back to you — but the work is wasted. --- -## The loop +## Explicit Enrichment Loop -```bash -tldrgraph init # 1. approve the full campaign in agent sessions -# 2. read every requested source and write enrichment_response.yaml -tldrgraph init # 3. applies it and emits the next batch; approval is remembered -# 4. repeat steps 2-3 without asking until status: done -``` - -Inside an existing coding-agent session, nested-agent protection can prevent the CLI -from launching another agent. In that case **you are the enrichment agent** and must -process every batch yourself. Do not stop at `needs_enrichment`. - -`--batch 200` means process all candidates in chunks of 200. `--limit 200` means stop -after only 200 candidates and is only for an explicitly requested partial run. Never add -`--limit` or `--embeddings off` unless the user explicitly requests that behavior. +The legacy enrichment files below are for explicit `--agent-cli`, +`queue-enrichment`, or `apply-enrichment` work. Do not process them as part of +default Feature Workflow Explorer generation. Never add `--limit`, +`--agent-cli`, `--llm-links`, or `--embeddings off` unless the user explicitly +requests that behavior. Request and response are **separate files**. Never write your answer back into `enrichment_request.yaml`; it is regenerated on every run and your work would be lost. @@ -76,6 +60,42 @@ Request and response are **separate files**. Never write your answer back into --- +## Feature Workflow Explorer artifacts + +`tldrgraph init` also creates the saved workflow artifacts used by the visualizer: + +| File | Written by | Read by | +| --- | --- | --- | +| `.tldrgraph/features.yaml` | `tldrgraph init` | Workflow Explorer | +| `.tldrgraph/workflows/.yaml` | `tldrgraph init` | Workflow Explorer | + +The Workflow Explorer tab is intentionally file-driven. It must read only +`.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`; if those +files are missing, invalid, incomplete, or a feature has no generated workflow +yet, show an explicit empty or pending state. + +Do **not** reintroduce Workflow Explorer fallback discovery through +`discover_workflows()`, curated workflow blueprints, route-link workflow +discovery, `llm_http_route_link`, `http_route_link`, `calls_endpoint`, or +BPMN-derived workflow generation. Graph views elsewhere may still show route +links or BPMN data, but saved feature workflows must remain independent. + +Every saved workflow step must be simple enough for non-technical users and vibe +coders, and each step must carry source evidence: `node_id`, symbol, file, and +line/range. If the evidence is absent, mark the workflow pending instead of +guessing. + +A saved feature workflow should describe the complete flow when evidence exists: +the exact user button/menu/form action, event handler, validation, client +request code, request payload construction, API route/controller, +middleware/auth, service or use-case logic, persistence/database, background +job, external system, response payload creation, client response parsing, state +update, navigation/toast/rendered result, and visible success or error handling. +Do not stop at only the frontend or only the backend when the source proves the +handoff, and do not collapse multiple proven source hops into one vague step. + +--- + ## Request schema (`enrichment_request.yaml`) ```yaml diff --git a/tests/test_agent_loop.py b/tests/test_agent_loop.py index 4f7731a..3956037 100644 --- a/tests/test_agent_loop.py +++ b/tests/test_agent_loop.py @@ -642,7 +642,8 @@ def test_rules_tell_the_agent_to_read_the_source_and_not_invent(tmp_path): assert "0.35" in contract command = Path(written["Claude Code (command)"]).read_text(encoding="utf-8").lower() - assert "open the source file of every node" in command + assert "do not complete this" in command + assert "do not\nprocess enrichment batches" in command assert "never invent" in command @@ -716,8 +717,8 @@ def test_contract_documents_persistent_full_campaign_approval(tmp_path): written = installer_module.install_agent_rules(str(tmp_path)) text = Path(written["contract"]).read_text(encoding="utf-8") assert "enrichment_approval.json" in text - assert "--batch 200" in text and "--limit 200" in text - assert "without asking" in text + assert "--agent-cli" in text and "--limit" in text + assert "explicitly asks" in text assert "Never add" in text and "--embeddings off" in text diff --git a/tests/test_auto_agent.py b/tests/test_auto_agent.py index 1f01dec..178a914 100644 --- a/tests/test_auto_agent.py +++ b/tests/test_auto_agent.py @@ -54,6 +54,23 @@ def fake_agent(name: str = "fake") -> agent_runner.AgentCLI: ) +def complete_pending_workflows(root: Path) -> None: + manifest = yaml.safe_load((root / ".tldrgraph" / "features.yaml").read_text(encoding="utf-8")) + for feature in manifest.get("features", []): + path = root / feature["workflow_path"] + workflow = yaml.safe_load(path.read_text(encoding="utf-8")) + evidence_nodes = workflow.get("evidence_nodes") or [] + evidence = (evidence_nodes[0] if evidence_nodes else {}).get("evidence") or (feature.get("evidence") or [{}])[0] + workflow["status"] = "generated" + workflow["steps"] = [{ + "number": 1, + "title": f"Run {feature['title']}", + "text": "The current agent completed this workflow from source evidence.", + "evidence": [evidence], + }] + path.write_text(yaml.safe_dump(workflow, sort_keys=False), encoding="utf-8") + + @pytest.fixture def agent_allowed(monkeypatch): """ @@ -243,14 +260,14 @@ def test_auto_configure_prefers_the_agent_over_the_archetype(monkeypatch, cli_re assert reg.ids() == ("entry", "core", "shared") -def test_no_agent_means_no_layers_and_no_config_file(cli_repo): - """The archetype fallback is gone: nothing writes layers it did not derive.""" +def test_no_agent_uses_bootstrap_layers(cli_repo): + """Without architecture AI, init uses the honest single-bucket layer set.""" reg, cfg_path, source = auto_configure_layers( str(cli_repo), enricher=None, use_llm=False, use_agent=False ) - assert source == NEEDS_LAYERS - assert reg is None and cfg_path is None - assert not (cli_repo / ".tldrgraph" / "layers.config.yaml").exists() + assert source == "bootstrap" + assert reg is not None and reg.ids() == ("utility",) + assert cfg_path and (cli_repo / ".tldrgraph" / "layers.config.yaml").exists() def test_an_agent_authored_config_is_never_silently_replaced(monkeypatch, cli_repo): @@ -324,13 +341,12 @@ def _stub_agent_cli(monkeypatch, answer=_fake_answer): ) -def test_init_stops_and_asks_for_layers_first(cli_repo): - """Phase 1: no architecture, no template, so it must stop and ask.""" +def test_init_does_not_ask_ai_for_layers_by_default(cli_repo): res = CliRunner().invoke(cli, ["init", str(cli_repo)]) assert res.exit_code == 0, res.output - assert "status: needs_layers" in res.output - assert "propose_layers_request.json" in res.output - assert not (cli_repo / ".tldrgraph" / "layers.config.yaml").exists() + assert "status: needs_layers" not in res.output + assert not (cli_repo / ".tldrgraph" / "propose_layers_request.json").exists() + assert (cli_repo / ".tldrgraph" / "layers.config.yaml").exists() def test_init_extracts_before_asking_so_the_evidence_has_real_symbols(cli_repo): @@ -339,7 +355,10 @@ def test_init_extracts_before_asking_so_the_evidence_has_real_symbols(cli_repo): listing -- two repos with identical file trees can do entirely different things, and the agent is being asked to name what this one does. """ + from tldrgraph.propose_layers import generate_propose_request + CliRunner().invoke(cli, ["init", str(cli_repo)]) + generate_propose_request(str(cli_repo)) payload = json.loads( (cli_repo / ".tldrgraph" / "propose_layers_request.json").read_text(encoding="utf-8") ) @@ -353,7 +372,7 @@ def test_init_resumes_after_the_agent_answers_the_layers(cli_repo): CliRunner().invoke(cli, ["init", str(cli_repo)]) _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo)]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli"]) assert res.exit_code == 0, res.output assert "status: needs_layers" not in res.output assert (cli_repo / ".tldrgraph" / "layers.config.yaml").is_file() @@ -363,7 +382,7 @@ def test_init_resumes_after_the_agent_answers_the_layers(cli_repo): def test_init_asks_before_spending_tokens_and_shows_the_estimate(cli_repo): """Phase 3 gate: the user is told the size of the job before it starts.""" _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo)]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli"]) assert res.exit_code == 0, res.output assert "status: needs_confirmation" in res.output @@ -371,24 +390,24 @@ def test_init_asks_before_spending_tokens_and_shows_the_estimate(cli_repo): assert "tldrgraph init --yes" in res.output -def test_coding_agent_init_auto_approves_full_campaign(monkeypatch, cli_repo): +def test_coding_agent_init_does_not_auto_approve_enrichment_by_default(monkeypatch, cli_repo): _answer_layers(cli_repo) monkeypatch.setenv("AI_AGENT", "1") res = CliRunner().invoke(cli, ["init", str(cli_repo)]) assert res.exit_code == 0, res.output - assert "Detected coding-agent session ($AI_AGENT)" in res.output + assert "Detected coding-agent session ($AI_AGENT)" not in res.output assert "status: needs_confirmation" not in res.output - assert "status: needs_enrichment" in res.output - assert (cli_repo / ".tldrgraph" / APPROVAL_FILENAME).is_file() + assert "status: needs_enrichment" not in res.output + assert not (cli_repo / ".tldrgraph" / APPROVAL_FILENAME).exists() def test_coding_agent_init_with_limit_is_not_full_auto_approval(monkeypatch, cli_repo): _answer_layers(cli_repo) monkeypatch.setenv("AI_AGENT", "1") - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--limit", "1"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli", "--limit", "1"]) assert res.exit_code == 0, res.output assert "status: needs_confirmation" in res.output @@ -397,7 +416,7 @@ def test_coding_agent_init_with_limit_is_not_full_auto_approval(monkeypatch, cli def test_the_estimate_is_machine_readable(cli_repo): _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--json"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli", "--json"]) assert res.exit_code == 0, res.output payload = json.loads(res.output) @@ -410,7 +429,7 @@ def test_the_estimate_is_machine_readable(cli_repo): def test_yes_hands_out_an_enrichment_batch(cli_repo): _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes"]) assert res.exit_code == 0, res.output assert "status: needs_enrichment" in res.output @@ -423,7 +442,7 @@ def test_init_applies_the_agents_enrichment_and_reaches_done(cli_repo): """The full loop, played out the way an agent would: init, answer, init.""" _answer_layers(cli_repo) runner = CliRunner() - runner.invoke(cli, ["init", str(cli_repo), "--yes"]) + runner.invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes"]) state = cli_repo / ".tldrgraph" for _ in range(20): @@ -441,12 +460,12 @@ def test_init_applies_the_agents_enrichment_and_reaches_done(cli_repo): ]), encoding="utf-8", ) - res = runner.invoke(cli, ["init", str(cli_repo)]) + res = runner.invoke(cli, ["init", str(cli_repo), "--agent-cli"]) assert res.exit_code == 0, res.output - if "status: done" in res.output: + if "status: needs_feature_workflows" in res.output: break else: - raise AssertionError("init never reached status: done") + raise AssertionError("init never reached status: needs_feature_workflows") snapshot = json.loads((state / "graph.json").read_text(encoding="utf-8")) assert all( @@ -456,11 +475,16 @@ def test_init_applies_the_agents_enrichment_and_reaches_done(cli_repo): ) assert not (state / APPROVAL_FILENAME).exists() + complete_pending_workflows(cli_repo) + final = runner.invoke(cli, ["init", str(cli_repo), "--agent-cli"]) + assert final.exit_code == 0, final.output + assert "status: done" in final.output + def test_full_approval_survives_manual_batches_without_reconfirmation(cli_repo): _answer_layers(cli_repo) runner = CliRunner() - first = runner.invoke(cli, ["init", str(cli_repo), "--yes", "--batch", "1"]) + first = runner.invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes", "--batch", "1"]) assert "status: needs_enrichment" in first.output state = cli_repo / ".tldrgraph" @@ -470,16 +494,16 @@ def test_full_approval_survives_manual_batches_without_reconfirmation(cli_repo): encoding="utf-8", ) - continued = runner.invoke(cli, ["init", str(cli_repo), "--batch", "1"]) + continued = runner.invoke(cli, ["init", str(cli_repo), "--agent-cli", "--batch", "1"]) assert continued.exit_code == 0, continued.output assert "status: needs_confirmation" not in continued.output - assert "status: needs_enrichment" in continued.output or "status: done" in continued.output + assert "status: needs_enrichment" in continued.output or "status: needs_feature_workflows" in continued.output def test_limited_approval_does_not_authorize_the_remaining_campaign(cli_repo): _answer_layers(cli_repo) runner = CliRunner() - first = runner.invoke(cli, ["init", str(cli_repo), "--yes", "--limit", "1"]) + first = runner.invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes", "--limit", "1"]) assert "status: needs_enrichment" in first.output assert not (cli_repo / ".tldrgraph" / APPROVAL_FILENAME).exists() @@ -490,7 +514,7 @@ def test_limited_approval_does_not_authorize_the_remaining_campaign(cli_repo): yaml.dump([{"id": request["nodes"][0]["id"], "intent": "One approved node."}]), encoding="utf-8", ) - resumed = runner.invoke(cli, ["init", str(cli_repo)]) + resumed = runner.invoke(cli, ["init", str(cli_repo), "--agent-cli"]) assert "status: needs_confirmation" in resumed.output @@ -508,7 +532,7 @@ def test_an_applied_response_is_not_applied_twice(cli_repo): """ _answer_layers(cli_repo) runner = CliRunner() - runner.invoke(cli, ["init", str(cli_repo), "--yes"]) + runner.invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes"]) state = cli_repo / ".tldrgraph" request = yaml.safe_load((state / "enrichment_request.yaml").read_text(encoding="utf-8")) @@ -516,7 +540,7 @@ def test_an_applied_response_is_not_applied_twice(cli_repo): yaml.dump([{"id": n["id"], "intent": "Does a thing."} for n in request["nodes"]]), encoding="utf-8", ) - runner.invoke(cli, ["init", str(cli_repo), "--yes"]) + runner.invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes"]) assert not (state / "enrichment_response.yaml").exists() assert (state / "enrichment_response.applied.yaml").is_file() @@ -524,7 +548,7 @@ def test_an_applied_response_is_not_applied_twice(cli_repo): def test_limit_caps_the_first_pass(cli_repo): _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--batch", "2", "--limit", "2"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes", "--batch", "2", "--limit", "2"]) assert res.exit_code == 0, res.output request = yaml.safe_load( @@ -533,28 +557,26 @@ def test_limit_caps_the_first_pass(cli_repo): assert len(request["nodes"]) == 2 -def test_agent_cli_is_automatic_by_default(monkeypatch, cli_repo, agent_allowed): +def test_agent_cli_enrichment_is_explicit_opt_in(monkeypatch, cli_repo, agent_allowed): calls = [] monkeypatch.setattr(agent_runner, "find_agent_cli", lambda **kw: calls.append(1) or fake_agent()) monkeypatch.setattr(agent_runner, "run_agent", lambda *a, **k: _fake_answer(a[1])) _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes"]) assert res.exit_code == 0, res.output - assert calls, "init must auto-detect an agent CLI by default" - assert "status: done" in res.output + assert calls, "explicit --agent-cli must detect an agent CLI" + assert "status: needs_feature_workflows" in res.output -def test_no_agent_cli_forces_the_manual_handoff(monkeypatch, cli_repo, agent_allowed): - calls = [] - monkeypatch.setattr(agent_runner, "find_agent_cli", - lambda **kw: calls.append(1) or fake_agent()) +def test_no_agent_cli_skips_enrichment_handoff(monkeypatch, cli_repo, agent_allowed): + monkeypatch.setattr(agent_runner, "find_agent_cli", lambda **kw: None) _answer_layers(cli_repo) res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--no-agent-cli"]) assert res.exit_code == 0, res.output - assert calls == [] - assert "status: needs_enrichment" in res.output + assert "status: needs_enrichment" not in res.output + assert not (cli_repo / ".tldrgraph" / "enrichment_request.yaml").exists() def test_init_defaults_to_two_hundred_node_batches(): @@ -575,17 +597,17 @@ def test_interactive_init_asks_once_then_finishes(monkeypatch, cli_repo, agent_a _answer_layers(cli_repo) _stub_agent_cli(monkeypatch) monkeypatch.setattr(cli_pipeline, "stdin_is_interactive", lambda: True) - res = CliRunner().invoke(cli, ["init", str(cli_repo)], input="\n") + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli"], input="\n") assert res.exit_code == 0, res.output assert res.output.count("Enrich now?") == 1 - assert "status: done" in res.output + assert "status: needs_feature_workflows" in res.output def test_automatic_agent_keeps_json_output_parseable(monkeypatch, cli_repo, agent_allowed): _stub_agent_cli(monkeypatch) res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--json"]) assert res.exit_code == 0, res.output - assert json.loads(res.stdout)["status"] == "done" + assert json.loads(res.stdout)["status"] == "needs_feature_workflows" def test_embedding_failure_is_resumable(monkeypatch, cli_repo, agent_allowed): @@ -599,10 +621,10 @@ def test_embedding_failure_is_resumable(monkeypatch, cli_repo, agent_allowed): def test_agent_cli_runs_the_whole_loop_when_asked(monkeypatch, cli_repo, agent_allowed): _stub_agent_cli(monkeypatch) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--agent-cli"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--agent-cli", "--llm-links"]) assert res.exit_code == 0, res.output - assert "status: done" in res.output + assert "status: needs_feature_workflows" in res.output snapshot = json.loads((cli_repo / ".tldrgraph" / "graph.json").read_text(encoding="utf-8")) assert any(n.get("enrichment_source") == "agent" for n in snapshot["nodes"]) @@ -635,7 +657,7 @@ def _answer(prompt): return _fake_answer(prompt) _stub_agent_cli(monkeypatch, answer=_answer) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--agent-cli"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--agent-cli", "--llm-links"]) assert res.exit_code == 0, res.output snapshot = json.loads((cli_repo / ".tldrgraph" / "graph.json").read_text(encoding="utf-8")) @@ -668,7 +690,7 @@ def _answer(agent, prompt, cwd, timeout=None, model=None): agent_runner, "agent_status", lambda: {"agent": fake_agent(), "reason": "ready", "detail": "Fake fake"}, ) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--agent-cli"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--agent-cli", "--llm-links"]) assert res.exit_code == 0, res.output assert "status: needs_llm_links" in res.output @@ -846,7 +868,8 @@ def test_install_writes_the_gitignore_and_the_one_command(tmp_path): assert "tldrgraph init" in body assert "--batch 200" in body and "--limit 200" in body assert "Never add `--limit`" in body and "`--embeddings off` unless" in body - assert "without asking the user again" in body + assert "Do not complete this" in body + assert "Do not\nprocess enrichment batches" in body assert "/skills" in body and "$tldrgraph-init" in body # Every branch of the state machine must be documented in the command. for status in ("needs_layers", "needs_confirmation", "needs_enrichment"): @@ -1008,7 +1031,7 @@ def test_invented_ids_are_reported_not_silently_dropped(cli_repo): """ _answer_layers(cli_repo) runner = CliRunner() - runner.invoke(cli, ["init", str(cli_repo), "--yes"]) + runner.invoke(cli, ["init", str(cli_repo), "--agent-cli", "--yes"]) state = cli_repo / ".tldrgraph" request = yaml.safe_load((state / "enrichment_request.yaml").read_text(encoding="utf-8")) @@ -1106,7 +1129,7 @@ def test_json_mode_emits_parseable_json_only(cli_repo): front of the payload and break every parser reading it. """ _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--json"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli", "--json"]) assert res.exit_code == 0, res.output payload = json.loads(res.stdout) @@ -1120,7 +1143,7 @@ def test_enriched_count_does_not_include_excluded_nodes(cli_repo): A fresh graph claimed hundreds enriched before a single intent existed. """ _answer_layers(cli_repo) - res = CliRunner().invoke(cli, ["init", str(cli_repo), "--json"]) + res = CliRunner().invoke(cli, ["init", str(cli_repo), "--agent-cli", "--json"]) payload = json.loads(res.stdout) assert payload["progress"]["enriched"] == 0, "nothing has been enriched yet" diff --git a/tests/test_dynamic_layers.py b/tests/test_dynamic_layers.py index b4ebca2..c4a5b52 100644 --- a/tests/test_dynamic_layers.py +++ b/tests/test_dynamic_layers.py @@ -212,9 +212,8 @@ def test_auto_configure_layers_with_llm(tmp_path): def test_no_layer_source_means_no_layers_not_a_template(tmp_path): """ - HARD GATE. TLDRGraph used to synthesize a generic archetype layer set here. - That silently became the answer and classified badly. With nothing able to - read the code, the only honest result is "ask the agent". + Without architecture AI, TLDRGraph writes the honest single-bucket bootstrap + layer set instead of asking an agent to design architecture. """ (tmp_path / "pyproject.toml").write_text( '[project.scripts]\nmycmd = "mycmd.cli:main"\n', encoding="utf-8" @@ -222,28 +221,26 @@ def test_no_layer_source_means_no_layers_not_a_template(tmp_path): reg, cfg_path, source = auto_configure_layers( str(tmp_path), enricher=None, use_llm=False, use_agent=False ) - assert source == NEEDS_LAYERS - assert reg is None and cfg_path is None - assert not (tmp_path / ".tldrgraph" / "layers.config.yaml").exists(), ( - "a layer config must never be written without a real source" - ) + assert source == "bootstrap" + assert reg is not None and reg.ids() == ("utility",) + assert cfg_path and (tmp_path / ".tldrgraph" / "layers.config.yaml").exists() # --------------------------------------------------------------------------- # # 4. End-to-End CLI Scan with Dynamic Layers # --------------------------------------------------------------------------- # -def test_cli_propose_layers_falls_through_to_a_request(tmp_path): - """With nothing able to read the code, --auto must queue a request, not a template.""" +def test_cli_propose_layers_auto_uses_bootstrap_without_ai(tmp_path): + """With no architecture AI, --auto writes the honest bootstrap layer.""" (tmp_path / "pyproject.toml").write_text( '[project.scripts]\ntool = "tool.cli:main"\n', encoding="utf-8" ) runner = CliRunner() res = runner.invoke(cli, ["propose-layers", "--path", str(tmp_path), "--auto"]) assert res.exit_code == 0 - assert "no template to fall back on" in res.output - assert (tmp_path / ".tldrgraph" / "propose_layers_request.json").is_file() - assert not (tmp_path / ".tldrgraph" / "layers.config.yaml").exists() + assert "bootstrap" in res.output + assert not (tmp_path / ".tldrgraph" / "propose_layers_request.json").exists() + assert (tmp_path / ".tldrgraph" / "layers.config.yaml").exists() def test_cli_scan_initializes_dynamic_layers_automatically(tmp_path): @@ -266,28 +263,9 @@ def test_cli_scan_initializes_dynamic_layers_automatically(tmp_path): runner = CliRunner() res = runner.invoke(cli, ["scan", str(tmp_path)]) assert res.exit_code == 0 - # No agent is reachable in tests, so the scan must stop and ask rather than - # classify this repo with layers it never derived. - assert "status: needs_layers" in res.output - assert "tldrgraph init" in res.output - - assert not (tmp_path / ".tldrgraph" / "layers.config.yaml").exists() - - # The request the agent is asked to answer must actually be there, carrying - # the symbols extraction already found -- filenames alone are not evidence. - request = tmp_path / ".tldrgraph" / "propose_layers_request.json" - assert request.is_file() - payload = json.loads(request.read_text(encoding="utf-8")) - assert payload["evidence"]["extracted_symbols"]["total_symbols"] > 0 - - # Answer it the way an agent would, and the next run gets all the way through. - (tmp_path / ".tldrgraph" / "propose_layers_response.json").write_text( - json.dumps(SAMPLE_LAYER_SET), encoding="utf-8" - ) - res2 = runner.invoke(cli, ["init", str(tmp_path)]) - assert res2.exit_code == 0, res2.output - assert "status: needs_layers" not in res2.output - assert (tmp_path / ".tldrgraph" / "layers.config.yaml").is_file() + assert "status: needs_layers" not in res.output + + assert (tmp_path / ".tldrgraph" / "layers.config.yaml").exists() res_layers = runner.invoke(cli, ["layers", "--path", str(tmp_path)]) assert res_layers.exit_code == 0 diff --git a/tests/test_feature_workflows.py b/tests/test_feature_workflows.py new file mode 100644 index 0000000..f2f1e33 --- /dev/null +++ b/tests/test_feature_workflows.py @@ -0,0 +1,185 @@ +"""Feature Workflow Explorer saved-file contract.""" + +import yaml + + +def test_feature_workflow_files_are_written_from_source_evidence(loader, mini_repo): + from tldrgraph.feature_workflows import ( + FEATURES_FILENAME, + WORKFLOW_SCHEMA, + generate_feature_workflow_files, + load_saved_feature_workflows, + ) + + graph = loader.load_or_extract(enrich_llm=False) + stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) + + assert stats["features"] > 0 + features_path = mini_repo.tldrgraph_dir / FEATURES_FILENAME + assert features_path.exists() + + manifest = yaml.safe_load(features_path.read_text(encoding="utf-8")) + assert manifest["schema"] == "codechakra/features@1" + assert manifest["features"] + + for feature in manifest["features"]: + workflow_file = mini_repo.root / feature["workflow_path"] + assert workflow_file.exists() + workflow = yaml.safe_load(workflow_file.read_text(encoding="utf-8")) + assert workflow["schema"] == WORKFLOW_SCHEMA + assert workflow["feature_id"] == feature["id"] + assert workflow["status"] == "pending" + assert workflow["steps"] == [] + assert "current coding agent" in workflow["pending_reason"] + assert workflow["evidence"] + assert workflow["evidence_nodes"] + assert "required_step_shape" in workflow + assert any("Open every source file" in line for line in workflow["instructions"]) + + payload = load_saved_feature_workflows(str(mini_repo.root)) + assert payload["state"] == "ready" + assert payload["ready_count"] == 0 + assert payload["pending_count"] == len(manifest["features"]) + assert all(wf["process"]["elements"] for wf in payload["workflows"]) + + +def test_feature_workflow_validation_rejects_steps_without_evidence(): + from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow + + assert not validate_workflow({ + "schema": WORKFLOW_SCHEMA, + "feature_id": "missing_evidence", + "status": "generated", + "steps": [{"number": 1, "title": "Guess", "text": "No backing source."}], + }) + + +def test_feature_workflows_do_not_spawn_agent_cli(monkeypatch, loader, mini_repo): + from tldrgraph import feature_workflow_agent + from tldrgraph.feature_workflows import generate_feature_workflow_files, workflow_path + + monkeypatch.setattr(feature_workflow_agent, "generate_feature_manifest", lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("spawned feature agent"))) + monkeypatch.setattr(feature_workflow_agent, "generate_feature_workflow", lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("spawned feature agent"))) + + graph = loader.load_or_extract(enrich_llm=False) + stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=True) + workflow = yaml.safe_load(open(workflow_path(str(mini_repo.root), "submitcasebutton"), encoding="utf-8")) + + assert stats["agent_reason"] == "" + assert workflow["status"] == "pending" + assert "current coding agent" in workflow["pending_reason"] + + +def test_missing_workflow_file_becomes_pending_state(mini_repo): + from tldrgraph.cli_enrichment import write_payload + from tldrgraph.feature_workflows import FEATURE_SCHEMA, load_saved_feature_workflows + + write_payload(str(mini_repo.tldrgraph_dir / "features.yaml"), { + "schema": FEATURE_SCHEMA, + "graph_hash": "test", + "features": [{ + "id": "not_generated", + "title": "Not Generated", + "audience": "developer", + "summary": "A feature without a workflow file.", + "status": "pending", + "workflow_path": ".tldrgraph/workflows/not_generated.yaml", + "evidence": [{ + "node_id": mini_repo.nid("devops_ci"), + "symbol": mini_repo.label("devops_ci"), + "file": mini_repo.source_file("devops_ci"), + "line": 1, + }], + }], + }) + + payload = load_saved_feature_workflows(str(mini_repo.root)) + + assert payload["state"] == "ready" + assert payload["ready_count"] == 0 + assert payload["pending_count"] == 1 + assert payload["workflows"][0]["status"] == "pending" + assert "not been generated" in payload["workflows"][0]["pending_reason"] + + +def test_init_pipeline_writes_all_feature_workflows(monkeypatch, mini_repo): + from tldrgraph.cli_pipeline import init_pipeline + from tldrgraph.feature_workflows import load_saved_feature_workflows + + monkeypatch.setattr("tldrgraph.graph_loader.GraphLoader._run_graphify", lambda self: None) + + status = init_pipeline( + str(mini_repo.root), + assume_yes=True, + batch_size=200, + max_nodes=0, + rebuild=False, + relayer=False, + agent_cli=False, + agent_model=None, + embeddings="off", + llm_links=False, + as_json=True, + ) + + payload = load_saved_feature_workflows(str(mini_repo.root)) + assert status == "needs_feature_workflows" + assert payload["workflows"] + assert payload["ready_count"] == 0 + assert payload["pending_count"] == len(payload["workflows"]) + + +def test_existing_generated_workflow_is_preserved(mini_repo): + from tldrgraph.feature_workflows import generate_feature_workflow_files, workflow_path + import yaml + + from tldrgraph.graph_loader import GraphLoader + graph = GraphLoader(str(mini_repo.root)).load_or_extract(enrich_llm=False) + stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) + path = workflow_path(str(mini_repo.root), "submitcasebutton") + workflow = yaml.safe_load(open(path, encoding="utf-8")) + workflow["status"] = "generated" + workflow["steps"] = [{ + "number": 1, + "title": "Use the page", + "text": "The user starts from the case page.", + "evidence": [workflow["evidence_nodes"][0]["evidence"]], + }] + with open(path, "w", encoding="utf-8") as f: + yaml.safe_dump(workflow, f, sort_keys=False) + + stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) + workflow = yaml.safe_load(open(path, encoding="utf-8")) + + assert stats["generated"] == 1 + assert workflow["status"] == "generated" + assert workflow["steps"][0]["evidence"][0]["symbol"] == "SubmitCaseButton" + + +def test_stale_workflows_without_current_generator_are_regenerated_or_pending(mini_repo): + from tldrgraph.cli_enrichment import write_payload + from tldrgraph.feature_workflows import generate_feature_workflow_files, graph_hash, workflow_path, WORKFLOW_SCHEMA + from tldrgraph.graph_loader import GraphLoader + + graph = GraphLoader(str(mini_repo.root)).load_or_extract(enrich_llm=False) + current_hash = graph_hash(graph) + write_payload(workflow_path(str(mini_repo.root), "submitcasebutton"), { + "schema": WORKFLOW_SCHEMA, + "graph_hash": current_hash, + "feature_id": "submitcasebutton", + "title": "Submit Case Button", + "summary": "Old shallow workflow.", + "status": "generated", + "steps": [{"number": 1, "title": "Old", "text": "Old.", "evidence": [{ + "node_id": mini_repo.nid("ui_page"), + "symbol": mini_repo.label("ui_page"), + "file": mini_repo.source_file("ui_page"), + "line": 1, + }]}], + }) + + generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) + workflow = yaml.safe_load(open(workflow_path(str(mini_repo.root), "submitcasebutton"), encoding="utf-8")) + + assert workflow["status"] == "pending" + assert workflow["steps"] == [] diff --git a/tests/test_visualizer.py b/tests/test_visualizer.py index bd3260a..60ac310 100644 --- a/tests/test_visualizer.py +++ b/tests/test_visualizer.py @@ -164,12 +164,56 @@ def test_generated_html_contains_no_project_source(mini_repo): def test_workflows_payload_structure(mini_repo): - """Payload includes extracted workflow sequences mapping methods to files and layers.""" + """Workflow Explorer reads only saved feature workflow files.""" from tldrgraph.visualizer import prepare_visualizer_data + from tldrgraph.cli_enrichment import write_payload + from tldrgraph.feature_workflows import FEATURE_SCHEMA, WORKFLOW_SCHEMA data = prepare_visualizer_data(str(mini_repo.root)) assert "workflows" in data - assert isinstance(data["workflows"], list) + assert data["workflows"] == [] + assert data["workflow_state"]["state"] == "missing_features" + + write_payload(str(mini_repo.tldrgraph_dir / "features.yaml"), { + "schema": FEATURE_SCHEMA, + "graph_hash": "test", + "features": [{ + "id": "submit_case", + "title": "Submit Case", + "audience": "user", + "summary": "Send a case through the project.", + "status": "generated", + "workflow_path": ".tldrgraph/workflows/submit_case.yaml", + "evidence": [{ + "node_id": mini_repo.nid("ui_page"), + "symbol": mini_repo.label("ui_page"), + "file": mini_repo.source_file("ui_page"), + "line": 1, + }], + }], + }) + write_payload(str(mini_repo.tldrgraph_dir / "workflows" / "submit_case.yaml"), { + "schema": WORKFLOW_SCHEMA, + "graph_hash": "test", + "feature_id": "submit_case", + "title": "Submit Case", + "summary": "Send a case through the project.", + "status": "generated", + "steps": [{ + "number": 1, + "title": "Open the case page", + "text": "The user starts from the case page.", + "evidence": [{ + "node_id": mini_repo.nid("ui_page"), + "symbol": mini_repo.label("ui_page"), + "file": mini_repo.source_file("ui_page"), + "line": 1, + }], + }], + }) + + data = prepare_visualizer_data(str(mini_repo.root)) + assert len(data["workflows"]) == 1 for wf in data["workflows"]: assert "id" in wf @@ -187,36 +231,24 @@ def test_workflows_payload_structure(mini_repo): assert "node_id" in s -def test_workflow_extraction_is_not_capped_at_twenty(monkeypatch): - """Every distinct feature journey is retained after curated workflows.""" - import networkx as nx - from tldrgraph.visualizer.flows_data import extract_visualizer_workflows +def test_workflow_explorer_does_not_call_discovery_or_bpmn(monkeypatch, mini_repo): + """The tab is file-driven, not discovered from routes, blueprints, or BPMN.""" + from tldrgraph.visualizer import prepare_visualizer_data + import tldrgraph.visualizer.bpmn_data + import tldrgraph.visualizer.flows_data + import tldrgraph.visualizer.flows_discover - graph = nx.DiGraph() - nodes_by_id = {} - for index in range(21): - root = f"root_{index}" - handler = f"handler_{index}" - store = f"store_{index}" - for node_id, label, path in ( - (root, f"OrdersPage{index}()", f"frontend/src/app/orders_{index}/page.tsx"), - (handler, f"handleOrder{index}()", f"backend/src/orders_{index}.controller.ts"), - (store, f"saveOrder{index}()", f"src/data/orders_{index}.py"), - ): - nodes_by_id[node_id] = { - "label": label, "file": path, - "layer_id": "ui" if node_id == root else ("api" if node_id == handler else "data"), - "layer": "UI" if node_id == root else ("API" if node_id == handler else "Data"), - "is_test": False, - } - graph.add_node(node_id, label=label, file=path) - graph.add_edge(root, handler, relation="llm_http_route_link") - graph.add_edge(handler, store, relation="calls") + def boom(*_args, **_kwargs): + raise AssertionError("old Workflow Explorer discovery path was called") - monkeypatch.setattr("tldrgraph.visualizer.flows_data.CURATED_BLUEPRINTS", []) - workflows = extract_visualizer_workflows(graph, nodes_by_id, sources=None) # type: ignore[arg-type] + monkeypatch.setattr("tldrgraph.visualizer.flows_discover.discover_workflows", boom) + monkeypatch.setattr("tldrgraph.visualizer.flows_data.extract_visualizer_workflows", boom) + monkeypatch.setattr("tldrgraph.visualizer.bpmn_data.attach_bpmn_processes", boom) - assert len(workflows) == 21 + data = prepare_visualizer_data(str(mini_repo.root)) + + assert data["workflows"] == [] + assert data["workflow_state"]["state"] == "missing_features" def test_next_root_page_is_a_workflow_entry_with_one_component_edge(): @@ -400,64 +432,10 @@ def format_step(node_id, step_number): assert [step["node_id"] for step in workflows[0]["steps"]] == ["prompt", "api", "endpoint", "handler"] -def test_visualizer_keeps_only_complete_frontend_to_backend_flows(monkeypatch): - import networkx as nx - from tldrgraph.visualizer.flows_data import extract_visualizer_workflows - - graph = nx.DiGraph() - nodes = { - "page": { - "label": "CasesPage()", "file": "frontend/src/app/cases/page.tsx", - "layer_id": "client_experience", "layer": "Client Experience", "is_test": False, - }, - "handler": { - "label": "createCase()", "file": "backend/src/cases.controller.ts", - "layer_id": "api_delivery", "layer": "API Delivery", "is_test": False, - }, - "service": { - "label": "createCaseRecord()", "file": "backend/src/cases.service.ts", - "layer_id": "application_services", "layer": "Application Services", "is_test": False, - }, - "backend_only": { - "label": "nightlySync()", "file": "backend/src/jobs/sync.ts", - "layer_id": "async", "layer": "Async", "is_test": False, - }, - "backend_service": { - "label": "syncCases()", "file": "backend/src/cases.service.ts", - "layer_id": "service", "layer": "Service", "is_test": False, - }, - "backend_endpoint": { - "label": "GET /cases/sync", "file": "backend/src/routes/cases.ts", - "layer_id": "api_delivery", "layer": "API Delivery", "is_test": False, - }, - "ui_only": { - "label": "HelpPage()", "file": "frontend/src/app/help/page.tsx", - "layer_id": "ui", "layer": "UI", "is_test": False, - }, - "component": { - "label": "HelpContent()", "file": "frontend/src/app/help/HelpContent.tsx", - "layer_id": "ui", "layer": "UI", "is_test": False, - }, - } - graph.add_nodes_from((node_id, data) for node_id, data in nodes.items()) - graph.add_edge("page", "handler", relation="llm_http_route_link") - graph.add_edge("handler", "service", relation="calls") - graph.add_edge("backend_only", "backend_service", relation="calls_endpoint") - graph.add_edge("backend_service", "backend_endpoint", relation="calls") - graph.add_edge("ui_only", "component", relation="calls") - - monkeypatch.setattr("tldrgraph.visualizer.flows_data.CURATED_BLUEPRINTS", []) - workflows = extract_visualizer_workflows(graph, nodes, sources=None) # type: ignore[arg-type] - - assert [w["root_id"] for w in workflows] == ["page"] - assert workflows[0]["feature_flow"] is True - assert workflows[0]["completeness"] == "frontend_to_backend" - assert workflows[0]["route_link_relation"] == "llm_http_route_link" - - -def test_visualizer_uses_deterministic_route_link_as_fallback(monkeypatch): +def test_saved_feature_generation_ignores_route_link_relations(): import networkx as nx - from tldrgraph.visualizer.flows_data import extract_visualizer_workflows + import yaml + from tldrgraph.feature_workflows import generate_feature_workflow_files, load_saved_feature_workflows, workflow_path graph = nx.DiGraph() nodes = { @@ -472,34 +450,16 @@ def test_visualizer_uses_deterministic_route_link_as_fallback(monkeypatch): } graph.add_nodes_from((node_id, data) for node_id, data in nodes.items()) graph.add_edge("page", "handler", relation="http_route_link") + graph.add_edge("handler", "store", relation="calls") - monkeypatch.setattr("tldrgraph.visualizer.flows_data.CURATED_BLUEPRINTS", []) - workflows = extract_visualizer_workflows(graph, nodes, sources=None) # type: ignore[arg-type] - - assert len(workflows) == 1 - assert workflows[0]["route_link_relation"] == "http_route_link" - - -def test_visualizer_accepts_legacy_endpoint_calls_as_route_fallback(monkeypatch): - import networkx as nx - from tldrgraph.visualizer.flows_data import extract_visualizer_workflows + import tempfile - graph = nx.DiGraph() - nodes = { - "page": { - "label": "BillingPage()", "file": "frontend/src/app/billing/page.tsx", - "layer_id": "client_experience", "layer": "Client Experience", "is_test": False, - }, - "endpoint": { - "label": "GET /billing/pricing", "file": "backend/src/routes/billing.ts", - "layer_id": "api_delivery", "layer": "API Delivery", "is_test": False, - }, - } - graph.add_nodes_from((node_id, data) for node_id, data in nodes.items()) - graph.add_edge("page", "endpoint", relation="calls_endpoint") - - monkeypatch.setattr("tldrgraph.visualizer.flows_data.CURATED_BLUEPRINTS", []) - workflows = extract_visualizer_workflows(graph, nodes, sources=None) # type: ignore[arg-type] + with tempfile.TemporaryDirectory() as root: + generate_feature_workflow_files(root, graph, use_agent=True) + raw = yaml.safe_load(open(workflow_path(root, "orderspage"), encoding="utf-8")) + payload = load_saved_feature_workflows(root) - assert len(workflows) == 1 - assert workflows[0]["route_link_relation"] == "calls_endpoint" + assert payload["workflows"] + assert payload["workflows"][0]["status"] == "pending" + outgoing = raw["evidence_nodes"][0]["outgoing"] + assert all(item["target"]["node_id"] != "handler" for item in outgoing) diff --git a/tldrgraph/agent_commands.py b/tldrgraph/agent_commands.py index c111b2b..d0ff3a1 100644 --- a/tldrgraph/agent_commands.py +++ b/tldrgraph/agent_commands.py @@ -119,12 +119,18 @@ class AgentTarget: 2. Use `view_file` on the target file path returned by TLDRGraph to inspect the code. Those are read-only and never trigger enrichment. - -**To build or refresh the graph**, run `tldrgraph init`. It automatically handles -layer design, extraction, source-aware enrichment in 200-node batches, and dense -embeddings when a supported agent CLI is available. If it prints a `NEXT ACTION` -fallback, follow that handoff without guessing from symbol names. - +**To build or refresh the graph**, run `tldrgraph init`. It handles layer setup, +extraction, embeddings, and writes the file-backed Feature Workflow Explorer +artifacts: `.tldrgraph/features.yaml` plus `.tldrgraph/workflows/.yaml`. + +Feature workflows are owned by the agent running `tldrgraph init`. If a workflow +file is pending, open `.tldrgraph/features.yaml`, then complete each pending +`.tldrgraph/workflows/.yaml` from its evidence. Workflow Explorer +reads only saved YAML, never curated blueprints, route-link workflow discovery, +BPMN-derived generation, `discover_workflows()`, `llm_http_route_link`, +`http_route_link`, or `calls_endpoint`. When writing feature workflows, start at +the user's button/menu/form action and continue through client request, backend +work, response payload, client handling, and final UI update. Do not skip proven steps. Full workflow: `.claude/commands/{COMMAND_NAME}.md` (identical copies live in every other agent directory). Schema: `.tldrgraph/AGENT_CONTRACT.md`. @@ -143,101 +149,53 @@ class AgentTarget: In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select `tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles layer design, extraction, enrichment, LLM route links, and embeddings: +One command handles extraction, feature-file scaffolding, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph detects `claude`, `cursor-agent`, or `gemini`, processes -200-node batches, and builds embeddings. In a detected coding-agent session, -plain `tldrgraph init` auto-approves the full enrichment campaign; normal -terminal users and non-agent automation still get the confirmation gate. +By default TLDRGraph does not ask another AI process to design architecture +layers, enrich every symbol, infer route links, or generate BPMN workflows. It +writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` +file per feature. The coding agent that ran `tldrgraph init` owns completing any +pending workflow files from source evidence. Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent `needs_confirmation` response explicitly asks for approval. `--batch 200` means all nodes in chunks; `--limit 200` means stop after only 200 nodes. -Never add `--limit`, `--no-llm-links`, or `--embeddings off` unless the user +Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless the user explicitly requests it. -If no supported agent is available or dense embeddings cannot be built, `init` -preserves the graph and prints a resumable status. It never guesses source intent, -route links, or architectural layers. - -## `status: needs_layers` +## Feature Workflow Explorer artifacts -TLDRGraph ships **no layer templates** and will not invent an architecture. - -1. Read `.tldrgraph/propose_layers_request.json`. It carries the symbols and - files extraction already found -- a starting point, not a substitute for - opening the code. -2. **Open real source files**: entry points first, then a representative file - from each cluster in the evidence. Work out what this codebase actually does - and where responsibility changes hands. -3. Write `.tldrgraph/propose_layers_response.json`: - -```json -{ - "utility_id": "", - "layers": [ - {"id": "short_machine_id", "name": "Layer 1: Human Friendly Name", - "order": 1, "description": "One sentence", "rules": [{"file_contains": ["substring"]}]} - ] -} -``` +Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +Workflow Explorer reads only those files; missing or invalid workflows show pending states. +Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. +Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +When `init` reports pending feature workflows: -4. Run `tldrgraph init` again; it continues with enrichment and embeddings. +1. Open `.tldrgraph/features.yaml`. +2. Open every pending `.tldrgraph/workflows/.yaml`. +3. Open the source files referenced in `evidence_nodes`; do not guess. +4. Replace `steps: []` with source-backed steps and set `status: generated`. +5. Run `tldrgraph init` again. -### Rules that hold for any answer +## `status: needs_layers` -- 3 to 6 layers, plus exactly one catch-all whose `id` equals `utility_id` and - whose `rules` are `[]`. -- Unique `id` and `name` per layer; sequential integer `order` from 1. -- Rule keys: `file_contains`, `exclude_file`, `path_regex`, `label_contains`, - `exclude_label`, `label_ends_with`, `type_in`, `id_prefix`. Values are lists of - strings. Rules are evaluated in `order` and the first match wins. -- Derive rules from paths and symbol names you actually saw. A rule matching - nothing is worse than no rule; a rule matching everything collapses the map. +This should only appear when the user explicitly opted into architecture AI with +`--agent-cli` or an older TLDRGraph build is running. Do not complete this +handoff unless the user asked for architecture layer design. ## `status: needs_confirmation` -Detected coding-agent sessions should not reach this state for a full run. If a -non-agent run does, the output shows how many nodes need enrichment and how many -agent round-trips that implies; ask the user whether to proceed. - -- They agree: `tldrgraph init --yes` saves approval for the full campaign -- Smaller first pass: `tldrgraph init --yes --limit 100` -- They decline: stop. The graph is already built and queryable. +This belongs to explicit `--agent-cli` enrichment. Ask before continuing. ## `status: needs_enrichment` -1. Read `.tldrgraph/enrichment_request.yaml`. -2. **Open the source file of every node in it.** This is the entire point: an - intent paraphrased from a symbol name poisons semantic search with - confident-sounding noise. -3. Write `.tldrgraph/enrichment_response.yaml` -- a *different* file from the - request, which is regenerated on every run: - -```yaml -- id: "" - intent: | - What this symbol does, why it exists, and its execution logic. - input_fields: [caseId, remarks] - output_fields: [status, disposition] - calls: [ApplicationsService, pension_cases] -``` - -Every `intent` must contain **2-3 complete sentences** covering what the symbol does, -why it exists, and its source-backed behavior. Markdown headings and list markers do not -count as sentences. - -4. Run `tldrgraph init` again. Approval is saved; process any next - `needs_enrichment` batch immediately without asking the user again. - -Inside an existing Codex/Claude/Cursor session, nested-agent protection may stop -the CLI from launching a second agent. In that case **you are the enrichment -agent**: process every 200-node batch yourself. A `needs_enrichment` status is a -continuation instruction, not a reason to stop or request confirmation. +This should only appear when the user explicitly opted into architecture +enrichment with `--agent-cli` or an older TLDRGraph build is running. Do not +process enrichment batches unless the user asked for full graph enrichment. **Copy every `id` verbatim.** A constructed id matches nothing, is dropped, and gets reported back to you -- but the work is wasted. @@ -251,8 +209,8 @@ class AgentTarget: files, then write `.tldrgraph/llm_links_response.yaml` as a YAML list of `{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. Only include source-backed links with file and line evidence. Run `tldrgraph init` -again. This is a required continuation state for a complete init run unless the -user explicitly requested `--no-llm-links`. +again. This continuation state only appears when the user explicitly opted into +route-link inference with `--llm-links`. ## Once it says DONE diff --git a/tldrgraph/cli.py b/tldrgraph/cli.py index a37a6c9..b89d8b7 100644 --- a/tldrgraph/cli.py +++ b/tldrgraph/cli.py @@ -100,15 +100,14 @@ help="Partial-run cap. 0 authorizes every current candidate."), click.option("--rebuild", is_flag=True, help="Re-extract and rebuild enrichment from scratch"), click.option("--relayer", is_flag=True, help="Discard the layer set and design it again"), - click.option("--agent-cli/--no-agent-cli", default=True, show_default=True, - help="Automatically use a supported agent CLI for layers and enrichment; " - "disable to use the file handoff workflow."), + click.option("--agent-cli/--no-agent-cli", default=False, show_default=True, + help="Opt in to a supported agent CLI for architecture layers and enrichment."), click.option("--agent-model", default=None, help="Model for --agent-cli (e.g. opus, sonnet, gemini-2.5-pro). Defaults " "to $TLDRGRAPH_AGENT_MODEL. Ignored on the handshake path, where your " "own agent session picks the model."), - click.option("--llm-links/--no-llm-links", default=True, show_default=True, - help="Infer evidence-backed frontend/backend links during init."), + click.option("--llm-links/--no-llm-links", default=False, show_default=True, + help="Opt in to evidence-backed frontend/backend route-link inference during init."), click.option("--json", "as_json", is_flag=True, help="Emit machine-readable status"), embeddings_option, ] diff --git a/tldrgraph/cli_pipeline.py b/tldrgraph/cli_pipeline.py index a8c7194..20d9b69 100644 --- a/tldrgraph/cli_pipeline.py +++ b/tldrgraph/cli_pipeline.py @@ -1,6 +1,4 @@ -""" -Init workflow pipeline for TLDRGraph CLI. -""" +"""Init workflow pipeline for TLDRGraph CLI.""" from __future__ import annotations @@ -10,16 +8,16 @@ import sys from typing import Any, Dict, List, Optional import click - -from . import agent_runner, paths +from . import agent_runner from .cli_agent_loop import run_agent_enrichment from .cli_enrichment import ( AGENT_ENRICHMENT_SOURCE, REQUEST_FILENAME, RESPONSE_FILENAME, STATE_DIR, apply_enrichment_items, build_enrichment_batch, coerce_enrichment_items, - compute_degrees, enrichment_candidates, needs_agent_enrichment, read_payload, + compute_degrees, enrichment_candidates, read_payload, stamp_degrees, state_path, write_payload, ) from .cli_llm_links import apply_pending_llm_links_response, run_llm_link_step +from .feature_workflows import generate_feature_workflow_files from .graph_loader import GraphLoader from .installer import ensure_gitignore, install_agent_rules from .layer_config import config_path @@ -34,13 +32,14 @@ auto_configure_layers, generate_propose_request, ) from .visualizer import generate_visualizer_html - STATUS_DONE = "done" STATUS_NEEDS_LAYERS = "needs_layers" STATUS_NEEDS_CONFIRMATION = "needs_confirmation" STATUS_NEEDS_ENRICHMENT = "needs_enrichment" STATUS_NEEDS_EMBEDDINGS = "needs_embeddings" +STATUS_NEEDS_FEATURE_WORKFLOWS = "needs_feature_workflows" APPLIED_RESPONSE_FILENAME = "enrichment_response.applied.yaml" + @contextlib.contextmanager def stdout_to_stderr_if(active: bool): if not active: @@ -49,14 +48,12 @@ def stdout_to_stderr_if(active: bool): with contextlib.redirect_stdout(sys.stderr): yield - def stdin_is_interactive() -> bool: try: return bool(sys.stdin and sys.stdin.isatty()) except Exception: return False - def emit_status(status: str, phase: str, lines: List[str], progress: Optional[Dict[str, Any]] = None, as_json: bool = False) -> None: if as_json: click.echo(json.dumps({ @@ -76,7 +73,6 @@ def emit_status(status: str, phase: str, lines: List[str], progress: Optional[Di click.echo(line) click.echo(f"{rule}\n") - def apply_pending_layer_response(path: str) -> Optional[str]: for filename in (PROPOSE_RESPONSE_FILENAME, "propose_layers_response.yaml"): candidate = state_path(path, filename) @@ -84,7 +80,6 @@ def apply_pending_layer_response(path: str) -> Optional[str]: return apply_proposed_layers(path, candidate) return None - def apply_pending_enrichment_response(path: str, loader: GraphLoader) -> Optional[Dict[str, Any]]: for filename in (RESPONSE_FILENAME, "enrichment_response.json", "pending_enrichment.yaml", "pending_enrichment.json"): candidate = state_path(path, filename) @@ -100,7 +95,6 @@ def apply_pending_enrichment_response(path: str, loader: GraphLoader) -> Optiona return stats return None - def _check_confirmation(candidates: List[Dict[str, Any]], total: int, enriched: int, excluded: int, rounds: int, batch_size: int, progress: Dict[str, Any], as_json: bool) -> Optional[str]: if stdin_is_interactive(): click.echo(f"\n🧠 {len(candidates)} node(s) need an intent read from the source ({rounds} batch(es) of {batch_size}).") @@ -128,6 +122,7 @@ def _run_agent_cli_enrichment( progress: Dict[str, Any], as_json: bool, llm_links: bool, + feature_stats: Optional[Dict[str, Any]] = None, ) -> Optional[str]: agent = agent_runner.find_agent_cli() if agent is None: @@ -148,8 +143,9 @@ def _run_agent_cli_enrichment( link_status = run_llm_link_step(path, os.path.abspath(path), loader, True, agent_model, as_json, emit_status) if link_status: return link_status embedding_error = None if rem else _embedding_failure(loader) + pending_workflows = int((feature_stats or {}).get("pending") or 0) status = STATUS_NEEDS_ENRICHMENT if rem else ( - STATUS_NEEDS_EMBEDDINGS if embedding_error else STATUS_DONE + STATUS_NEEDS_EMBEDDINGS if embedding_error else (STATUS_NEEDS_FEATURE_WORKFLOWS if pending_workflows else STATUS_DONE) ) resume = "tldrgraph init" if progress.get("approval_persisted") else "tldrgraph init --yes" retry = [f"Run `{resume}` to continue."] if rem or embedding_error else [] @@ -157,9 +153,10 @@ def _run_agent_cli_enrichment( f"Enriched {totals['applied']} node(s) in {totals['batches']} batch(es); {totals['bridges']} bridge edge(s).", f"⚠️ {totals['intent_length_violations']} intent(s) were outside the recommended 2-3 sentences." if totals["intent_length_violations"] else "All applied intents met the recommended 2-3 sentence length.", f"{rem} still un-enriched." if rem else "Nothing left to enrich.", + f"{pending_workflows} feature workflow file(s) still need source-backed steps." if pending_workflows and not rem else "", f"Dense embeddings could not be completed: {embedding_error}" if embedding_error else _embedding_summary(loader), ] + retry, - progress={**progress, "remaining": rem, "embedding_backend": loader.vector_store.backend}, as_json=as_json) + progress={**progress, "remaining": rem, "feature_workflows_pending": pending_workflows, "embedding_backend": loader.vector_store.backend}, as_json=as_json) return status @@ -186,10 +183,10 @@ def _emit_manual_enrichment_handoff( ], progress=progress, as_json=as_json) return STATUS_NEEDS_ENRICHMENT - -def _emit_enrichment_done(loader: GraphLoader, total: int, enriched: int, excluded: int, registry: Any, as_json: bool) -> str: +def _emit_enrichment_done(loader: GraphLoader, total: int, enriched: int, excluded: int, registry: Any, as_json: bool, feature_stats: Optional[Dict[str, Any]] = None) -> str: embedding_error = _embedding_failure(loader) - status = STATUS_NEEDS_EMBEDDINGS if embedding_error else STATUS_DONE + pending_workflows = int((feature_stats or {}).get("pending") or 0) + status = STATUS_NEEDS_EMBEDDINGS if embedding_error else (STATUS_NEEDS_FEATURE_WORKFLOWS if pending_workflows else STATUS_DONE) lines = [ f"{total} nodes across {len(registry)} layers. {enriched} enriched from source; {excluded} not eligible (utility bucket and prose nodes).", "", @@ -200,20 +197,22 @@ def _emit_enrichment_done(loader: GraphLoader, total: int, enriched: int, exclud "Run `tldrgraph init` again after fixing model access.", ]) else: - lines.extend([ - _embedding_summary(loader), - "", - ' tldrgraph query ""', - ' tldrgraph trace "" ""', - " tldrgraph layers", - " tldrgraph ui --serve", - ]) + workflow_lines = [ + f"{pending_workflows} feature workflow file(s) still need source-backed steps.", + " 1. Open .tldrgraph/features.yaml", + " 2. Complete each pending .tldrgraph/workflows/.yaml", + " 3. Run: tldrgraph init", + ] if pending_workflows else ["Feature workflow files are complete."] + lines.extend([_embedding_summary(loader), "", *workflow_lines, "", + ' tldrgraph query ""', + ' tldrgraph trace "" ""', + " tldrgraph layers", " tldrgraph ui --serve"]) emit_status(status, "embeddings" if embedding_error else "enrichment", lines, progress={"total_nodes": total, "enriched": enriched, "remaining": 0, + "feature_workflows_pending": pending_workflows, "embedding_backend": loader.vector_store.backend}, as_json=as_json) return status - def _handle_enrichment_step( path: str, root: str, @@ -226,6 +225,7 @@ def _handle_enrichment_step( agent_model: Optional[str], as_json: bool, llm_links: bool, + feature_stats: Optional[Dict[str, Any]] = None, ) -> str: candidates = enrichment_candidates(loader, compute_degrees(loader.graph)) total = loader.graph.number_of_nodes() @@ -240,7 +240,7 @@ def _handle_enrichment_step( if llm_links: link_status = run_llm_link_step(path, root, loader, agent_cli, agent_model, as_json, emit_status) if link_status: return link_status - return _emit_enrichment_done(loader, total, enriched, excluded, registry, as_json) + return _emit_enrichment_done(loader, total, enriched, excluded, registry, as_json, feature_stats) planned = min(len(candidates), max_nodes) if max_nodes else len(candidates) rounds = (planned + batch_size - 1) // batch_size @@ -255,6 +255,8 @@ def _handle_enrichment_step( "approval_persisted": enrichment_approval_is_active(path, candidates), } + if not agent_cli: + return _emit_enrichment_done(loader, total, enriched, excluded, registry, as_json, feature_stats) agent_marker = agent_runner.nesting_marker() auto_agent_approved = bool(agent_marker) and not max_nodes authorized = assume_yes or enrichment_approval_is_active(path, candidates) or auto_agent_approved @@ -270,10 +272,9 @@ def _handle_enrichment_step( remember_full_enrichment_approval(path, candidates) progress["approval_persisted"] = True - if agent_cli: - res = _run_agent_cli_enrichment(path, loader, batch_size, max_nodes, agent_model, progress, as_json, llm_links) - if res is not None: - return res + res = _run_agent_cli_enrichment(path, loader, batch_size, max_nodes, agent_model, progress, as_json, llm_links, feature_stats) + if res is not None: + return res return _emit_manual_enrichment_handoff(path, root, loader, candidates, progress, batch_size, max_nodes, as_json) @@ -294,7 +295,7 @@ def _ensure_layers_configured( notes: List[str] = [] registry, cfg_path, source = auto_configure_layers( - path, force=relayer and not applied_cfg, use_agent=agent_cli, + path, force=relayer and not applied_cfg, use_llm=agent_cli, use_agent=agent_cli, agent_model=agent_model, notes=notes, ) for note in notes: @@ -323,7 +324,6 @@ def _ensure_layers_configured( return registry, cfg_path, source - def _report_enrichment_applied_status(applied: Optional[Dict[str, Any]], as_json: bool) -> None: if not applied or as_json: return @@ -336,7 +336,6 @@ def _report_enrichment_applied_status(applied: Optional[Dict[str, Any]], as_json preview = ", ".join(sorted(set(applied["unresolved"]))[:4]) click.echo(f" ⚠️ {len(applied['unresolved'])} call target(s) matched nothing above the score floor: {preview}") - def init_pipeline( path: str, assume_yes: bool, @@ -367,7 +366,7 @@ def init_pipeline( loader._run_graphify() loader.file_hashes = loader._load_file_hashes() - registry, cfg_path, source = _ensure_layers_configured(path, root, relayer, agent_cli, agent_model, as_json) + registry, _, _ = _ensure_layers_configured(path, root, relayer, agent_cli, agent_model, as_json) if registry is None: return STATUS_NEEDS_LAYERS @@ -388,8 +387,12 @@ def init_pipeline( if link_applied and not as_json: click.echo(f"🔗 Applied {len(link_applied['applied'])} LLM route link(s)") + feature_stats = generate_feature_workflow_files(path, loader.graph, agent_model=agent_model, use_agent=False) + if not as_json: + click.echo(f"🧭 Feature workflows: {feature_stats['generated']} generated, {feature_stats['pending']} pending across {feature_stats['features']} feature(s)") + if feature_stats["pending"]: + click.echo(" Current agent must complete pending .tldrgraph/workflows/*.yaml files from source evidence.") + generate_visualizer_html(path) - return _handle_enrichment_step( - path, root, loader, registry, assume_yes, batch_size, max_nodes, agent_cli, agent_model, as_json, llm_links - ) + return _handle_enrichment_step(path, root, loader, registry, assume_yes, batch_size, max_nodes, agent_cli, agent_model, as_json, llm_links, feature_stats) diff --git a/tldrgraph/feature_workflow_agent.py b/tldrgraph/feature_workflow_agent.py new file mode 100644 index 0000000..402afdd --- /dev/null +++ b/tldrgraph/feature_workflow_agent.py @@ -0,0 +1,70 @@ +"""Agent prompts for source-backed feature workflow files.""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + +from . import agent_runner +from .feature_workflows import relative_workflow_path + + +def _run_json(root: str, prompt: str, agent_model: Optional[str]) -> Optional[Dict[str, Any]]: + agent = agent_runner.find_agent_cli(respect_nesting=False) + if agent is None: + return None + try: + raw = agent_runner.run_agent_json(agent, prompt, root, model=agent_model) + except agent_runner.AgentError: + return None + return raw if isinstance(raw, dict) else None + + +def generate_feature_manifest( + root: str, + graph_hash: str, + candidates: List[Dict[str, Any]], + agent_model: Optional[str], +) -> Optional[Dict[str, Any]]: + prompt = ( + "Read this source-backed graph evidence and return JSON only. " + "List the project's major user-facing and developer-facing features. " + "Prefer features that represent an end-to-end user or developer goal, not isolated UI or backend helpers. " + "Do not use curated workflows, BPMN, route-link workflow discovery, or guessed product flows. " + "Every feature must include at least one evidence item copied from candidates.\n\n" + "Return {\"features\":[{\"id\",\"title\",\"audience\",\"summary\",\"evidence\":[...] }]}.\n" + f"Evidence:\n{{'graph_hash': {graph_hash!r}, 'candidates': {candidates!r}}}" + ) + raw = _run_json(root, prompt, agent_model) + items = raw.get("features") if raw else None + if not isinstance(items, list): + return None + return {"features": items} + + +def generate_feature_workflow( + root: str, + graph_hash: str, + feature: Dict[str, Any], + evidence_nodes: List[Dict[str, Any]], + agent_model: Optional[str], +) -> Optional[Dict[str, Any]]: + prompt = ( + "Create one saved Feature Workflow Explorer workflow as JSON only. " + "It must describe the FULL feature flow from the user's button/menu/form action all the way to the final response or UI update. " + "Do not summarize away important hops or combine unrelated source symbols into one vague step. " + "When source evidence supports it, include the path from the user-facing screen/component through request/client code, " + "event handler, validation, request payload construction, API route or controller, middleware/auth, service/use-case logic, " + "persistence/database, background jobs, external services, response payload creation, client response parsing, state update, " + "toast/navigation/rendered result, and any user-visible error or success handling. " + "Use simple language for non-technical users and vibe coders. " + "Do not use curated workflows, BPMN generation, route-link workflow discovery, or guessed steps. " + "Every step must include evidence copied from the provided source-backed nodes. " + "If a frontend-to-backend or backend-to-frontend bridge is not present in evidence, include only the proven side and make the feature pending rather than inventing missing steps.\n\n" + "Return exactly {\"workflow\":{\"schema\":\"codechakra/feature-workflow@1\",\"graph_hash\":...,\"feature_id\":...," + "\"title\":...,\"summary\":...,\"status\":\"generated\",\"steps\":[{\"number\":1,\"title\":...,\"text\":...,\"evidence\":[...]}]}}.\n" + f"Feature:\n{feature!r}\nWorkflow path: {relative_workflow_path(str(feature.get('id') or 'feature'))}\n" + f"Graph hash: {graph_hash}\nEvidence nodes:\n{evidence_nodes!r}" + ) + raw = _run_json(root, prompt, agent_model) + workflow = raw.get("workflow") if raw else None + return workflow if isinstance(workflow, dict) else None diff --git a/tldrgraph/feature_workflow_loader.py b/tldrgraph/feature_workflow_loader.py new file mode 100644 index 0000000..5390806 --- /dev/null +++ b/tldrgraph/feature_workflow_loader.py @@ -0,0 +1,177 @@ +"""Load saved feature workflow files for the visualizer.""" + +from __future__ import annotations + +import os +from typing import Any, Dict, List, Optional, Tuple + +from .cli_enrichment import read_payload +from .feature_workflows import ( + FEATURE_SCHEMA, + WORKFLOW_SCHEMA, + features_path, + relative_workflow_path, + validate_workflow, + workflow_path, +) + + +def _humanize(text: str) -> str: + return str(text or "").replace("_", " ").replace("-", " ").title() or "Project Feature" + + +def _pending_workflow(feature: Dict[str, Any], reason: str) -> Dict[str, Any]: + return { + "schema": WORKFLOW_SCHEMA, + "graph_hash": feature.get("graph_hash", ""), + "feature_id": feature.get("id"), + "title": feature.get("title") or _humanize(feature.get("id")), + "summary": feature.get("summary") or reason, + "status": "pending", + "pending_reason": reason, + "steps": [], + } + + +def load_feature_manifest(root: str) -> Tuple[Optional[Dict[str, Any]], str]: + data = read_payload(features_path(root)) + if data is None: + return None, "missing_features" + if not isinstance(data, dict) or data.get("schema") != FEATURE_SCHEMA: + return None, "invalid_features" + features = data.get("features") + if not isinstance(features, list): + return None, "invalid_features" + if not features: + return data, "empty_features" + return data, "ready" + + +def load_saved_feature_workflows(root: str) -> Dict[str, Any]: + manifest, state = load_feature_manifest(root) + if not manifest: + return {"state": state, "workflows": []} + + workflows = [_visualizer_workflow(root, f) for f in manifest.get("features", []) if isinstance(f, dict)] + ready_count = sum(1 for wf in workflows if wf.get("status") == "generated") + return { + "state": "ready" if workflows else "empty_features", + "graph_hash": manifest.get("graph_hash"), + "workflows": workflows, + "ready_count": ready_count, + "pending_count": len(workflows) - ready_count, + } + + +def _visualizer_workflow(root: str, feature: Dict[str, Any]) -> Dict[str, Any]: + feature_id = str(feature.get("id") or "") + path = workflow_path(root, feature_id) + data = read_payload(path) + if not os.path.isfile(path): + data = _pending_workflow(feature, "Workflow file has not been generated yet.") + elif not isinstance(data, dict) or not validate_workflow(data): + data = _pending_workflow(feature, "Workflow file is invalid or incomplete.") + + status = str(data.get("status") or feature.get("status") or "pending") + steps = _visualizer_steps(data.get("steps") if isinstance(data, dict) else []) + return { + "id": feature_id, + "title": data.get("title") or feature.get("title") or _humanize(feature_id), + "category": feature.get("audience") or "Feature", + "root_node": feature.get("title") or feature_id, + "root_id": _first_node_id(steps, feature), + "file": _first_file(steps, feature), + "layer_id": "feature", + "layer": "Feature Workflow", + "summary": data.get("summary") or feature.get("summary") or "", + "step_count": len(steps), + "layers_involved": [], + "node_ids": [s["node_id"] for s in steps if s.get("node_id")], + "steps": steps, + "support": [], + "status": status, + "pending_reason": data.get("pending_reason") or ("" if status == "generated" else "Workflow is pending."), + "workflow_path": relative_workflow_path(feature_id), + "process": _simple_process(feature_id, steps, status), + } + + +def _visualizer_steps(raw_steps: Any) -> List[Dict[str, Any]]: + if not isinstance(raw_steps, list): + return [] + steps = [] + for index, step in enumerate(raw_steps, 1): + if not isinstance(step, dict): + continue + evidence = step.get("evidence") or [] + ev = evidence[0] if evidence and isinstance(evidence[0], dict) else {} + steps.append({ + "step_number": int(step.get("number") or index), + "node_id": str(ev.get("node_id") or ""), + "symbol": ev.get("symbol") or step.get("title") or f"Step {index}", + "display_label": step.get("title") or ev.get("symbol") or f"Step {index}", + "file": ev.get("file") or "", + "layer_id": "feature", + "layer": "Feature Workflow", + "type": "feature_step", + "intent": step.get("text") or "", + "code_start": int(ev.get("code_start") or ev.get("line") or 0), + "code_end": int(ev.get("code_end") or ev.get("line") or 0), + "evidence": evidence, + }) + return steps + + +def _simple_process(feature_id: str, steps: List[Dict[str, Any]], status: str) -> Dict[str, Any]: + elements = [_event(f"{feature_id}__start", "start", "Start the feature", 0)] + flows = [] + previous = elements[0]["id"] + for step in steps: + element_id = f"{feature_id}__step_{step['step_number']}" + elements.append({ + "id": element_id, + "kind": "task", + "label": step.get("intent") or step.get("display_label") or step.get("symbol"), + "detail": step.get("symbol") or "", + "lane": "system", + "step": step["step_number"], + "step_title": step.get("display_label") or step.get("symbol"), + "file": step.get("file") or "", + "line": step.get("code_start") or 0, + "node_id": step.get("node_id") or None, + "minor": False, + }) + flows.append({"source": previous, "target": element_id, "label": "", "kind": "sequence"}) + previous = element_id + label = "Workflow pending" if status != "generated" else "Feature workflow complete" + elements.append(_event(f"{feature_id}__finish", "end", label, len(steps) + 1)) + flows.append({"source": previous, "target": elements[-1]["id"], "label": "", "kind": "sequence"}) + return { + "lanes": [ + {"id": "user", "name": "You", "note": "What a person starts"}, + {"id": "system", "name": "TLDRGraph", "note": "What the source-backed workflow does"}, + ], + "elements": elements, + "flows": flows, + } + + +def _event(event_id: str, kind: str, label: str, step: int) -> Dict[str, Any]: + return { + "id": event_id, "kind": kind, "label": label, "detail": "", "lane": "user" if kind == "start" else "system", + "step": step, "line": 0, "node_id": None, "minor": False, + } + + +def _first_node_id(steps: List[Dict[str, Any]], feature: Dict[str, Any]) -> str: + if steps and steps[0].get("node_id"): + return steps[0]["node_id"] + evidence = feature.get("evidence") or [] + return str((evidence[0] if evidence else {}).get("node_id") or "") + + +def _first_file(steps: List[Dict[str, Any]], feature: Dict[str, Any]) -> str: + if steps and steps[0].get("file"): + return steps[0]["file"] + evidence = feature.get("evidence") or [] + return str((evidence[0] if evidence else {}).get("file") or "") diff --git a/tldrgraph/feature_workflows.py b/tldrgraph/feature_workflows.py new file mode 100644 index 0000000..22386a0 --- /dev/null +++ b/tldrgraph/feature_workflows.py @@ -0,0 +1,330 @@ +"""Saved feature workflow generation and loading.""" + +from __future__ import annotations + +import hashlib +import os +import re +from typing import Any, Dict, List, Tuple + +import networkx as nx + +from .cli_enrichment import read_payload, write_payload +from .layers import layer_id_of + +FEATURES_FILENAME = "features.yaml" +WORKFLOWS_DIRNAME = "workflows" +FEATURE_SCHEMA = "codechakra/features@1" +WORKFLOW_SCHEMA = "codechakra/feature-workflow@1" +WORKFLOW_GENERATOR = "feature-workflow-agent-owned@3" +BANNED_WORKFLOW_RELATIONS = {"llm_http_route_link", "http_route_link", "calls_endpoint"} +SKIP_DIRS = (".tldrgraph/", "tests/", "test/", "spec/", "__tests__/", "node_modules/", "dist/", "build/", "vendor/", "migrations/") + + +def features_path(root: str) -> str: + return os.path.join(root, ".tldrgraph", FEATURES_FILENAME) + + +def workflows_dir(root: str) -> str: + return os.path.join(root, ".tldrgraph", WORKFLOWS_DIRNAME) + + +def workflow_path(root: str, feature_id: str) -> str: + return os.path.join(workflows_dir(root), f"{feature_id}.yaml") + + +def relative_workflow_path(feature_id: str) -> str: + return os.path.join(".tldrgraph", WORKFLOWS_DIRNAME, f"{feature_id}.yaml") + + +def graph_hash(graph: nx.DiGraph) -> str: + """Stable content-ish hash for deciding when feature files are stale.""" + parts: List[str] = [] + for nid, node in sorted(graph.nodes(data=True), key=lambda item: str(item[0])): + parts.append("|".join([ + str(nid), + str(node.get("label") or ""), + str(node.get("file") or ""), + str(node.get("source_location") or ""), + str(node.get("intent") or ""), + ])) + for src, dst, data in sorted(graph.edges(data=True), key=lambda item: (str(item[0]), str(item[1]))): + relation = str(data.get("relation") or "calls") + if relation in BANNED_WORKFLOW_RELATIONS: + continue + parts.append(f"{src}>{dst}:{relation}") + return hashlib.sha256("\n".join(parts).encode("utf-8")).hexdigest() + + +def _clean_path(path: str) -> str: + return (path or "").replace("\\", "/") + + +def _is_candidate_node(node: Dict[str, Any]) -> bool: + file_path = _clean_path(node.get("file")).lower() + if not file_path or any(part in file_path for part in SKIP_DIRS): + return False + if node.get("is_test"): + return False + if node.get("dead_code_status") in {"not_code", "dead"}: + return False + label = str(node.get("label") or "") + return bool(label.strip()) + + +def _source_line(node: Dict[str, Any]) -> int: + raw = node.get("source_location") or node.get("code_start") or "" + if isinstance(raw, int): + return raw + match = re.search(r"\d+", str(raw)) + return int(match.group(0)) if match else 0 + + +def _evidence(node_id: str, node: Dict[str, Any]) -> Dict[str, Any]: + line = _source_line(node) + return { + "node_id": str(node_id), "symbol": node.get("label") or str(node_id), + "file": node.get("file") or "", "line": line, "code_start": int(node.get("code_start") or line or 0), + "code_end": int(node.get("code_end") or line or 0), + } + + +def _slug(text: str, fallback: str) -> str: + slug = re.sub(r"[^a-z0-9]+", "_", text.lower()).strip("_") + return (slug or fallback)[:64] + + +def _humanize(text: str) -> str: + text = re.sub(r"\([^)]*\)", "", str(text or "")) + text = text.split(".")[-1] + text = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", text) + text = text.replace("_", " ").replace("-", " ").strip() + return " ".join(text.split()).title() or "Project Feature" + + +def _node_summary(node: Dict[str, Any]) -> str: + intent = str(node.get("intent") or "").strip() + if intent and not intent.lower().startswith("the symbol "): + return intent.split(".")[0].strip() + "." + return f"Follow how {_humanize(node.get('label'))} works through the project." + + +def _usable_out_degree(graph: nx.DiGraph, node_id: str) -> int: + ignored = BANNED_WORKFLOW_RELATIONS | {"contains", "rationale_for", "imports", "imports_from"} + return sum(1 for _, _, data in graph.out_edges(node_id, data=True) if data.get("relation") not in ignored) + + +def _root_score(graph: nx.DiGraph, node_id: str, node: Dict[str, Any]) -> Tuple[int, str]: + path = _clean_path(node.get("file")).lower() + label = str(node.get("label") or "") + score = _usable_out_degree(graph, node_id) * 3 - graph.in_degree(node_id) + audience = "developer" + if any(part in path for part in ("cli", "commands", ".github", "installer", "agent")): + score += 8 + if any(part in path for part in ("app/", "pages/", "components/", "controller", "routes", "api")): + score += 10 + audience = "user" + if re.search(r"^(main|run|init|scan|serve|start|build|generate)", label, re.IGNORECASE): + score += 7 + if _usable_out_degree(graph, node_id) < 1: + score -= 5 + return score, audience + + +def _candidate_roots(graph: nx.DiGraph, limit: int = 12) -> List[Tuple[str, str]]: + scored: List[Tuple[int, str, str]] = [] + for node_id, node in graph.nodes(data=True): + if not _is_candidate_node(node): + continue + score, audience = _root_score(graph, str(node_id), node) + if score <= 0: + continue + scored.append((score, str(node_id), audience)) + scored.sort(key=lambda item: (-item[0], item[1])) + return [(node_id, audience) for _, node_id, audience in scored[:limit]] + + +def _walk_feature_steps(graph: nx.DiGraph, root_id: str, max_steps: int = 12) -> List[str]: + chain = [root_id] + seen = {root_id} + current = root_id + while len(chain) < max_steps: + ranked: List[Tuple[int, str]] = [] + current_layer = layer_id_of(graph.nodes.get(current, {})) + for _, target, data in graph.out_edges(current, data=True): + target = str(target) + if target in seen or not _is_candidate_node(graph.nodes.get(target, {})): + continue + relation = data.get("relation") or "calls" + if relation in BANNED_WORKFLOW_RELATIONS: + continue + if relation in {"contains", "rationale_for", "imports", "imports_from"}: + continue + node = graph.nodes[target] + score = graph.out_degree(target) + 2 + if layer_id_of(node) != current_layer: + score += 10 + if node.get("file") != graph.nodes[current].get("file"): + score += 4 + ranked.append((score, target)) + if not ranked: + break + ranked.sort(key=lambda item: (-item[0], item[1])) + current = ranked[0][1] + chain.append(current) + seen.add(current) + return chain + + +def _fallback_features(graph: nx.DiGraph, current_hash: str) -> Dict[str, Any]: + features: List[Dict[str, Any]] = [] + used_ids = set() + for root_id, audience in _candidate_roots(graph): + node = graph.nodes[root_id] + base_id = _slug(str(node.get("label") or root_id), f"feature_{len(features) + 1}") + feature_id = base_id + counter = 2 + while feature_id in used_ids: + feature_id = f"{base_id}_{counter}" + counter += 1 + used_ids.add(feature_id) + title = _humanize(node.get("display_label") or node.get("label") or root_id) + features.append({ + "id": feature_id, + "title": title, + "audience": audience, + "summary": _node_summary(node), + "status": "pending", + "workflow_path": relative_workflow_path(feature_id), + "evidence": [_evidence(root_id, node)], + }) + return {"schema": FEATURE_SCHEMA, "graph_hash": current_hash, "features": features} + + +def _evidence_node(graph: nx.DiGraph, node_id: str) -> Dict[str, Any]: + node = graph.nodes[node_id] + return { + "evidence": _evidence(node_id, node), + "label": node.get("label"), + "intent": node.get("intent"), + "layer": node.get("layer"), + "layer_id": node.get("layer_id"), + "outgoing": [ + { + "relation": d.get("relation"), + "target": _evidence(str(t), graph.nodes[t]), + "intent": graph.nodes[t].get("intent"), + "layer": graph.nodes[t].get("layer"), + } + for _, t, d in graph.out_edges(node_id, data=True) + if ( + t in graph + and d.get("relation") not in BANNED_WORKFLOW_RELATIONS + and d.get("relation") not in {"contains", "rationale_for", "imports", "imports_from"} + and _is_candidate_node(graph.nodes[t]) + ) + ][:24], + } + + +def _workflow_evidence(graph: nx.DiGraph, feature: Dict[str, Any]) -> List[Dict[str, Any]]: + root_id = str((feature.get("evidence") or [{}])[0].get("node_id") or "") + return [_evidence_node(graph, node_id) for node_id in _walk_feature_steps(graph, root_id) if node_id in graph] + + +def _pending_workflow(feature: Dict[str, Any], graph: nx.DiGraph, current_hash: str, reason: str) -> Dict[str, Any]: + return { + "schema": WORKFLOW_SCHEMA, + "graph_hash": current_hash, + "generator": WORKFLOW_GENERATOR, + "feature_id": feature.get("id"), + "title": feature.get("title") or _humanize(feature.get("id")), + "summary": feature.get("summary") or reason, + "status": "pending", + "pending_reason": reason, + "evidence": feature.get("evidence") or [], + "evidence_nodes": _workflow_evidence(graph, feature), + "instructions": [ + "The same coding agent running `tldrgraph init` must complete this file.", + "Open every source file referenced in evidence_nodes before writing steps.", + "Start at the user's button/menu/form action when present.", + "Continue through request, backend work, response payload, client handling, and final UI update when proven.", + "Use only source-backed evidence; leave status pending if a hop is not proven.", + ], + "required_step_shape": {"number": 1, "title": "...", "text": "...", "evidence": ["copy evidence objects from evidence_nodes"]}, + "steps": [], + } + + +def validate_workflow(workflow: Dict[str, Any]) -> bool: + if workflow.get("schema") != WORKFLOW_SCHEMA: + return False + steps = workflow.get("steps") + if not isinstance(steps, list) or not steps: + return workflow.get("status") == "pending" + for step in steps: + if not isinstance(step, dict): + return False + evidence = step.get("evidence") + if not isinstance(evidence, list) or not evidence: + return False + for ev in evidence: + if not isinstance(ev, dict): + return False + if not ev.get("node_id") or not ev.get("file") or not ev.get("symbol"): + return False + return True + + +def generate_feature_workflow_files( + root: str, + graph: nx.DiGraph, + agent_model: Any = None, + use_agent: bool = True, +) -> Dict[str, Any]: + """Writes features.yaml and all stale/missing workflow files.""" + root = os.path.abspath(root) + current_hash = graph_hash(graph) + manifest = _fallback_features(graph, current_hash) + reason = "Feature workflow is pending for the current coding agent to complete from source evidence." + + os.makedirs(workflows_dir(root), exist_ok=True) + generated = 0 + pending = 0 + updated_features = [] + for feature in manifest.get("features", []): + if not isinstance(feature, dict) or not feature.get("id"): + continue + feature_id = _slug(str(feature["id"]), f"feature_{len(updated_features) + 1}") + feature = {**feature, "id": feature_id, "workflow_path": relative_workflow_path(feature_id)} + existing = read_payload(workflow_path(root, feature_id)) + stale = ( + not isinstance(existing, dict) + or existing.get("graph_hash") != current_hash + or existing.get("generator") != WORKFLOW_GENERATOR + ) + workflow = existing if isinstance(existing, dict) and not stale else None + if workflow is None: + workflow = _pending_workflow(feature, graph, current_hash, reason) + if not validate_workflow(workflow): + workflow = _pending_workflow(feature, graph, current_hash, "Saved workflow is invalid or incomplete.") + write_payload(workflow_path(root, feature_id), workflow) + status = str(workflow.get("status") or "pending") + feature["status"] = status + if status == "generated": + generated += 1 + else: + pending += 1 + updated_features.append(feature) + + manifest = {"schema": FEATURE_SCHEMA, "graph_hash": current_hash, "features": updated_features} + write_payload(features_path(root), manifest) + return {"features": len(updated_features), "generated": generated, "pending": pending, + "graph_hash": current_hash, "agent_reason": ""} + + +def load_feature_manifest(root: str) -> Tuple[Optional[Dict[str, Any]], str]: + from . import feature_workflow_loader as loader; return loader.load_feature_manifest(root) + +def load_saved_feature_workflows(root: str) -> Dict[str, Any]: + from . import feature_workflow_loader as loader; return loader.load_saved_feature_workflows(root) diff --git a/tldrgraph/installer_contract.py b/tldrgraph/installer_contract.py index dcf52fb..8e7c409 100644 --- a/tldrgraph/installer_contract.py +++ b/tldrgraph/installer_contract.py @@ -32,22 +32,20 @@ def generate_layers_prose(registry: Optional[LayerRegistry] = None) -> str: return "\n".join(lines) -_LOOP = """One command does everything -- layers, extraction, enrichment, LLM route links, and embeddings: +_LOOP = """One command does extraction, saved feature workflows, and embeddings: ```bash -tldrgraph init # agents auto-approve full runs; terminals may ask once -tldrgraph init --yes # non-agent explicit approval when confirmation asks for it +tldrgraph init ``` -`init` automatically detects a supported agent CLI, uses 200-node enrichment batches, -infers evidence-backed frontend/backend route links, and downloads/builds dense embeddings. -It never guesses: when no agent is usable it prints a manual layer, enrichment, or route-link handoff. +`init` does not use AI for architecture layer design, all-symbol enrichment, +route-link inference, or BPMN workflow generation by default. It only attempts +source-backed saved Feature Workflow Explorer files and writes pending states +when a full workflow cannot be generated. -Full approval is persisted across continuation runs. In a coding-agent session, -plain `tldrgraph init` approves the full campaign and nested-agent protection means -the host agent must read, answer, and apply every 200-node batch without asking again. -`--batch 200` means all nodes in chunks; `--limit 200` means only 200 total. Never add -`--limit`, `--no-llm-links`, or `--embeddings off` unless the user explicitly requests it. +`--agent-cli` is now explicit opt-in for architecture layer design and enrichment. +Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless +the user explicitly requests it. The underlying steps stay available for scripting: @@ -74,10 +72,11 @@ def generate_layers_prose(registry: Optional[LayerRegistry] = None) -> str: - **Complete `needs_llm_links` when shown.** Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend files, and write `.tldrgraph/llm_links_response.yaml` with `{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. - This is required for a complete init run unless the user explicitly requested - `--no-llm-links`. -- **Continue after approval until `status: done`.** A `needs_enrichment` batch is work to - process, not a reason to ask again. Do not add `--limit`, `--no-llm-links`, or `--embeddings off`.""" + This only appears when route-link inference was explicitly enabled with `--llm-links`. +- **Do not process `needs_layers` or `needs_enrichment` unless requested.** Those + states belong to explicit `--agent-cli` architecture/enrichment runs or older builds. +- **Continue until `status: done`.** Do not add `--limit`, `--agent-cli`, + `--llm-links`, or `--embeddings off`.""" _RESPONSE_SCHEMA = """```yaml - id: "" diff --git a/tldrgraph/propose_layers.py b/tldrgraph/propose_layers.py index 5926886..c6d0b9e 100644 --- a/tldrgraph/propose_layers.py +++ b/tldrgraph/propose_layers.py @@ -31,7 +31,7 @@ extracted_symbol_evidence, sample_repo_files, ) -from .layers import LayerRegistry, get_registry +from .layers import LayerRegistry, bootstrap_registry, get_registry REQUEST_FILENAME = "propose_layers_request.json" RESPONSE_FILENAME = "propose_layers_response.json" @@ -300,4 +300,6 @@ def auto_configure_layers( out_path = save_layer_config(root, registry) return registry, out_path, "llm_synthesis" - return None, None, NEEDS_LAYERS + registry = bootstrap_registry() + out_path = save_layer_config(root, registry) + return registry, out_path, "bootstrap" diff --git a/tldrgraph/visualizer/assets/app.js b/tldrgraph/visualizer/assets/app.js index 2abfd76..9dbb9ce 100644 --- a/tldrgraph/visualizer/assets/app.js +++ b/tldrgraph/visualizer/assets/app.js @@ -2649,10 +2649,11 @@ function getLayerColor(layerId) { function initWorkflowsExplorer() { const workflows = DATA.workflows || []; const badgeEl = document.getElementById('flows-badge-count'); - if (badgeEl) badgeEl.textContent = workflows.length; + const readyCount = workflows.filter(w => (w.status || 'generated') === 'generated').length; + if (badgeEl) badgeEl.textContent = readyCount; const countEl = document.getElementById('flows-list-count'); - if (countEl) countEl.textContent = `Workflows (${workflows.length})`; + if (countEl) countEl.textContent = `Workflows (${readyCount}/${workflows.length})`; // Setup search input const searchInput = document.getElementById('flows-search-input'); @@ -2707,6 +2708,7 @@ function renderWorkflowsList() { const listEl = document.getElementById('flows-list'); if (!listEl) return; + const allWorkflows = DATA.workflows || []; const workflows = (DATA.workflows || []).filter(w => { if (!flowSearchQuery) return true; const matchTitle = (w.title || '').toLowerCase().includes(flowSearchQuery); @@ -2721,10 +2723,19 @@ function renderWorkflowsList() { }); const countEl = document.getElementById('flows-list-count'); - if (countEl) countEl.textContent = `Workflows (${workflows.length})`; + const readyCount = workflows.filter(w => (w.status || 'generated') === 'generated').length; + if (countEl) countEl.textContent = `Workflows (${readyCount}/${workflows.length})`; if (workflows.length === 0) { - listEl.innerHTML = `
No matching workflows found.
`; + const state = (DATA.workflow_state || {}).state || 'missing_features'; + const messages = { + missing_features: 'No feature manifest found. Run tldrgraph init to create .tldrgraph/features.yaml.', + invalid_features: '.tldrgraph/features.yaml is invalid. Run tldrgraph init to refresh it.', + empty_features: 'No features were saved for this project yet.', + ready: allWorkflows.length ? 'No matching saved workflows found.' : 'No saved feature workflows found.', + }; + listEl.innerHTML = `
${escapeHtml(messages[state] || messages.ready)}
`; + selectWorkflow(null); return; } @@ -2739,13 +2750,13 @@ function renderWorkflowsList() {
${escapeHtml(w.title)} - ${w.step_count} steps + ${(w.status || 'generated') === 'generated' ? `${w.step_count} steps` : 'pending'}
- ${escapeHtml(w.layer || 'Layer')} + ${escapeHtml(w.category || w.layer || 'Feature')}
${escapeHtml(w.summary || '')}
-
${layerBadges}
+
${(w.status || 'generated') === 'generated' ? layerBadges : `${escapeHtml(w.pending_reason || 'Workflow file pending')}`}
`; }).join(''); @@ -3668,6 +3679,9 @@ function selectWorkflow(flowId) { if (!w) { if (emptyState) emptyState.style.display = 'flex'; if (headerCard) headerCard.style.display = 'none'; + flowNodes = []; + flowEdges = []; + requestFlowFrame(); return; } @@ -3688,8 +3702,12 @@ function selectWorkflow(flowId) { badgeEl.style.background = getLayerColor(w.layer_id); } if (summaryEl) summaryEl.textContent = w.summary; - if (stepsMetaEl) stepsMetaEl.textContent = `${w.step_count} Logical Steps`; - if (entryMetaEl) entryMetaEl.textContent = `Starts with: ${w.root_node || w.title}`; + if (stepsMetaEl) stepsMetaEl.textContent = (w.status || 'generated') === 'generated' + ? `${w.step_count} Saved Steps` + : 'Workflow Pending'; + if (entryMetaEl) entryMetaEl.textContent = (w.status || 'generated') === 'generated' + ? `Starts with: ${w.root_node || w.title}` + : (w.pending_reason || 'The workflow file has not been generated yet.'); if (layersMetaEl) { layersMetaEl.innerHTML = (w.layers_involved || []).map(lname => { diff --git a/tldrgraph/visualizer/data.py b/tldrgraph/visualizer/data.py index aab051f..0dca344 100644 --- a/tldrgraph/visualizer/data.py +++ b/tldrgraph/visualizer/data.py @@ -18,11 +18,10 @@ import networkx as nx +from ..feature_workflow_loader import load_saved_feature_workflows from ..hierarchy import is_test_node from ..layer_config import load_layer_config from ..layers import get_registry -from .flows_data import extract_visualizer_workflows -from .bpmn_data import attach_bpmn_processes from .palette import FALLBACK_COLOR, palette_at from .source import SourceIndex, language_for, symbol_name @@ -372,9 +371,8 @@ def prepare_visualizer_data(root_dir: str) -> Dict[str, Any]: child_edges, module_edges = _build_edges(raw_edges, nodes_by_id, modules_by_id) modules = _serialize_modules(modules_by_id) - graph = _build_nx_graph(raw_nodes, raw_edges) - workflows = extract_visualizer_workflows(graph, nodes_by_id, sources) - attach_bpmn_processes(root_dir, workflows, graph, nodes_by_id) + workflow_payload = load_saved_feature_workflows(root_dir) + workflows = workflow_payload["workflows"] active_layer_ids = {m["layer_id"] for m in modules} layers = sorted((l for l in layer_map.values() if l["id"] in active_layer_ids), key=lambda x: x["order"]) @@ -385,6 +383,7 @@ def prepare_visualizer_data(root_dir: str) -> Dict[str, Any]: "modules": modules, "nodes": list(nodes_by_id.values()), "workflows": workflows, + "workflow_state": {k: v for k, v in workflow_payload.items() if k != "workflows"}, "module_edges": module_edges, "child_edges": child_edges, "stats": { From b6d1ff1c6a27190a6698759fb1ad65f989f4cecb Mon Sep 17 00:00:00 2001 From: Ashwani Kharwar Date: Mon, 14 Sep 2026 13:28:48 +0530 Subject: [PATCH 02/13] refactor: delegate feature workflow generation to source subagents Replace heuristic workflow generation with a request/response handoff, add workflow validation and endpoint-aware evidence traversal, and update CLI contracts, agent instructions, tests, and visualizer messaging. --- .agents/skills/tldrgraph-init/SKILL.md | 27 +- .claude/commands/tldrgraph-init.md | 27 +- .cursor/commands/tldrgraph-init.md | 27 +- AGENTS.md | 12 +- AGENT_CONTRACT.md | 23 +- tests/test_auto_agent.py | 46 ++- tests/test_feature_workflows.py | 380 ++++++++++++++++++----- tests/test_visualizer.py | 12 +- tldrgraph/agent_commands.py | 39 ++- tldrgraph/cli_pipeline.py | 21 +- tldrgraph/feature_workflow_agent.py | 70 ----- tldrgraph/feature_workflow_bridges.py | 163 ++++++++++ tldrgraph/feature_workflow_handoff.py | 324 +++++++++++++++++++ tldrgraph/feature_workflow_loader.py | 7 + tldrgraph/feature_workflow_validation.py | 159 ++++++++++ tldrgraph/feature_workflows.py | 180 ++--------- tldrgraph/installer_contract.py | 13 +- tldrgraph/visualizer/assets/app.js | 1 + 18 files changed, 1129 insertions(+), 402 deletions(-) delete mode 100644 tldrgraph/feature_workflow_agent.py create mode 100644 tldrgraph/feature_workflow_bridges.py create mode 100644 tldrgraph/feature_workflow_handoff.py create mode 100644 tldrgraph/feature_workflow_validation.py diff --git a/.agents/skills/tldrgraph-init/SKILL.md b/.agents/skills/tldrgraph-init/SKILL.md index 2240a04..2accc83 100644 --- a/.agents/skills/tldrgraph-init/SKILL.md +++ b/.agents/skills/tldrgraph-init/SKILL.md @@ -8,17 +8,16 @@ description: Build or continue this repository's TLDRGraph architecture graph (l In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select `tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles extraction, feature-file scaffolding, and embeddings: +One command handles extraction, feature-workflow handoff, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph does not ask another AI process to design architecture -layers, enrich every symbol, infer route links, or generate BPMN workflows. It -writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` -file per feature. The coding agent that ran `tldrgraph init` owns completing any -pending workflow files from source evidence. +TLDRGraph never launches an AI process for feature generation and never invents +heuristic features. When feature artifacts are missing or stale, it writes +`.tldrgraph/feature_workflows_request.yaml`. The coding agent that ran +`tldrgraph init` must delegate that request to a source-reading subagent. Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent `needs_confirmation` response explicitly asks for approval. `--batch 200` means @@ -28,16 +27,18 @@ explicitly requests it. ## Feature Workflow Explorer artifacts -Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. -When `init` reports pending feature workflows: - -1. Open `.tldrgraph/features.yaml`. -2. Open every pending `.tldrgraph/workflows/.yaml`. -3. Open the source files referenced in `evidence_nodes`; do not guess. -4. Replace `steps: []` with source-backed steps and set `status: generated`. +Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. +When `init` reports `status: needs_feature_workflows`: + +1. Open `.tldrgraph/feature_workflows_request.yaml`. +2. Spawn a source-reading subagent and delegate the entire request to it. +3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. ## `status: needs_layers` diff --git a/.claude/commands/tldrgraph-init.md b/.claude/commands/tldrgraph-init.md index 2240a04..2accc83 100644 --- a/.claude/commands/tldrgraph-init.md +++ b/.claude/commands/tldrgraph-init.md @@ -8,17 +8,16 @@ description: Build or continue this repository's TLDRGraph architecture graph (l In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select `tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles extraction, feature-file scaffolding, and embeddings: +One command handles extraction, feature-workflow handoff, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph does not ask another AI process to design architecture -layers, enrich every symbol, infer route links, or generate BPMN workflows. It -writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` -file per feature. The coding agent that ran `tldrgraph init` owns completing any -pending workflow files from source evidence. +TLDRGraph never launches an AI process for feature generation and never invents +heuristic features. When feature artifacts are missing or stale, it writes +`.tldrgraph/feature_workflows_request.yaml`. The coding agent that ran +`tldrgraph init` must delegate that request to a source-reading subagent. Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent `needs_confirmation` response explicitly asks for approval. `--batch 200` means @@ -28,16 +27,18 @@ explicitly requests it. ## Feature Workflow Explorer artifacts -Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. -When `init` reports pending feature workflows: - -1. Open `.tldrgraph/features.yaml`. -2. Open every pending `.tldrgraph/workflows/.yaml`. -3. Open the source files referenced in `evidence_nodes`; do not guess. -4. Replace `steps: []` with source-backed steps and set `status: generated`. +Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. +When `init` reports `status: needs_feature_workflows`: + +1. Open `.tldrgraph/feature_workflows_request.yaml`. +2. Spawn a source-reading subagent and delegate the entire request to it. +3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. ## `status: needs_layers` diff --git a/.cursor/commands/tldrgraph-init.md b/.cursor/commands/tldrgraph-init.md index 2240a04..2accc83 100644 --- a/.cursor/commands/tldrgraph-init.md +++ b/.cursor/commands/tldrgraph-init.md @@ -8,17 +8,16 @@ description: Build or continue this repository's TLDRGraph architecture graph (l In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select `tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles extraction, feature-file scaffolding, and embeddings: +One command handles extraction, feature-workflow handoff, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph does not ask another AI process to design architecture -layers, enrich every symbol, infer route links, or generate BPMN workflows. It -writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` -file per feature. The coding agent that ran `tldrgraph init` owns completing any -pending workflow files from source evidence. +TLDRGraph never launches an AI process for feature generation and never invents +heuristic features. When feature artifacts are missing or stale, it writes +`.tldrgraph/feature_workflows_request.yaml`. The coding agent that ran +`tldrgraph init` must delegate that request to a source-reading subagent. Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent `needs_confirmation` response explicitly asks for approval. `--batch 200` means @@ -28,16 +27,18 @@ explicitly requests it. ## Feature Workflow Explorer artifacts -Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. -When `init` reports pending feature workflows: - -1. Open `.tldrgraph/features.yaml`. -2. Open every pending `.tldrgraph/workflows/.yaml`. -3. Open the source files referenced in `evidence_nodes`; do not guess. -4. Replace `steps: []` with source-backed steps and set `status: generated`. +Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. +When `init` reports `status: needs_feature_workflows`: + +1. Open `.tldrgraph/feature_workflows_request.yaml`. +2. Spawn a source-reading subagent and delegate the entire request to it. +3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. ## `status: needs_layers` diff --git a/AGENTS.md b/AGENTS.md index fc74970..8bf471e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,14 +27,20 @@ Those are read-only and never trigger enrichment. extraction, embeddings, and writes the file-backed Feature Workflow Explorer artifacts: `.tldrgraph/features.yaml` plus `.tldrgraph/workflows/.yaml`. -Feature workflows are owned by the agent running `tldrgraph init`. If a workflow -file is pending, open `.tldrgraph/features.yaml`, then complete each pending -`.tldrgraph/workflows/.yaml` from its evidence. Workflow Explorer +TLDRGraph never invents heuristic features or launches an AI process for feature +generation. If `init` returns `needs_feature_workflows`, the agent running it must +open `.tldrgraph/feature_workflows_request.yaml`, spawn a source-reading subagent, +and have that subagent write `.tldrgraph/feature_workflows_response.yaml` with both +features and complete workflows. Run `tldrgraph init` again to validate and apply +the response; do not edit final feature/workflow files directly. Workflow Explorer reads only saved YAML, never curated blueprints, route-link workflow discovery, BPMN-derived generation, `discover_workflows()`, `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When writing feature workflows, start at the user's button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update. Do not skip proven steps. +Do not set `status: generated` unless the saved steps cover the end-to-end flow +for the proven feature boundary, and do not use route-link relations as workflow +evidence. Full workflow: `.claude/commands/tldrgraph-init.md` (identical copies live in every other agent directory). Schema: `.tldrgraph/AGENT_CONTRACT.md`. diff --git a/AGENT_CONTRACT.md b/AGENT_CONTRACT.md index ee4ccde..d5dbdd2 100644 --- a/AGENT_CONTRACT.md +++ b/AGENT_CONTRACT.md @@ -2,11 +2,10 @@ **Audience: the coding agent with this repository open** (Codex, Claude Code, Cursor, Antigravity). -TLDRGraph builds an architectural graph from the graphify AST export. By default, -`tldrgraph init` does not ask AI to design architecture layers, enrich every -symbol, infer route links, or generate BPMN workflows. The only default AI-shaped -artifact is the saved Feature Workflow Explorer YAML, and it must stay backed by -real source evidence. +TLDRGraph builds an architectural graph from the graphify AST export. It never +launches an AI process or invents heuristic features for Feature Workflow Explorer. +The coding agent running `tldrgraph init` owns delegating feature generation to a +source-reading subagent through the request/response handshake below. --- @@ -28,6 +27,7 @@ independent and does not require `--agent-cli`. | `needs_confirmation` | Only belongs to explicit `--agent-cli` enrichment. Ask before continuing. | | `needs_enrichment` | Only process if the user explicitly asked for full graph enrichment. | | `needs_embeddings` | Enrichment finished but the required dense model/index could not be built. Fix model access and rerun init. | +| `needs_feature_workflows` | Delegate `.tldrgraph/feature_workflows_request.yaml` to a source-reading subagent, write the response, and rerun init. | | `done` | Nothing left. Use `query` / `trace` / `layers`. | `--json` gives you the same thing machine-readably. The sections below document the file @@ -66,8 +66,10 @@ Request and response are **separate files**. Never write your answer back into | File | Written by | Read by | | --- | --- | --- | -| `.tldrgraph/features.yaml` | `tldrgraph init` | Workflow Explorer | -| `.tldrgraph/workflows/.yaml` | `tldrgraph init` | Workflow Explorer | +| `.tldrgraph/feature_workflows_request.yaml` | `tldrgraph init` | host coding agent and its subagent | +| `.tldrgraph/feature_workflows_response.yaml` | source-reading subagent | `tldrgraph init` | +| `.tldrgraph/features.yaml` | `tldrgraph init` after validation | Workflow Explorer | +| `.tldrgraph/workflows/.yaml` | `tldrgraph init` after validation | Workflow Explorer | The Workflow Explorer tab is intentionally file-driven. It must read only `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`; if those @@ -94,6 +96,13 @@ update, navigation/toast/rendered result, and visible success or error handling. Do not stop at only the frontend or only the backend when the source proves the handoff, and do not collapse multiple proven source hops into one vague step. +When `init` returns `needs_feature_workflows`, open the request and spawn a +source-reading subagent. Delegate the entire request: the subagent must generate +both the feature list and every complete workflow in +`.tldrgraph/feature_workflows_response.yaml`. Run `tldrgraph init` again so +TLDRGraph can validate and atomically apply the response. Never edit the final +manifest or workflow files directly, and never substitute heuristic features. + --- ## Request schema (`enrichment_request.yaml`) diff --git a/tests/test_auto_agent.py b/tests/test_auto_agent.py index 178a914..d524a43 100644 --- a/tests/test_auto_agent.py +++ b/tests/test_auto_agent.py @@ -55,20 +55,35 @@ def fake_agent(name: str = "fake") -> agent_runner.AgentCLI: def complete_pending_workflows(root: Path) -> None: - manifest = yaml.safe_load((root / ".tldrgraph" / "features.yaml").read_text(encoding="utf-8")) - for feature in manifest.get("features", []): - path = root / feature["workflow_path"] - workflow = yaml.safe_load(path.read_text(encoding="utf-8")) - evidence_nodes = workflow.get("evidence_nodes") or [] - evidence = (evidence_nodes[0] if evidence_nodes else {}).get("evidence") or (feature.get("evidence") or [{}])[0] - workflow["status"] = "generated" - workflow["steps"] = [{ - "number": 1, - "title": f"Run {feature['title']}", - "text": "The current agent completed this workflow from source evidence.", + state = root / ".tldrgraph" + request = yaml.safe_load((state / "feature_workflows_request.yaml").read_text(encoding="utf-8")) + evidence = request["candidates"][0]["root"] + response = { + "schema": "codechakra/feature-workflows-response@1", + "graph_hash": request["graph_hash"], + "features": [{ + "id": "run_sample_cli", + "title": "Run Sample CLI", + "audience": "developer", + "summary": "Run the sample command through its engine.", "evidence": [evidence], - }] - path.write_text(yaml.safe_dump(workflow, sort_keys=False), encoding="utf-8") + "workflow": { + "status": "generated", + "summary": "Invoke the command and return the engine result.", + "steps": [ + {"number": 1, "phase": "backend", "title": "Invoke command", + "text": "The developer invokes the command entrypoint.", "evidence": [evidence]}, + {"number": 2, "phase": "backend", "title": "Run engine", + "text": "The entrypoint runs the source-backed engine work.", "evidence": [evidence]}, + {"number": 3, "phase": "response", "title": "Return result", + "text": "The command returns the engine result.", "evidence": [evidence]}, + ], + }, + }], + } + (state / "feature_workflows_response.yaml").write_text( + yaml.safe_dump(response, sort_keys=False), encoding="utf-8" + ) @pytest.fixture @@ -601,13 +616,16 @@ def test_interactive_init_asks_once_then_finishes(monkeypatch, cli_repo, agent_a assert res.exit_code == 0, res.output assert res.output.count("Enrich now?") == 1 assert "status: needs_feature_workflows" in res.output + assert "Delegate the entire request to a source-reading subagent" in res.output def test_automatic_agent_keeps_json_output_parseable(monkeypatch, cli_repo, agent_allowed): _stub_agent_cli(monkeypatch) res = CliRunner().invoke(cli, ["init", str(cli_repo), "--yes", "--json"]) assert res.exit_code == 0, res.output - assert json.loads(res.stdout)["status"] == "needs_feature_workflows" + payload = json.loads(res.stdout) + assert payload["status"] == "needs_feature_workflows" + assert any("source-reading subagent" in line for line in payload["next_action"]) def test_embedding_failure_is_resumable(monkeypatch, cli_repo, agent_allowed): diff --git a/tests/test_feature_workflows.py b/tests/test_feature_workflows.py index f2f1e33..66629c9 100644 --- a/tests/test_feature_workflows.py +++ b/tests/test_feature_workflows.py @@ -1,46 +1,81 @@ """Feature Workflow Explorer saved-file contract.""" +import copy + +import pytest import yaml -def test_feature_workflow_files_are_written_from_source_evidence(loader, mini_repo): - from tldrgraph.feature_workflows import ( - FEATURES_FILENAME, - WORKFLOW_SCHEMA, - generate_feature_workflow_files, - load_saved_feature_workflows, - ) +def _step(number, phase, key, mini_repo, title=None, text=None, relation=None): + evidence = { + "node_id": mini_repo.nid(key), + "symbol": mini_repo.label(key), + "file": mini_repo.source_file(key), + "line": 1, + } + if relation: + evidence["relation"] = relation + return { + "number": number, + "phase": phase, + "title": title or f"{phase} step", + "text": text or f"Source-backed {phase} behavior.", + "evidence": [evidence], + } + + +def _write_valid_response(mini_repo, graph, **feature_overrides): + from tldrgraph.cli_enrichment import write_payload + from tldrgraph.feature_workflow_handoff import RESPONSE_FILENAME, RESPONSE_SCHEMA + from tldrgraph.feature_workflows import graph_hash + + feature = { + "id": "submit_case", + "title": "Submit Case", + "audience": "user", + "summary": "Submit and persist a case.", + "evidence": [_step(1, "user_action", "ui_page", mini_repo)["evidence"][0]], + "workflow": { + "status": "generated", + "summary": "Submit a case and render the response.", + "steps": [ + _step(1, "user_action", "ui_page", mini_repo, "Click submit"), + _step(2, "frontend", "ui_page", mini_repo, "Build request"), + _step(3, "request", "ui_page", mini_repo, "Send case"), + _step(4, "backend", "api_controller", mini_repo, "Receive case"), + _step(5, "backend", "svc_workflow", mini_repo, "Run workflow"), + _step(6, "persistence", "data_prisma", mini_repo, "Persist case"), + _step(7, "response", "api_controller", mini_repo, "Return payload"), + _step(8, "ui_update", "ui_page", mini_repo, "Render result"), + ], + }, + } + feature.update(feature_overrides) + return write_payload(str(mini_repo.tldrgraph_dir / RESPONSE_FILENAME), { + "schema": RESPONSE_SCHEMA, + "graph_hash": graph_hash(graph), + "features": [feature], + }) + + +def test_missing_feature_artifacts_create_host_subagent_request(loader, mini_repo): + from tldrgraph.feature_workflow_handoff import REQUEST_SCHEMA + from tldrgraph.feature_workflows import FEATURES_FILENAME, generate_feature_workflow_files graph = loader.load_or_extract(enrich_llm=False) - stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) + stats = generate_feature_workflow_files(str(mini_repo.root), graph) - assert stats["features"] > 0 + assert stats["features"] == 0 + assert stats["pending"] == 1 features_path = mini_repo.tldrgraph_dir / FEATURES_FILENAME - assert features_path.exists() - - manifest = yaml.safe_load(features_path.read_text(encoding="utf-8")) - assert manifest["schema"] == "codechakra/features@1" - assert manifest["features"] - - for feature in manifest["features"]: - workflow_file = mini_repo.root / feature["workflow_path"] - assert workflow_file.exists() - workflow = yaml.safe_load(workflow_file.read_text(encoding="utf-8")) - assert workflow["schema"] == WORKFLOW_SCHEMA - assert workflow["feature_id"] == feature["id"] - assert workflow["status"] == "pending" - assert workflow["steps"] == [] - assert "current coding agent" in workflow["pending_reason"] - assert workflow["evidence"] - assert workflow["evidence_nodes"] - assert "required_step_shape" in workflow - assert any("Open every source file" in line for line in workflow["instructions"]) - - payload = load_saved_feature_workflows(str(mini_repo.root)) - assert payload["state"] == "ready" - assert payload["ready_count"] == 0 - assert payload["pending_count"] == len(manifest["features"]) - assert all(wf["process"]["elements"] for wf in payload["workflows"]) + assert not features_path.exists() + request = yaml.safe_load(open(stats["request_path"], encoding="utf-8")) + assert request["schema"] == REQUEST_SCHEMA + assert request["graph_hash"] == stats["graph_hash"] + assert request["candidates"] + assert request["candidates"][0]["root"]["node_id"] + assert request["candidates"][0]["evidence_nodes"] + assert "delegate this entire request" in "\n".join(request["instructions"]) def test_feature_workflow_validation_rejects_steps_without_evidence(): @@ -54,20 +89,172 @@ def test_feature_workflow_validation_rejects_steps_without_evidence(): }) +def test_user_workflow_validation_rejects_shallow_generated_steps(mini_repo): + from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow + + assert not validate_workflow({ + "schema": WORKFLOW_SCHEMA, + "feature_id": "submit_case", + "status": "generated", + "evidence": [_step(1, "frontend", "ui_page", mini_repo)["evidence"][0]], + "steps": [ + _step(1, "frontend", "ui_page", mini_repo), + _step(2, "request", "ui_page", mini_repo), + _step(3, "ui_update", "ui_page", mini_repo), + ], + }) + + +def test_user_workflow_requires_backend_step_when_backend_evidence_exists(mini_repo): + from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow + + workflow = { + "schema": WORKFLOW_SCHEMA, + "feature_id": "submit_case", + "status": "generated", + "evidence": [_step(1, "frontend", "ui_page", mini_repo)["evidence"][0]], + "evidence_nodes": [{ + "evidence": _step(1, "frontend", "ui_page", mini_repo)["evidence"][0], + "outgoing": [{ + "relation": "calls", + "target": _step(2, "backend", "api_controller", mini_repo)["evidence"][0], + }], + }], + "steps": [ + _step(1, "user_action", "ui_page", mini_repo), + _step(2, "frontend", "ui_page", mini_repo), + _step(3, "request", "ui_page", mini_repo), + _step(4, "ui_update", "ui_page", mini_repo), + ], + } + + assert not validate_workflow(workflow) + + +def test_backend_only_generated_workflow_can_pass(mini_repo): + from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow + + assert validate_workflow({ + "schema": WORKFLOW_SCHEMA, + "feature_id": "case_workflow", + "status": "generated", + "evidence": [_step(1, "backend", "svc_workflow", mini_repo)["evidence"][0]], + "steps": [ + _step(1, "backend", "api_controller", mini_repo, "Receive command"), + _step(2, "backend", "svc_workflow", mini_repo, "Run service work"), + _step(3, "persistence", "data_prisma", mini_repo, "Persist case"), + _step(4, "response", "api_controller", mini_repo, "Return result"), + ], + }) + + +def test_generated_workflow_with_scaffold_markers_is_incomplete(mini_repo): + from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow + + assert not validate_workflow({ + "schema": WORKFLOW_SCHEMA, + "feature_id": "submit_case", + "status": "generated", + "pending_reason": "Feature workflow is pending for the current coding agent.", + "evidence": [_step(1, "frontend", "ui_page", mini_repo)["evidence"][0]], + "steps": [ + _step(1, "user_action", "ui_page", mini_repo), + _step(2, "frontend", "ui_page", mini_repo), + _step(3, "request", "ui_page", mini_repo), + _step(4, "backend", "api_controller", mini_repo), + _step(5, "response", "api_controller", mini_repo), + _step(6, "ui_update", "ui_page", mini_repo), + ], + }) + + +def test_route_link_relations_are_not_valid_workflow_evidence(mini_repo): + from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow + + workflow = { + "schema": WORKFLOW_SCHEMA, + "feature_id": "submit_case", + "status": "generated", + "evidence": [_step(1, "frontend", "ui_page", mini_repo)["evidence"][0]], + "steps": [ + _step(1, "user_action", "ui_page", mini_repo), + _step(2, "frontend", "ui_page", mini_repo), + _step(3, "request", "ui_page", mini_repo, relation="http_route_link"), + _step(4, "backend", "api_controller", mini_repo), + _step(5, "response", "api_controller", mini_repo), + _step(6, "ui_update", "ui_page", mini_repo), + ], + } + + assert not validate_workflow(workflow) + + +def test_feature_request_evidence_crosses_endpoint_context(tmp_path): + import networkx as nx + from tldrgraph.feature_workflows import generate_feature_workflow_files + + graph = nx.DiGraph() + graph.add_nodes_from([ + ("page", { + "label": "ChatPanel()", "file": "frontend/src/app/projects/page.tsx", + "layer_id": "ui", "layer": "UI", "source_location": "L10", + }), + ("api", { + "label": "streamBuildProgress()", "file": "frontend/src/services/api.ts", + "layer_id": "ui", "layer": "UI", "source_location": "L40", + }), + ("endpoint", { + "label": "POST /containers/:id/stream", "file": "backend/src/routes/containers.ts", + "layer_id": "api", "layer": "API", "source_location": "L80", + }), + ("handler", { + "label": "streamContainer()", "file": "backend/src/routes/containers.ts", + "layer_id": "api", "layer": "API", "source_location": "L90", + }), + ("service", { + "label": "streamMessage()", "file": "backend/src/services/opencode/OpencodeService.ts", + "layer_id": "service", "layer": "Service", "source_location": "L120", + }), + ("checkpoint", { + "label": "createPromptRollbackCheckpoint()", + "file": "backend/src/services/checkpoints.ts", + "layer_id": "data", "layer": "Data", "source_location": "L150", + }), + ]) + graph.add_edge("page", "api", relation="calls") + graph.add_edge("api", "endpoint", relation="calls_endpoint") + graph.add_edge("endpoint", "handler", relation="handled_by") + graph.add_edge("handler", "service", relation="calls") + graph.add_edge("service", "checkpoint", relation="calls") + + stats = generate_feature_workflow_files(str(tmp_path), graph) + request = yaml.safe_load(open(stats["request_path"], encoding="utf-8")) + candidate = next(item for item in request["candidates"] if item["root"]["node_id"] == "page") + node_ids = [node["evidence"]["node_id"] for node in candidate["evidence_nodes"]] + + assert node_ids[:6] == ["page", "api", "endpoint", "handler", "service", "checkpoint"] + api_node = next(node for node in candidate["evidence_nodes"] if node["evidence"]["node_id"] == "api") + assert api_node["endpoint_context"][0]["endpoint"]["node_id"] == "endpoint" + assert api_node["endpoint_context"][0]["handler"]["node_id"] == "handler" + assert all( + outgoing["relation"] not in {"llm_http_route_link", "http_route_link", "calls_endpoint"} + for node in candidate["evidence_nodes"] + for outgoing in node.get("outgoing", []) + ) + + def test_feature_workflows_do_not_spawn_agent_cli(monkeypatch, loader, mini_repo): - from tldrgraph import feature_workflow_agent - from tldrgraph.feature_workflows import generate_feature_workflow_files, workflow_path + from tldrgraph import agent_runner + from tldrgraph.feature_workflows import generate_feature_workflow_files - monkeypatch.setattr(feature_workflow_agent, "generate_feature_manifest", lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("spawned feature agent"))) - monkeypatch.setattr(feature_workflow_agent, "generate_feature_workflow", lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("spawned feature agent"))) + monkeypatch.setattr(agent_runner, "find_agent_cli", lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("spawned feature agent"))) graph = loader.load_or_extract(enrich_llm=False) - stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=True) - workflow = yaml.safe_load(open(workflow_path(str(mini_repo.root), "submitcasebutton"), encoding="utf-8")) + stats = generate_feature_workflow_files(str(mini_repo.root), graph) assert stats["agent_reason"] == "" - assert workflow["status"] == "pending" - assert "current coding agent" in workflow["pending_reason"] + assert stats["pending"] == 1 + assert stats["request_path"] def test_missing_workflow_file_becomes_pending_state(mini_repo): @@ -104,6 +291,7 @@ def test_missing_workflow_file_becomes_pending_state(mini_repo): def test_init_pipeline_writes_all_feature_workflows(monkeypatch, mini_repo): from tldrgraph.cli_pipeline import init_pipeline + from tldrgraph.feature_workflow_handoff import REQUEST_FILENAME from tldrgraph.feature_workflows import load_saved_feature_workflows monkeypatch.setattr("tldrgraph.graph_loader.GraphLoader._run_graphify", lambda self: None) @@ -124,62 +312,90 @@ def test_init_pipeline_writes_all_feature_workflows(monkeypatch, mini_repo): payload = load_saved_feature_workflows(str(mini_repo.root)) assert status == "needs_feature_workflows" - assert payload["workflows"] - assert payload["ready_count"] == 0 - assert payload["pending_count"] == len(payload["workflows"]) + assert payload["state"] == "missing_features" + assert not payload["workflows"] + assert (mini_repo.tldrgraph_dir / REQUEST_FILENAME).exists() def test_existing_generated_workflow_is_preserved(mini_repo): + from tldrgraph.feature_workflow_handoff import APPLIED_RESPONSE_FILENAME, RESPONSE_FILENAME from tldrgraph.feature_workflows import generate_feature_workflow_files, workflow_path - import yaml from tldrgraph.graph_loader import GraphLoader graph = GraphLoader(str(mini_repo.root)).load_or_extract(enrich_llm=False) - stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) - path = workflow_path(str(mini_repo.root), "submitcasebutton") - workflow = yaml.safe_load(open(path, encoding="utf-8")) - workflow["status"] = "generated" - workflow["steps"] = [{ - "number": 1, - "title": "Use the page", - "text": "The user starts from the case page.", - "evidence": [workflow["evidence_nodes"][0]["evidence"]], - }] - with open(path, "w", encoding="utf-8") as f: - yaml.safe_dump(workflow, f, sort_keys=False) - - stats = generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) + _write_valid_response(mini_repo, graph) + stats = generate_feature_workflow_files(str(mini_repo.root), graph) + path = workflow_path(str(mini_repo.root), "submit_case") + original = yaml.safe_load(open(path, encoding="utf-8")) + assert (mini_repo.tldrgraph_dir / APPLIED_RESPONSE_FILENAME).exists() + assert not (mini_repo.tldrgraph_dir / RESPONSE_FILENAME).exists() + + stats = generate_feature_workflow_files(str(mini_repo.root), graph) workflow = yaml.safe_load(open(path, encoding="utf-8")) assert stats["generated"] == 1 - assert workflow["status"] == "generated" + assert workflow == original assert workflow["steps"][0]["evidence"][0]["symbol"] == "SubmitCaseButton" -def test_stale_workflows_without_current_generator_are_regenerated_or_pending(mini_repo): +@pytest.mark.parametrize("invalid_case", [ + "malformed", "wrong_hash", "duplicate_id", "unknown_evidence", "shallow_workflow", + "banned_relation", +]) +def test_invalid_subagent_response_never_overwrites_valid_artifacts(mini_repo, invalid_case): from tldrgraph.cli_enrichment import write_payload - from tldrgraph.feature_workflows import generate_feature_workflow_files, graph_hash, workflow_path, WORKFLOW_SCHEMA + from tldrgraph.feature_workflow_handoff import REQUEST_FILENAME, RESPONSE_FILENAME + from tldrgraph.feature_workflows import features_path, generate_feature_workflow_files from tldrgraph.graph_loader import GraphLoader graph = GraphLoader(str(mini_repo.root)).load_or_extract(enrich_llm=False) - current_hash = graph_hash(graph) - write_payload(workflow_path(str(mini_repo.root), "submitcasebutton"), { - "schema": WORKFLOW_SCHEMA, - "graph_hash": current_hash, - "feature_id": "submitcasebutton", - "title": "Submit Case Button", - "summary": "Old shallow workflow.", - "status": "generated", - "steps": [{"number": 1, "title": "Old", "text": "Old.", "evidence": [{ - "node_id": mini_repo.nid("ui_page"), - "symbol": mini_repo.label("ui_page"), - "file": mini_repo.source_file("ui_page"), - "line": 1, - }]}], - }) + _write_valid_response(mini_repo, graph) + generate_feature_workflow_files(str(mini_repo.root), graph) + original = open(features_path(str(mini_repo.root)), encoding="utf-8").read() + + _write_valid_response(mini_repo, graph) + response_path = mini_repo.tldrgraph_dir / RESPONSE_FILENAME + if invalid_case == "malformed": + response_path.write_text("features: [", encoding="utf-8") + else: + response = yaml.safe_load(response_path.read_text(encoding="utf-8")) + feature = response["features"][0] + if invalid_case == "wrong_hash": + response["graph_hash"] = "stale" + elif invalid_case == "duplicate_id": + response["features"].append(copy.deepcopy(feature)) + elif invalid_case == "unknown_evidence": + feature["evidence"][0]["node_id"] = "missing-node" + elif invalid_case == "shallow_workflow": + feature["workflow"]["steps"] = feature["workflow"]["steps"][:2] + else: + feature["workflow"]["steps"][0]["evidence"][0]["relation"] = "http_route_link" + write_payload(str(response_path), response) + + stats = generate_feature_workflow_files(str(mini_repo.root), graph) + request = yaml.safe_load((mini_repo.tldrgraph_dir / REQUEST_FILENAME).read_text(encoding="utf-8")) + + assert stats["pending"] == 1 + assert stats["agent_reason"] + assert request["previous_response_error"] == stats["agent_reason"] + assert open(features_path(str(mini_repo.root)), encoding="utf-8").read() == original + + +def test_stale_generated_workflows_are_preserved_but_not_exposed(mini_repo): + from tldrgraph.feature_workflows import generate_feature_workflow_files, load_saved_feature_workflows, workflow_path + from tldrgraph.graph_loader import GraphLoader - generate_feature_workflow_files(str(mini_repo.root), graph, use_agent=False) - workflow = yaml.safe_load(open(workflow_path(str(mini_repo.root), "submitcasebutton"), encoding="utf-8")) + graph = GraphLoader(str(mini_repo.root)).load_or_extract(enrich_llm=False) + _write_valid_response(mini_repo, graph) + generate_feature_workflow_files(str(mini_repo.root), graph) + path = workflow_path(str(mini_repo.root), "submit_case") + original = open(path, encoding="utf-8").read() + + graph.nodes[mini_repo.nid("ui_page")]["intent"] = "Changed source-backed intent." + stats = generate_feature_workflow_files(str(mini_repo.root), graph) + payload = load_saved_feature_workflows(str(mini_repo.root)) - assert workflow["status"] == "pending" - assert workflow["steps"] == [] + assert stats["pending"] == 1 + assert open(path, encoding="utf-8").read() == original + assert payload["state"] == "stale_features" + assert payload["workflows"] == [] diff --git a/tests/test_visualizer.py b/tests/test_visualizer.py index 60ac310..3c11a34 100644 --- a/tests/test_visualizer.py +++ b/tests/test_visualizer.py @@ -435,7 +435,7 @@ def format_step(node_id, step_number): def test_saved_feature_generation_ignores_route_link_relations(): import networkx as nx import yaml - from tldrgraph.feature_workflows import generate_feature_workflow_files, load_saved_feature_workflows, workflow_path + from tldrgraph.feature_workflows import generate_feature_workflow_files graph = nx.DiGraph() nodes = { @@ -455,11 +455,9 @@ def test_saved_feature_generation_ignores_route_link_relations(): import tempfile with tempfile.TemporaryDirectory() as root: - generate_feature_workflow_files(root, graph, use_agent=True) - raw = yaml.safe_load(open(workflow_path(root, "orderspage"), encoding="utf-8")) - payload = load_saved_feature_workflows(root) + stats = generate_feature_workflow_files(root, graph) + request = yaml.safe_load(open(stats["request_path"], encoding="utf-8")) - assert payload["workflows"] - assert payload["workflows"][0]["status"] == "pending" - outgoing = raw["evidence_nodes"][0]["outgoing"] + page = next(item for item in request["candidates"] if item["root"]["node_id"] == "page") + outgoing = page["evidence_nodes"][0]["outgoing"] assert all(item["target"]["node_id"] != "handler" for item in outgoing) diff --git a/tldrgraph/agent_commands.py b/tldrgraph/agent_commands.py index d0ff3a1..f42ce76 100644 --- a/tldrgraph/agent_commands.py +++ b/tldrgraph/agent_commands.py @@ -123,14 +123,20 @@ class AgentTarget: extraction, embeddings, and writes the file-backed Feature Workflow Explorer artifacts: `.tldrgraph/features.yaml` plus `.tldrgraph/workflows/.yaml`. -Feature workflows are owned by the agent running `tldrgraph init`. If a workflow -file is pending, open `.tldrgraph/features.yaml`, then complete each pending -`.tldrgraph/workflows/.yaml` from its evidence. Workflow Explorer +TLDRGraph never invents heuristic features or launches an AI process for feature +generation. If `init` returns `needs_feature_workflows`, the agent running it must +open `.tldrgraph/feature_workflows_request.yaml`, spawn a source-reading subagent, +and have that subagent write `.tldrgraph/feature_workflows_response.yaml` with both +features and complete workflows. Run `tldrgraph init` again to validate and apply +the response; do not edit final feature/workflow files directly. Workflow Explorer reads only saved YAML, never curated blueprints, route-link workflow discovery, BPMN-derived generation, `discover_workflows()`, `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When writing feature workflows, start at the user's button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update. Do not skip proven steps. +Do not set `status: generated` unless the saved steps cover the end-to-end flow +for the proven feature boundary, and do not use route-link relations as workflow +evidence. Full workflow: `.claude/commands/{COMMAND_NAME}.md` (identical copies live in every other agent directory). Schema: `.tldrgraph/AGENT_CONTRACT.md`. @@ -149,17 +155,16 @@ class AgentTarget: In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select `tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. -One command handles extraction, feature-file scaffolding, and embeddings: +One command handles extraction, feature-workflow handoff, and embeddings: ```bash tldrgraph init ``` -By default TLDRGraph does not ask another AI process to design architecture -layers, enrich every symbol, infer route links, or generate BPMN workflows. It -writes `.tldrgraph/features.yaml` and one `.tldrgraph/workflows/.yaml` -file per feature. The coding agent that ran `tldrgraph init` owns completing any -pending workflow files from source evidence. +TLDRGraph never launches an AI process for feature generation and never invents +heuristic features. When feature artifacts are missing or stale, it writes +`.tldrgraph/feature_workflows_request.yaml`. The coding agent that ran +`tldrgraph init` must delegate that request to a source-reading subagent. Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent `needs_confirmation` response explicitly asks for approval. `--batch 200` means @@ -169,16 +174,18 @@ class AgentTarget: ## Feature Workflow Explorer artifacts -Every init run writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. -When `init` reports pending feature workflows: - -1. Open `.tldrgraph/features.yaml`. -2. Open every pending `.tldrgraph/workflows/.yaml`. -3. Open the source files referenced in `evidence_nodes`; do not guess. -4. Replace `steps: []` with source-backed steps and set `status: generated`. +Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. +When `init` reports `status: needs_feature_workflows`: + +1. Open `.tldrgraph/feature_workflows_request.yaml`. +2. Spawn a source-reading subagent and delegate the entire request to it. +3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. ## `status: needs_layers` diff --git a/tldrgraph/cli_pipeline.py b/tldrgraph/cli_pipeline.py index 20d9b69..27af2a5 100644 --- a/tldrgraph/cli_pipeline.py +++ b/tldrgraph/cli_pipeline.py @@ -18,6 +18,7 @@ ) from .cli_llm_links import apply_pending_llm_links_response, run_llm_link_step from .feature_workflows import generate_feature_workflow_files +from .feature_workflow_handoff import feature_workflow_status_lines from .graph_loader import GraphLoader from .installer import ensure_gitignore, install_agent_rules from .layer_config import config_path @@ -149,13 +150,13 @@ def _run_agent_cli_enrichment( ) resume = "tldrgraph init" if progress.get("approval_persisted") else "tldrgraph init --yes" retry = [f"Run `{resume}` to continue."] if rem or embedding_error else [] + feature_lines = feature_workflow_status_lines(path, feature_stats) if pending_workflows and not rem and not embedding_error else [] emit_status(status, "embeddings" if embedding_error else "enrichment", [ f"Enriched {totals['applied']} node(s) in {totals['batches']} batch(es); {totals['bridges']} bridge edge(s).", f"⚠️ {totals['intent_length_violations']} intent(s) were outside the recommended 2-3 sentences." if totals["intent_length_violations"] else "All applied intents met the recommended 2-3 sentence length.", f"{rem} still un-enriched." if rem else "Nothing left to enrich.", - f"{pending_workflows} feature workflow file(s) still need source-backed steps." if pending_workflows and not rem else "", f"Dense embeddings could not be completed: {embedding_error}" if embedding_error else _embedding_summary(loader), - ] + retry, + ] + feature_lines + retry, progress={**progress, "remaining": rem, "feature_workflows_pending": pending_workflows, "embedding_backend": loader.vector_store.backend}, as_json=as_json) return status @@ -183,6 +184,7 @@ def _emit_manual_enrichment_handoff( ], progress=progress, as_json=as_json) return STATUS_NEEDS_ENRICHMENT + def _emit_enrichment_done(loader: GraphLoader, total: int, enriched: int, excluded: int, registry: Any, as_json: bool, feature_stats: Optional[Dict[str, Any]] = None) -> str: embedding_error = _embedding_failure(loader) pending_workflows = int((feature_stats or {}).get("pending") or 0) @@ -197,12 +199,7 @@ def _emit_enrichment_done(loader: GraphLoader, total: int, enriched: int, exclud "Run `tldrgraph init` again after fixing model access.", ]) else: - workflow_lines = [ - f"{pending_workflows} feature workflow file(s) still need source-backed steps.", - " 1. Open .tldrgraph/features.yaml", - " 2. Complete each pending .tldrgraph/workflows/.yaml", - " 3. Run: tldrgraph init", - ] if pending_workflows else ["Feature workflow files are complete."] + workflow_lines = feature_workflow_status_lines(loader.root_dir, feature_stats) lines.extend([_embedding_summary(loader), "", *workflow_lines, "", ' tldrgraph query ""', ' tldrgraph trace "" ""', @@ -387,11 +384,13 @@ def init_pipeline( if link_applied and not as_json: click.echo(f"🔗 Applied {len(link_applied['applied'])} LLM route link(s)") - feature_stats = generate_feature_workflow_files(path, loader.graph, agent_model=agent_model, use_agent=False) + feature_stats = generate_feature_workflow_files(path, loader.graph) if not as_json: - click.echo(f"🧭 Feature workflows: {feature_stats['generated']} generated, {feature_stats['pending']} pending across {feature_stats['features']} feature(s)") if feature_stats["pending"]: - click.echo(" Current agent must complete pending .tldrgraph/workflows/*.yaml files from source evidence.") + click.echo("🧭 Feature workflows: waiting for host-agent subagent generation") + click.echo(" Delegate .tldrgraph/feature_workflows_request.yaml to a source-reading subagent.") + else: + click.echo(f"🧭 Feature workflows: {feature_stats['generated']} generated feature(s)") generate_visualizer_html(path) diff --git a/tldrgraph/feature_workflow_agent.py b/tldrgraph/feature_workflow_agent.py deleted file mode 100644 index 402afdd..0000000 --- a/tldrgraph/feature_workflow_agent.py +++ /dev/null @@ -1,70 +0,0 @@ -"""Agent prompts for source-backed feature workflow files.""" - -from __future__ import annotations - -from typing import Any, Dict, List, Optional - -from . import agent_runner -from .feature_workflows import relative_workflow_path - - -def _run_json(root: str, prompt: str, agent_model: Optional[str]) -> Optional[Dict[str, Any]]: - agent = agent_runner.find_agent_cli(respect_nesting=False) - if agent is None: - return None - try: - raw = agent_runner.run_agent_json(agent, prompt, root, model=agent_model) - except agent_runner.AgentError: - return None - return raw if isinstance(raw, dict) else None - - -def generate_feature_manifest( - root: str, - graph_hash: str, - candidates: List[Dict[str, Any]], - agent_model: Optional[str], -) -> Optional[Dict[str, Any]]: - prompt = ( - "Read this source-backed graph evidence and return JSON only. " - "List the project's major user-facing and developer-facing features. " - "Prefer features that represent an end-to-end user or developer goal, not isolated UI or backend helpers. " - "Do not use curated workflows, BPMN, route-link workflow discovery, or guessed product flows. " - "Every feature must include at least one evidence item copied from candidates.\n\n" - "Return {\"features\":[{\"id\",\"title\",\"audience\",\"summary\",\"evidence\":[...] }]}.\n" - f"Evidence:\n{{'graph_hash': {graph_hash!r}, 'candidates': {candidates!r}}}" - ) - raw = _run_json(root, prompt, agent_model) - items = raw.get("features") if raw else None - if not isinstance(items, list): - return None - return {"features": items} - - -def generate_feature_workflow( - root: str, - graph_hash: str, - feature: Dict[str, Any], - evidence_nodes: List[Dict[str, Any]], - agent_model: Optional[str], -) -> Optional[Dict[str, Any]]: - prompt = ( - "Create one saved Feature Workflow Explorer workflow as JSON only. " - "It must describe the FULL feature flow from the user's button/menu/form action all the way to the final response or UI update. " - "Do not summarize away important hops or combine unrelated source symbols into one vague step. " - "When source evidence supports it, include the path from the user-facing screen/component through request/client code, " - "event handler, validation, request payload construction, API route or controller, middleware/auth, service/use-case logic, " - "persistence/database, background jobs, external services, response payload creation, client response parsing, state update, " - "toast/navigation/rendered result, and any user-visible error or success handling. " - "Use simple language for non-technical users and vibe coders. " - "Do not use curated workflows, BPMN generation, route-link workflow discovery, or guessed steps. " - "Every step must include evidence copied from the provided source-backed nodes. " - "If a frontend-to-backend or backend-to-frontend bridge is not present in evidence, include only the proven side and make the feature pending rather than inventing missing steps.\n\n" - "Return exactly {\"workflow\":{\"schema\":\"codechakra/feature-workflow@1\",\"graph_hash\":...,\"feature_id\":...," - "\"title\":...,\"summary\":...,\"status\":\"generated\",\"steps\":[{\"number\":1,\"title\":...,\"text\":...,\"evidence\":[...]}]}}.\n" - f"Feature:\n{feature!r}\nWorkflow path: {relative_workflow_path(str(feature.get('id') or 'feature'))}\n" - f"Graph hash: {graph_hash}\nEvidence nodes:\n{evidence_nodes!r}" - ) - raw = _run_json(root, prompt, agent_model) - workflow = raw.get("workflow") if raw else None - return workflow if isinstance(workflow, dict) else None diff --git a/tldrgraph/feature_workflow_bridges.py b/tldrgraph/feature_workflow_bridges.py new file mode 100644 index 0000000..58c35c9 --- /dev/null +++ b/tldrgraph/feature_workflow_bridges.py @@ -0,0 +1,163 @@ +"""Endpoint-aware evidence expansion for saved feature workflows.""" + +from __future__ import annotations + +from typing import Any, Dict, List, Set, Tuple + +import networkx as nx + +from .layers import layer_id_of + +BANNED_WORKFLOW_RELATIONS = {"llm_http_route_link", "http_route_link", "calls_endpoint"} +SKIP_DIRS = (".tldrgraph/", "tests/", "test/", "spec/", "__tests__/", "node_modules/", "dist/", "build/", "vendor/", "migrations/") +STRUCTURAL_RELATIONS = {"contains", "rationale_for", "imports", "imports_from"} +ENDPOINT_CONTEXT_RELATIONS = {"calls_endpoint"} +HANDLER_RELATIONS = {"handled_by"} +MAX_EVIDENCE_NODES = 18 +MAX_CONTINUATION_STEPS = 5 + + +def endpoint_aware_walk(graph: nx.DiGraph, root_id: str) -> List[str]: + """Return source nodes that make a feature authoring prompt end-to-end.""" + chain = _walk_without_endpoint_shortcuts(graph, root_id) + ordered = list(chain) + seen = set(ordered) + + for node_id in chain: + for endpoint_id, handler_id in _endpoint_pairs(graph, node_id): + for candidate in (endpoint_id, handler_id): + if candidate and candidate not in seen: + ordered.append(candidate) + seen.add(candidate) + if handler_id: + for backend_id in _backend_continuation(graph, handler_id, seen): + ordered.append(backend_id) + seen.add(backend_id) + if len(ordered) >= MAX_EVIDENCE_NODES: + break + + return ordered[:MAX_EVIDENCE_NODES] + + +def endpoint_context(graph: nx.DiGraph, node_id: str) -> List[Dict[str, Any]]: + """Describe endpoint bridges without exposing banned relations as evidence.""" + items: List[Dict[str, Any]] = [] + for endpoint_id, handler_id in _endpoint_pairs(graph, node_id): + endpoint = graph.nodes.get(endpoint_id, {}) + handler = graph.nodes.get(handler_id, {}) if handler_id else {} + item = { + "endpoint": _evidence(endpoint_id, endpoint), + "bridge": "endpoint_context", + } + if handler_id and handler: + item["handler"] = _evidence(handler_id, handler) + items.append(item) + return items + + +def _evidence(node_id: str, node: Dict[str, Any]) -> Dict[str, Any]: + line = _source_line(node) + return { + "node_id": str(node_id), + "symbol": node.get("label") or str(node_id), + "file": node.get("file") or "", + "line": line, + "code_start": int(node.get("code_start") or line or 0), + "code_end": int(node.get("code_end") or line or 0), + } + + +def _source_line(node: Dict[str, Any]) -> int: + raw = node.get("source_location") or node.get("code_start") or "" + if isinstance(raw, int): + return raw + import re + + match = re.search(r"\d+", str(raw)) + return int(match.group(0)) if match else 0 + + +def _is_candidate_node(node: Dict[str, Any]) -> bool: + file_path = str(node.get("file") or "").replace("\\", "/").lower() + if not file_path or any(part in file_path for part in SKIP_DIRS): + return False + if node.get("is_test") or node.get("dead_code_status") in {"not_code", "dead"}: + return False + return bool(str(node.get("label") or "").strip()) + + +def _walk_without_endpoint_shortcuts(graph: nx.DiGraph, root_id: str) -> List[str]: + chain = [root_id] + seen = {root_id} + current = root_id + while len(chain) < MAX_CONTINUATION_STEPS: + nxt = _best_successor(graph, current, seen) + if not nxt: + break + chain.append(nxt) + seen.add(nxt) + current = nxt + return chain + + +def _best_successor(graph: nx.DiGraph, current: str, seen: Set[str]) -> str: + ranked: List[Tuple[int, str]] = [] + current_node = graph.nodes.get(current, {}) + current_layer = layer_id_of(current_node) + for _, target, data in graph.out_edges(current, data=True): + target = str(target) + relation = data.get("relation") or "calls" + if target in seen or relation in BANNED_WORKFLOW_RELATIONS or relation in STRUCTURAL_RELATIONS: + continue + node = graph.nodes.get(target, {}) + if not _is_candidate_node(node): + continue + score = graph.out_degree(target) + 2 + if layer_id_of(node) != current_layer: + score += 10 + if node.get("file") != current_node.get("file"): + score += 4 + ranked.append((score, target)) + ranked.sort(key=lambda item: (-item[0], item[1])) + return ranked[0][1] if ranked else "" + + +def _endpoint_pairs(graph: nx.DiGraph, node_id: str) -> List[Tuple[str, str]]: + pairs: List[Tuple[str, str]] = [] + for _, endpoint_id, data in graph.out_edges(node_id, data=True): + if data.get("relation") not in ENDPOINT_CONTEXT_RELATIONS: + continue + if endpoint_id not in graph: + continue + handler_id = _handler_for_endpoint(graph, str(endpoint_id)) + pairs.append((str(endpoint_id), handler_id)) + return pairs + + +def _handler_for_endpoint(graph: nx.DiGraph, endpoint_id: str) -> str: + for _, target, data in graph.out_edges(endpoint_id, data=True): + if data.get("relation") in HANDLER_RELATIONS and target in graph: + return str(target) + return "" + + +def _backend_continuation(graph: nx.DiGraph, start: str, seen: Set[str]) -> List[str]: + ordered: List[str] = [] + current = start + local_seen = set(seen) | {start} + while len(ordered) < MAX_CONTINUATION_STEPS: + nxt = _best_successor(graph, current, local_seen) + if not nxt: + break + node = graph.nodes.get(nxt, {}) + if _is_frontend_file(node.get("file")): + break + ordered.append(nxt) + local_seen.add(nxt) + current = nxt + return ordered + + +def _is_frontend_file(file_path: str) -> bool: + path = str(file_path or "").replace("\\", "/").lower() + return any(part in path for part in ("frontend/", "/app/", "/pages/", "/components/")) diff --git a/tldrgraph/feature_workflow_handoff.py b/tldrgraph/feature_workflow_handoff.py new file mode 100644 index 0000000..ad7a0f3 --- /dev/null +++ b/tldrgraph/feature_workflow_handoff.py @@ -0,0 +1,324 @@ +"""Host-agent handoff for source-backed feature workflow generation.""" + +from __future__ import annotations + +import os +import re +import tempfile +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional, Tuple + +import networkx as nx +import yaml + +from .cli_enrichment import read_payload, write_payload +from .feature_workflow_validation import validate_workflow + +REQUEST_FILENAME = "feature_workflows_request.yaml" +RESPONSE_FILENAME = "feature_workflows_response.yaml" +APPLIED_RESPONSE_FILENAME = "feature_workflows_response.applied.yaml" +REQUEST_SCHEMA = "codechakra/feature-workflows-request@1" +RESPONSE_SCHEMA = "codechakra/feature-workflows-response@1" +FEATURE_ID_RE = re.compile(r"^[a-z0-9][a-z0-9_]{0,63}$") + + +def _state_path(root: str, filename: str) -> str: + return os.path.join(os.path.abspath(root), ".tldrgraph", filename) + + +def clear_feature_workflow_request(root: str) -> None: + try: + os.remove(_state_path(root, REQUEST_FILENAME)) + except FileNotFoundError: + pass + + +def feature_workflow_status_lines(root: str, stats: Optional[Dict[str, Any]]) -> List[str]: + values = stats or {} + if not int(values.get("pending") or 0): + return ["Feature workflow files are complete."] + request_path = str(values.get("request_path") or _state_path(root, REQUEST_FILENAME)) + lines = [ + "Feature discovery and workflow generation require the active coding agent:", + f" 1. Read {os.path.relpath(request_path, os.path.abspath(root))}", + " 2. Delegate the entire request to a source-reading subagent.", + f" 3. Have the subagent write .tldrgraph/{RESPONSE_FILENAME}.", + " 4. Run: tldrgraph init", + ] + if values.get("agent_reason"): + lines.insert(1, f" Previous response rejected: {values['agent_reason']}") + return lines + + +def _candidate_payload(graph: nx.DiGraph) -> List[Dict[str, Any]]: + from .feature_workflows import ( + _candidate_roots, + _evidence, + _node_summary, + _workflow_evidence, + ) + + candidates: List[Dict[str, Any]] = [] + for rank, (node_id, audience) in enumerate(_candidate_roots(graph), 1): + node = graph.nodes[node_id] + feature_seed = {"evidence": [_evidence(node_id, node)]} + candidates.append({ + "rank": rank, + "audience": audience, + "summary": _node_summary(node), + "root": _evidence(node_id, node), + "label": node.get("label"), + "intent": node.get("intent"), + "layer_id": node.get("layer_id"), + "layer": node.get("layer"), + "evidence_nodes": _workflow_evidence(graph, feature_seed), + }) + return candidates + + +def _response_shape() -> Dict[str, Any]: + return { + "schema": RESPONSE_SCHEMA, + "graph_hash": "copy graph_hash from this request", + "features": [{ + "id": "lowercase_snake_case", + "title": "Feature title", + "audience": "user | developer", + "summary": "Source-backed feature goal.", + "evidence": ["copy evidence objects from candidates/evidence_nodes"], + "workflow": { + "status": "generated", + "summary": "Complete end-to-end flow.", + "steps": [{ + "number": 1, + "phase": "user_action | frontend | request | backend | persistence | response | ui_update", + "title": "Short step title", + "text": "Plain-language source-backed behavior.", + "evidence": ["copy evidence objects from candidates/evidence_nodes"], + }], + }, + }], + } + + +def write_feature_workflow_request( + root: str, graph: nx.DiGraph, current_hash: str, error: str = "" +) -> str: + payload = { + "schema": REQUEST_SCHEMA, + "generated_at": datetime.now(timezone.utc).isoformat(), + "graph_hash": current_hash, + "response_file": os.path.join(".tldrgraph", RESPONSE_FILENAME), + "instructions": [ + "The coding agent running tldrgraph init must delegate this entire request to a source-reading subagent.", + "The subagent must inspect the repository source and generate both the feature manifest and every workflow.", + "Select major end-to-end user-facing and developer-facing goals, not isolated helpers.", + "Start each workflow at the exact user action or command and follow every proven hop through the final response or UI update.", + "Every feature and step must copy source evidence from candidates/evidence_nodes; do not guess missing flows.", + "Do not use curated workflows, BPMN, route-link discovery, llm_http_route_link, http_route_link, or calls_endpoint as evidence.", + f"Write YAML matching response_shape to .tldrgraph/{RESPONSE_FILENAME}, then run tldrgraph init again.", + ], + "banned_evidence_relations": ["llm_http_route_link", "http_route_link", "calls_endpoint"], + "response_shape": _response_shape(), + "candidates": _candidate_payload(graph), + } + if error: + payload["previous_response_error"] = error + return write_payload(_state_path(root, REQUEST_FILENAME), payload) + + +def _canonical_evidence(graph: nx.DiGraph, raw: Any) -> Dict[str, Any]: + from .feature_workflows import _evidence + + if not isinstance(raw, dict): + raise ValueError("evidence entries must be objects") + node_id = str(raw.get("node_id") or "") + if not node_id or node_id not in graph: + raise ValueError(f"unknown evidence node_id: {node_id or ''}") + relation = str(raw.get("relation") or "") + if relation in {"llm_http_route_link", "http_route_link", "calls_endpoint"}: + raise ValueError(f"banned workflow evidence relation: {relation}") + evidence = _evidence(node_id, graph.nodes[node_id]) + if relation: + evidence["relation"] = relation + return evidence + + +def _canonical_evidence_list(graph: nx.DiGraph, raw: Any) -> List[Dict[str, Any]]: + if not isinstance(raw, list) or not raw: + raise ValueError("a non-empty evidence list is required") + return [_canonical_evidence(graph, item) for item in raw] + + +def _normalized_steps(graph: nx.DiGraph, raw: Any) -> List[Dict[str, Any]]: + if not isinstance(raw, list) or not raw: + raise ValueError("workflow steps must be a non-empty list") + steps: List[Dict[str, Any]] = [] + for index, item in enumerate(raw, 1): + if not isinstance(item, dict): + raise ValueError(f"workflow step {index} must be an object") + steps.append({ + "number": item.get("number"), + "phase": item.get("phase"), + "title": item.get("title"), + "text": item.get("text"), + "evidence": _canonical_evidence_list(graph, item.get("evidence")), + }) + return steps + + +def _feature_record(graph: nx.DiGraph, raw: Any, used_ids: set[str]) -> Dict[str, Any]: + from .feature_workflows import relative_workflow_path + + if not isinstance(raw, dict): + raise ValueError("each feature must be an object") + feature_id = str(raw.get("id") or "") + if not FEATURE_ID_RE.fullmatch(feature_id): + raise ValueError(f"invalid feature id: {feature_id or ''}") + if feature_id in used_ids: + raise ValueError(f"duplicate feature id: {feature_id}") + used_ids.add(feature_id) + title = str(raw.get("title") or "").strip() + summary = str(raw.get("summary") or "").strip() + audience = str(raw.get("audience") or "").strip() + if not title or not summary or audience not in {"user", "developer"}: + raise ValueError(f"feature {feature_id} requires title, summary, and user/developer audience") + evidence = _canonical_evidence_list(graph, raw.get("evidence")) + return { + "id": feature_id, + "title": title, + "audience": audience, + "summary": summary, + "status": "generated", + "workflow_path": relative_workflow_path(feature_id), + "evidence": evidence, + } + + +def _normalize_feature( + graph: nx.DiGraph, current_hash: str, raw: Any, used_ids: set[str] +) -> Tuple[Dict[str, Any], Dict[str, Any]]: + from .feature_workflows import ( + WORKFLOW_GENERATOR, + WORKFLOW_SCHEMA, + _workflow_evidence, + ) + + feature = _feature_record(graph, raw, used_ids) + feature_id = feature["id"] + title = feature["title"] + summary = feature["summary"] + evidence = feature["evidence"] + raw_workflow = raw.get("workflow") + if not isinstance(raw_workflow, dict) or raw_workflow.get("status") != "generated": + raise ValueError(f"feature {feature_id} requires a generated workflow") + workflow = { + "schema": WORKFLOW_SCHEMA, + "graph_hash": current_hash, + "generator": WORKFLOW_GENERATOR, + "feature_id": feature_id, + "title": title, + "summary": str(raw_workflow.get("summary") or summary).strip(), + "status": "generated", + "evidence": evidence, + "evidence_nodes": _workflow_evidence(graph, feature), + "steps": _normalized_steps(graph, raw_workflow.get("steps")), + } + if not validate_workflow(workflow): + raise ValueError(f"feature {feature_id} has an incomplete or invalid workflow") + return feature, workflow + + +def normalize_feature_workflow_response( + graph: nx.DiGraph, current_hash: str, payload: Any +) -> Tuple[Dict[str, Any], Dict[str, Dict[str, Any]]]: + from .feature_workflows import FEATURE_SCHEMA + + if not isinstance(payload, dict) or payload.get("schema") != RESPONSE_SCHEMA: + raise ValueError(f"response schema must be {RESPONSE_SCHEMA}") + if payload.get("graph_hash") != current_hash: + raise ValueError("response graph_hash does not match the current graph") + raw_features = payload.get("features") + if not isinstance(raw_features, list) or not raw_features: + raise ValueError("response must contain at least one feature") + features: List[Dict[str, Any]] = [] + workflows: Dict[str, Dict[str, Any]] = {} + used_ids: set[str] = set() + for raw in raw_features: + feature, workflow = _normalize_feature(graph, current_hash, raw, used_ids) + features.append(feature) + workflows[feature["id"]] = workflow + manifest = {"schema": FEATURE_SCHEMA, "graph_hash": current_hash, "features": features} + return manifest, workflows + + +def _atomic_write(path: str, payload: Dict[str, Any]) -> None: + os.makedirs(os.path.dirname(path), exist_ok=True) + handle = tempfile.NamedTemporaryFile("w", encoding="utf-8", dir=os.path.dirname(path), delete=False) + try: + with handle: + yaml.safe_dump(payload, handle, default_flow_style=False, sort_keys=False) + os.replace(handle.name, path) + except Exception: + try: + os.unlink(handle.name) + except OSError: + pass + raise + + +def apply_feature_workflow_response( + root: str, graph: nx.DiGraph, current_hash: str +) -> Tuple[Optional[Dict[str, Any]], str]: + from .feature_workflows import features_path, workflow_path + + response_path = _state_path(root, RESPONSE_FILENAME) + if not os.path.isfile(response_path): + return None, "" + try: + manifest, workflows = normalize_feature_workflow_response( + graph, current_hash, read_payload(response_path) + ) + for feature_id, workflow in workflows.items(): + _atomic_write(workflow_path(root, feature_id), workflow) + _atomic_write(features_path(root), manifest) + os.replace(response_path, _state_path(root, APPLIED_RESPONSE_FILENAME)) + try: + os.remove(_state_path(root, REQUEST_FILENAME)) + except FileNotFoundError: + pass + return manifest, "" + except (OSError, ValueError, TypeError, yaml.YAMLError) as err: + return None, str(err) + + +def current_manifest(root: str, current_hash: str) -> Optional[Dict[str, Any]]: + from .feature_workflows import FEATURE_SCHEMA, WORKFLOW_GENERATOR, features_path, workflow_path + + manifest = read_payload(features_path(root)) + if not isinstance(manifest, dict) or manifest.get("schema") != FEATURE_SCHEMA: + return None + features = manifest.get("features") + if manifest.get("graph_hash") != current_hash or not isinstance(features, list) or not features: + return None + feature_ids: set[str] = set() + for feature in features: + if not isinstance(feature, dict) or feature.get("status") != "generated": + return None + feature_id = str(feature.get("id") or "") + if not FEATURE_ID_RE.fullmatch(feature_id) or feature_id in feature_ids: + return None + feature_ids.add(feature_id) + if feature.get("workflow_path") != os.path.join(".tldrgraph", "workflows", f"{feature_id}.yaml"): + return None + workflow = read_payload(workflow_path(root, feature_id)) + if ( + not isinstance(workflow, dict) + or workflow.get("graph_hash") != current_hash + or workflow.get("generator") != WORKFLOW_GENERATOR + or workflow.get("feature_id") != feature_id + or workflow.get("status") != "generated" + or not validate_workflow(workflow) + ): + return None + return manifest diff --git a/tldrgraph/feature_workflow_loader.py b/tldrgraph/feature_workflow_loader.py index 5390806..6f78d3d 100644 --- a/tldrgraph/feature_workflow_loader.py +++ b/tldrgraph/feature_workflow_loader.py @@ -42,6 +42,13 @@ def load_feature_manifest(root: str) -> Tuple[Optional[Dict[str, Any]], str]: features = data.get("features") if not isinstance(features, list): return None, "invalid_features" + request = read_payload(os.path.join(root, ".tldrgraph", "feature_workflows_request.yaml")) + if ( + isinstance(request, dict) + and request.get("graph_hash") + and request.get("graph_hash") != data.get("graph_hash") + ): + return None, "stale_features" if not features: return data, "empty_features" return data, "ready" diff --git a/tldrgraph/feature_workflow_validation.py b/tldrgraph/feature_workflow_validation.py new file mode 100644 index 0000000..29ba581 --- /dev/null +++ b/tldrgraph/feature_workflow_validation.py @@ -0,0 +1,159 @@ +"""Validation for saved Feature Workflow Explorer files.""" + +from __future__ import annotations + +from typing import Any, Dict, Iterable, List, Set + + +WORKFLOW_SCHEMA = "codechakra/feature-workflow@1" +BANNED_WORKFLOW_RELATIONS = {"llm_http_route_link", "http_route_link", "calls_endpoint"} +FRONTEND_MARKERS = ("frontend/", "/app/", "/pages/", "/components/") +BACKEND_MARKERS = ("backend/", "/backend/", "server/", "/server/", "/routes/", "controller") +PHASE_ALIASES = { + "ui": "ui_update", + "client": "frontend", + "server": "backend", + "database": "persistence", + "result": "response", +} +SCAFFOLD_MARKERS = ( + "pending_reason", + "instructions", + "required_step_shape", +) + + +def validate_workflow(workflow: Dict[str, Any]) -> bool: + if workflow.get("schema") != WORKFLOW_SCHEMA: + return False + steps = workflow.get("steps") + if not isinstance(steps, list) or not steps: + return workflow.get("status") == "pending" + if not _steps_have_required_shape(steps): + return False + if _uses_banned_relation(steps): + return False + if workflow.get("status") != "generated": + return True + return _validate_generated_workflow(workflow, steps) + + +def _validate_generated_workflow(workflow: Dict[str, Any], steps: List[Dict[str, Any]]) -> bool: + if any(workflow.get(marker) for marker in SCAFFOLD_MARKERS): + return False + phases = {_normalize_phase(step.get("phase")) for step in steps} + phases.discard("") + if not phases: + return False + if _is_user_facing_workflow(workflow): + if len(steps) < 4: + return False + if not ({"user_action", "frontend"} & phases and "request" in phases): + return False + if not ({"response", "ui_update"} & phases): + return False + if ( + "request" in phases + and not workflow.get("external_only_request") + and not _steps_include_backend_evidence(steps) + ): + return False + if _has_backend_evidence(workflow) and not _steps_include_backend_evidence(steps): + return False + else: + if len(steps) < 3: + return False + if "backend" not in phases: + return False + if not ({"response", "ui_update"} & phases): + return False + return True + + +def _steps_have_required_shape(steps: List[Dict[str, Any]]) -> bool: + for step in steps: + if not isinstance(step, dict): + return False + if not isinstance(step.get("number"), int): + return False + if not _non_empty(step.get("phase")): + return False + if not _non_empty(step.get("title")) or not _non_empty(step.get("text")): + return False + evidence = step.get("evidence") + if not isinstance(evidence, list) or not evidence: + return False + for ev in evidence: + if not isinstance(ev, dict): + return False + if not ev.get("node_id") or not ev.get("file") or not ev.get("symbol"): + return False + return True + + +def _uses_banned_relation(steps: List[Dict[str, Any]]) -> bool: + for step in steps: + for ev in step.get("evidence") or []: + relation = str(ev.get("relation") or "") + if relation in BANNED_WORKFLOW_RELATIONS: + return True + return False + + +def _is_user_facing_workflow(workflow: Dict[str, Any]) -> bool: + files = _workflow_evidence_files(workflow) + return any(_is_frontend_file(file_path) for file_path in files) + + +def _has_backend_evidence(workflow: Dict[str, Any]) -> bool: + files = _workflow_evidence_files(workflow) + return any(_is_backend_file(file_path) for file_path in files) + + +def _steps_include_backend_evidence(steps: List[Dict[str, Any]]) -> bool: + return any(_is_backend_file(str(ev.get("file") or "")) for ev in _step_evidence(steps)) + + +def _workflow_evidence_files(workflow: Dict[str, Any]) -> Set[str]: + files = {str(ev.get("file") or "") for ev in workflow.get("evidence") or [] if isinstance(ev, dict)} + for node in workflow.get("evidence_nodes") or []: + if not isinstance(node, dict): + continue + evidence = node.get("evidence") + if isinstance(evidence, dict): + files.add(str(evidence.get("file") or "")) + for outgoing in node.get("outgoing") or []: + if not isinstance(outgoing, dict): + continue + target = outgoing.get("target") + if isinstance(target, dict): + files.add(str(target.get("file") or "")) + return files + + +def _step_evidence(steps: List[Dict[str, Any]]) -> Iterable[Dict[str, Any]]: + for step in steps: + for ev in step.get("evidence") or []: + if isinstance(ev, dict): + yield ev + + +def _normalize_phase(value: Any) -> str: + phase = str(value or "").strip().lower().replace("-", "_").replace(" ", "_") + return PHASE_ALIASES.get(phase, phase) + + +def _is_frontend_file(file_path: str) -> bool: + path = file_path.replace("\\", "/").lower() + return any(marker in path for marker in FRONTEND_MARKERS) + + +def _is_backend_file(file_path: str) -> bool: + path = file_path.replace("\\", "/").lower() + if any(marker in path for marker in BACKEND_MARKERS): + return True + return path.startswith("api/") or "/pages/api/" in path or "/app/api/" in path + + +def _non_empty(value: Any) -> bool: + return bool(str(value or "").strip()) diff --git a/tldrgraph/feature_workflows.py b/tldrgraph/feature_workflows.py index 22386a0..a4e423c 100644 --- a/tldrgraph/feature_workflows.py +++ b/tldrgraph/feature_workflows.py @@ -9,14 +9,15 @@ import networkx as nx -from .cli_enrichment import read_payload, write_payload +from .feature_workflow_bridges import endpoint_aware_walk, endpoint_context +from .feature_workflow_validation import validate_workflow from .layers import layer_id_of FEATURES_FILENAME = "features.yaml" WORKFLOWS_DIRNAME = "workflows" FEATURE_SCHEMA = "codechakra/features@1" WORKFLOW_SCHEMA = "codechakra/feature-workflow@1" -WORKFLOW_GENERATOR = "feature-workflow-agent-owned@3" +WORKFLOW_GENERATOR = "feature-workflow-subagent@1" BANNED_WORKFLOW_RELATIONS = {"llm_http_route_link", "http_route_link", "calls_endpoint"} SKIP_DIRS = (".tldrgraph/", "tests/", "test/", "spec/", "__tests__/", "node_modules/", "dist/", "build/", "vendor/", "migrations/") @@ -89,11 +90,6 @@ def _evidence(node_id: str, node: Dict[str, Any]) -> Dict[str, Any]: } -def _slug(text: str, fallback: str) -> str: - slug = re.sub(r"[^a-z0-9]+", "_", text.lower()).strip("_") - return (slug or fallback)[:64] - - def _humanize(text: str) -> str: text = re.sub(r"\([^)]*\)", "", str(text or "")) text = text.split(".")[-1] @@ -144,66 +140,9 @@ def _candidate_roots(graph: nx.DiGraph, limit: int = 12) -> List[Tuple[str, str] return [(node_id, audience) for _, node_id, audience in scored[:limit]] -def _walk_feature_steps(graph: nx.DiGraph, root_id: str, max_steps: int = 12) -> List[str]: - chain = [root_id] - seen = {root_id} - current = root_id - while len(chain) < max_steps: - ranked: List[Tuple[int, str]] = [] - current_layer = layer_id_of(graph.nodes.get(current, {})) - for _, target, data in graph.out_edges(current, data=True): - target = str(target) - if target in seen or not _is_candidate_node(graph.nodes.get(target, {})): - continue - relation = data.get("relation") or "calls" - if relation in BANNED_WORKFLOW_RELATIONS: - continue - if relation in {"contains", "rationale_for", "imports", "imports_from"}: - continue - node = graph.nodes[target] - score = graph.out_degree(target) + 2 - if layer_id_of(node) != current_layer: - score += 10 - if node.get("file") != graph.nodes[current].get("file"): - score += 4 - ranked.append((score, target)) - if not ranked: - break - ranked.sort(key=lambda item: (-item[0], item[1])) - current = ranked[0][1] - chain.append(current) - seen.add(current) - return chain - - -def _fallback_features(graph: nx.DiGraph, current_hash: str) -> Dict[str, Any]: - features: List[Dict[str, Any]] = [] - used_ids = set() - for root_id, audience in _candidate_roots(graph): - node = graph.nodes[root_id] - base_id = _slug(str(node.get("label") or root_id), f"feature_{len(features) + 1}") - feature_id = base_id - counter = 2 - while feature_id in used_ids: - feature_id = f"{base_id}_{counter}" - counter += 1 - used_ids.add(feature_id) - title = _humanize(node.get("display_label") or node.get("label") or root_id) - features.append({ - "id": feature_id, - "title": title, - "audience": audience, - "summary": _node_summary(node), - "status": "pending", - "workflow_path": relative_workflow_path(feature_id), - "evidence": [_evidence(root_id, node)], - }) - return {"schema": FEATURE_SCHEMA, "graph_hash": current_hash, "features": features} - - def _evidence_node(graph: nx.DiGraph, node_id: str) -> Dict[str, Any]: node = graph.nodes[node_id] - return { + record = { "evidence": _evidence(node_id, node), "label": node.get("label"), "intent": node.get("intent"), @@ -225,102 +164,45 @@ def _evidence_node(graph: nx.DiGraph, node_id: str) -> Dict[str, Any]: ) ][:24], } + context = endpoint_context(graph, node_id) + if context: + record["endpoint_context"] = context + return record def _workflow_evidence(graph: nx.DiGraph, feature: Dict[str, Any]) -> List[Dict[str, Any]]: root_id = str((feature.get("evidence") or [{}])[0].get("node_id") or "") - return [_evidence_node(graph, node_id) for node_id in _walk_feature_steps(graph, root_id) if node_id in graph] - - -def _pending_workflow(feature: Dict[str, Any], graph: nx.DiGraph, current_hash: str, reason: str) -> Dict[str, Any]: - return { - "schema": WORKFLOW_SCHEMA, - "graph_hash": current_hash, - "generator": WORKFLOW_GENERATOR, - "feature_id": feature.get("id"), - "title": feature.get("title") or _humanize(feature.get("id")), - "summary": feature.get("summary") or reason, - "status": "pending", - "pending_reason": reason, - "evidence": feature.get("evidence") or [], - "evidence_nodes": _workflow_evidence(graph, feature), - "instructions": [ - "The same coding agent running `tldrgraph init` must complete this file.", - "Open every source file referenced in evidence_nodes before writing steps.", - "Start at the user's button/menu/form action when present.", - "Continue through request, backend work, response payload, client handling, and final UI update when proven.", - "Use only source-backed evidence; leave status pending if a hop is not proven.", - ], - "required_step_shape": {"number": 1, "title": "...", "text": "...", "evidence": ["copy evidence objects from evidence_nodes"]}, - "steps": [], - } - - -def validate_workflow(workflow: Dict[str, Any]) -> bool: - if workflow.get("schema") != WORKFLOW_SCHEMA: - return False - steps = workflow.get("steps") - if not isinstance(steps, list) or not steps: - return workflow.get("status") == "pending" - for step in steps: - if not isinstance(step, dict): - return False - evidence = step.get("evidence") - if not isinstance(evidence, list) or not evidence: - return False - for ev in evidence: - if not isinstance(ev, dict): - return False - if not ev.get("node_id") or not ev.get("file") or not ev.get("symbol"): - return False - return True + return [_evidence_node(graph, node_id) for node_id in endpoint_aware_walk(graph, root_id) if node_id in graph] def generate_feature_workflow_files( root: str, graph: nx.DiGraph, - agent_model: Any = None, - use_agent: bool = True, ) -> Dict[str, Any]: - """Writes features.yaml and all stale/missing workflow files.""" + """Apply a host-subagent response or request one without heuristic output.""" + from .feature_workflow_handoff import ( + apply_feature_workflow_response, + clear_feature_workflow_request, + current_manifest, + write_feature_workflow_request, + ) + root = os.path.abspath(root) current_hash = graph_hash(graph) - manifest = _fallback_features(graph, current_hash) - reason = "Feature workflow is pending for the current coding agent to complete from source evidence." - - os.makedirs(workflows_dir(root), exist_ok=True) - generated = 0 - pending = 0 - updated_features = [] - for feature in manifest.get("features", []): - if not isinstance(feature, dict) or not feature.get("id"): - continue - feature_id = _slug(str(feature["id"]), f"feature_{len(updated_features) + 1}") - feature = {**feature, "id": feature_id, "workflow_path": relative_workflow_path(feature_id)} - existing = read_payload(workflow_path(root, feature_id)) - stale = ( - not isinstance(existing, dict) - or existing.get("graph_hash") != current_hash - or existing.get("generator") != WORKFLOW_GENERATOR - ) - workflow = existing if isinstance(existing, dict) and not stale else None - if workflow is None: - workflow = _pending_workflow(feature, graph, current_hash, reason) - if not validate_workflow(workflow): - workflow = _pending_workflow(feature, graph, current_hash, "Saved workflow is invalid or incomplete.") - write_payload(workflow_path(root, feature_id), workflow) - status = str(workflow.get("status") or "pending") - feature["status"] = status - if status == "generated": - generated += 1 - else: - pending += 1 - updated_features.append(feature) - - manifest = {"schema": FEATURE_SCHEMA, "graph_hash": current_hash, "features": updated_features} - write_payload(features_path(root), manifest) - return {"features": len(updated_features), "generated": generated, "pending": pending, - "graph_hash": current_hash, "agent_reason": ""} + manifest, error = apply_feature_workflow_response(root, graph, current_hash) + if error: + request_path = write_feature_workflow_request(root, graph, current_hash, error) + return {"features": 0, "generated": 0, "pending": 1, + "graph_hash": current_hash, "agent_reason": error, "request_path": request_path} + manifest = manifest or current_manifest(root, current_hash) + if manifest is not None: + clear_feature_workflow_request(root) + count = len(manifest.get("features") or []) + return {"features": count, "generated": count, "pending": 0, + "graph_hash": current_hash, "agent_reason": "", "request_path": ""} + request_path = write_feature_workflow_request(root, graph, current_hash, error) + return {"features": 0, "generated": 0, "pending": 1, + "graph_hash": current_hash, "agent_reason": error, "request_path": request_path} def load_feature_manifest(root: str) -> Tuple[Optional[Dict[str, Any]], str]: diff --git a/tldrgraph/installer_contract.py b/tldrgraph/installer_contract.py index 8e7c409..7044b81 100644 --- a/tldrgraph/installer_contract.py +++ b/tldrgraph/installer_contract.py @@ -38,10 +38,10 @@ def generate_layers_prose(registry: Optional[LayerRegistry] = None) -> str: tldrgraph init ``` -`init` does not use AI for architecture layer design, all-symbol enrichment, -route-link inference, or BPMN workflow generation by default. It only attempts -source-backed saved Feature Workflow Explorer files and writes pending states -when a full workflow cannot be generated. +`init` never launches AI for Feature Workflow Explorer and never invents +heuristic features. It writes `.tldrgraph/feature_workflows_request.yaml`; the +host coding agent must delegate that request to a source-reading subagent and +rerun init after `.tldrgraph/feature_workflows_response.yaml` is written. `--agent-cli` is now explicit opt-in for architecture layer design and enrichment. Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless @@ -69,6 +69,9 @@ def generate_layers_prose(registry: Optional[LayerRegistry] = None) -> str: match with 100% confidence; fallback vector search handles related terms with a 0.35 score floor. - **Write the response to a different file than the request.** The request is regenerated on every run. +- **Complete `needs_feature_workflows` through a subagent.** Delegate the entire + `.tldrgraph/feature_workflows_request.yaml` file to a source-reading subagent, + have it write both features and workflows to the named response file, then rerun init. - **Complete `needs_llm_links` when shown.** Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend files, and write `.tldrgraph/llm_links_response.yaml` with `{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. @@ -124,6 +127,8 @@ def generate_layers_prose(registry: Optional[LayerRegistry] = None) -> str: | `.tldrgraph/enrichment_response.json` | **you** | `apply-enrichment` | | `.tldrgraph/enrichment_approval.json` | `init --yes` | continuation runs | | `.tldrgraph/pending_enrichment.json` | *(legacy)* | `apply-enrichment`, only if no response file exists | +| `.tldrgraph/feature_workflows_request.yaml` | `init` | host coding agent and subagent | +| `.tldrgraph/feature_workflows_response.yaml` | source-reading subagent | `init` | ## Response schema diff --git a/tldrgraph/visualizer/assets/app.js b/tldrgraph/visualizer/assets/app.js index 9dbb9ce..47930a1 100644 --- a/tldrgraph/visualizer/assets/app.js +++ b/tldrgraph/visualizer/assets/app.js @@ -2731,6 +2731,7 @@ function renderWorkflowsList() { const messages = { missing_features: 'No feature manifest found. Run tldrgraph init to create .tldrgraph/features.yaml.', invalid_features: '.tldrgraph/features.yaml is invalid. Run tldrgraph init to refresh it.', + stale_features: 'Saved feature workflows are stale. Complete the host-agent subagent handoff from .tldrgraph/feature_workflows_request.yaml.', empty_features: 'No features were saved for this project yet.', ready: allWorkflows.length ? 'No matching saved workflows found.' : 'No saved feature workflows found.', }; From a49eec35c3b040f629ee9e9b9fd533fa034f9788 Mon Sep 17 00:00:00 2001 From: Ashwani Kharwar Date: Mon, 14 Sep 2026 16:47:09 +0530 Subject: [PATCH 03/13] feat: add v2 capability catalog and workflow coverage states - Introduce area-grouped feature/workflow schema with legacy compatibility - Validate generated, partial, and pending workflow evidence - Update visualizer with searchable capability catalog - Refresh documentation and test coverage --- .agents/skills/tldrgraph-init/SKILL.md | 13 +- .claude/commands/tldrgraph-init.md | 13 +- .cursor/commands/tldrgraph-init.md | 13 +- .tldrgraph/AGENT_CONTRACT.md | 42 +++- AGENTS.md | 8 +- tests/test_auto_agent.py | 9 +- tests/test_feature_workflows.py | 143 ++++++++++++- tests/test_visualizer.py | 45 +++- tldrgraph/agent_commands.py | 15 +- tldrgraph/feature_workflow_handoff.py | 195 +++++------------ tldrgraph/feature_workflow_loader.py | 140 ++++++++++--- tldrgraph/feature_workflow_schema.py | 197 ++++++++++++++++++ tldrgraph/feature_workflow_validation.py | 26 ++- tldrgraph/feature_workflows.py | 16 +- tldrgraph/installer_contract.py | 6 +- tldrgraph/visualizer/assets/app.js | 145 ++++++------- tldrgraph/visualizer/assets/index.html | 6 +- .../visualizer/assets/workflow-catalog.css | 85 ++++++++ .../visualizer/assets/workflow-catalog.js | 132 ++++++++++++ tldrgraph/visualizer/data.py | 1 + tldrgraph/visualizer/render.py | 2 + 21 files changed, 960 insertions(+), 292 deletions(-) create mode 100644 tldrgraph/feature_workflow_schema.py create mode 100644 tldrgraph/visualizer/assets/workflow-catalog.css create mode 100644 tldrgraph/visualizer/assets/workflow-catalog.js diff --git a/.agents/skills/tldrgraph-init/SKILL.md b/.agents/skills/tldrgraph-init/SKILL.md index 2accc83..94f15de 100644 --- a/.agents/skills/tldrgraph-init/SKILL.md +++ b/.agents/skills/tldrgraph-init/SKILL.md @@ -27,17 +27,24 @@ explicitly requests it. ## Feature Workflow Explorer artifacts -An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes the v2 capability catalog in `.tldrgraph/features.yaml` +and flows in `.tldrgraph/workflows/.yaml`. The manifest groups concrete +capabilities into inferred product or technical areas; ranked symbols in the request are +investigation leads, not the feature list. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. -Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +Define features as user, admin, developer, or operator outcomes, not code symbols. Good: +"Natural-language project creation." Bad: "OpencodeService"; that service is evidence. +Each short plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Use `partial` with `missing_coverage` for a proven fragment and `pending` when no reliable +sequence can be drawn. List source-backed capabilities in either state without inventing steps. Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When `init` reports `status: needs_feature_workflows`: 1. Open `.tldrgraph/feature_workflows_request.yaml`. 2. Spawn a source-reading subagent and delegate the entire request to it. -3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +3. Have the subagent inspect the whole repository and write areas, features, and generated/partial/pending workflows to `.tldrgraph/feature_workflows_response.yaml`. 4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. diff --git a/.claude/commands/tldrgraph-init.md b/.claude/commands/tldrgraph-init.md index 2accc83..94f15de 100644 --- a/.claude/commands/tldrgraph-init.md +++ b/.claude/commands/tldrgraph-init.md @@ -27,17 +27,24 @@ explicitly requests it. ## Feature Workflow Explorer artifacts -An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes the v2 capability catalog in `.tldrgraph/features.yaml` +and flows in `.tldrgraph/workflows/.yaml`. The manifest groups concrete +capabilities into inferred product or technical areas; ranked symbols in the request are +investigation leads, not the feature list. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. -Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +Define features as user, admin, developer, or operator outcomes, not code symbols. Good: +"Natural-language project creation." Bad: "OpencodeService"; that service is evidence. +Each short plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Use `partial` with `missing_coverage` for a proven fragment and `pending` when no reliable +sequence can be drawn. List source-backed capabilities in either state without inventing steps. Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When `init` reports `status: needs_feature_workflows`: 1. Open `.tldrgraph/feature_workflows_request.yaml`. 2. Spawn a source-reading subagent and delegate the entire request to it. -3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +3. Have the subagent inspect the whole repository and write areas, features, and generated/partial/pending workflows to `.tldrgraph/feature_workflows_response.yaml`. 4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. diff --git a/.cursor/commands/tldrgraph-init.md b/.cursor/commands/tldrgraph-init.md index 2accc83..94f15de 100644 --- a/.cursor/commands/tldrgraph-init.md +++ b/.cursor/commands/tldrgraph-init.md @@ -27,17 +27,24 @@ explicitly requests it. ## Feature Workflow Explorer artifacts -An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes the v2 capability catalog in `.tldrgraph/features.yaml` +and flows in `.tldrgraph/workflows/.yaml`. The manifest groups concrete +capabilities into inferred product or technical areas; ranked symbols in the request are +investigation leads, not the feature list. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. -Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +Define features as user, admin, developer, or operator outcomes, not code symbols. Good: +"Natural-language project creation." Bad: "OpencodeService"; that service is evidence. +Each short plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Use `partial` with `missing_coverage` for a proven fragment and `pending` when no reliable +sequence can be drawn. List source-backed capabilities in either state without inventing steps. Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When `init` reports `status: needs_feature_workflows`: 1. Open `.tldrgraph/feature_workflows_request.yaml`. 2. Spawn a source-reading subagent and delegate the entire request to it. -3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +3. Have the subagent inspect the whole repository and write areas, features, and generated/partial/pending workflows to `.tldrgraph/feature_workflows_response.yaml`. 4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. diff --git a/.tldrgraph/AGENT_CONTRACT.md b/.tldrgraph/AGENT_CONTRACT.md index ee4ccde..2512fac 100644 --- a/.tldrgraph/AGENT_CONTRACT.md +++ b/.tldrgraph/AGENT_CONTRACT.md @@ -62,13 +62,19 @@ Request and response are **separate files**. Never write your answer back into ## Feature Workflow Explorer artifacts -`tldrgraph init` also creates the saved workflow artifacts used by the visualizer: +`tldrgraph init` also creates the v2 capability catalog and saved flows used by the visualizer: | File | Written by | Read by | | --- | --- | --- | | `.tldrgraph/features.yaml` | `tldrgraph init` | Workflow Explorer | | `.tldrgraph/workflows/.yaml` | `tldrgraph init` | Workflow Explorer | +`features.yaml` contains ordered repository-specific areas (`product` before +`technical`) and concrete capabilities for user, admin, developer, or operator audiences. +Ranked symbols in the handoff request are investigation leads, not the feature list. +Good feature: **Natural-language project creation**. Bad feature: `OpencodeService`; +that service belongs in the capability's source evidence. + The Workflow Explorer tab is intentionally file-driven. It must read only `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`; if those files are missing, invalid, incomplete, or a feature has no generated workflow @@ -80,10 +86,11 @@ discovery, `llm_http_route_link`, `http_route_link`, `calls_endpoint`, or BPMN-derived workflow generation. Graph views elsewhere may still show route links or BPMN data, but saved feature workflows must remain independent. -Every saved workflow step must be simple enough for non-technical users and vibe +Every saved workflow step must have a short title simple enough for non-technical users and vibe coders, and each step must carry source evidence: `node_id`, symbol, file, and line/range. If the evidence is absent, mark the workflow pending instead of -guessing. +guessing. Use `status: partial` plus `missing_coverage` for a proven fragment and +`status: pending` when no reliable sequence can be drawn. Pending workflows contain no steps. A saved feature workflow should describe the complete flow when evidence exists: the exact user button/menu/form action, event handler, validation, client @@ -94,6 +101,35 @@ update, navigation/toast/rendered result, and visible success or error handling. Do not stop at only the frontend or only the backend when the source proves the handoff, and do not collapse multiple proven source hops into one vague step. +The v2 response shape is: + +```yaml +schema: codechakra/feature-workflows-response@2 +graph_hash: "copy from request" +areas: + - id: ai_builder + title: AI application builder + summary: Create and revise applications with AI. + perspective: product + order: 0 +features: + - id: natural_language_project_creation + area_id: ai_builder + title: Natural-language project creation + audience: user + summary: Turn a prompt into a new application project. + evidence: [{node_id: "copy exact graph node id"}] + workflow: + status: generated # or partial / pending + summary: Create the project and show its result. + steps: [{number: 1, phase: user_action, title: Submit a prompt, + text: The user submits the project request., + evidence: [{node_id: "copy exact graph node id"}]}] +``` + +`partial` and `pending` require `missing_coverage`; `pending` requires an empty +`steps` list. Allowed audiences are `user`, `admin`, `developer`, and `operator`. + --- ## Request schema (`enrichment_request.yaml`) diff --git a/AGENTS.md b/AGENTS.md index 8bf471e..8704877 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,14 +30,18 @@ artifacts: `.tldrgraph/features.yaml` plus `.tldrgraph/workflows/.ya TLDRGraph never invents heuristic features or launches an AI process for feature generation. If `init` returns `needs_feature_workflows`, the agent running it must open `.tldrgraph/feature_workflows_request.yaml`, spawn a source-reading subagent, -and have that subagent write `.tldrgraph/feature_workflows_response.yaml` with both -features and complete workflows. Run `tldrgraph init` again to validate and apply +and have that subagent write `.tldrgraph/feature_workflows_response.yaml` with inferred +product/technical areas and source-backed capabilities. Ranked symbols are investigation +leads, not the feature list. Run `tldrgraph init` again to validate and apply the response; do not edit final feature/workflow files directly. Workflow Explorer reads only saved YAML, never curated blueprints, route-link workflow discovery, BPMN-derived generation, `discover_workflows()`, `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When writing feature workflows, start at the user's button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update. Do not skip proven steps. +Define outcomes as features: "Natural-language project creation" is a feature; +`OpencodeService` is supporting evidence. Use `partial` with `missing_coverage` for a +proven fragment and `pending` with no steps when no reliable sequence can be drawn. Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary, and do not use route-link relations as workflow evidence. diff --git a/tests/test_auto_agent.py b/tests/test_auto_agent.py index d524a43..10bfe2c 100644 --- a/tests/test_auto_agent.py +++ b/tests/test_auto_agent.py @@ -57,12 +57,17 @@ def fake_agent(name: str = "fake") -> agent_runner.AgentCLI: def complete_pending_workflows(root: Path) -> None: state = root / ".tldrgraph" request = yaml.safe_load((state / "feature_workflows_request.yaml").read_text(encoding="utf-8")) - evidence = request["candidates"][0]["root"] + evidence = request["investigation_leads"][0]["root"] response = { - "schema": "codechakra/feature-workflows-response@1", + "schema": "codechakra/feature-workflows-response@2", "graph_hash": request["graph_hash"], + "areas": [{ + "id": "developer_tools", "title": "Developer tools", + "summary": "Commands used by developers.", "perspective": "technical", "order": 0, + }], "features": [{ "id": "run_sample_cli", + "area_id": "developer_tools", "title": "Run Sample CLI", "audience": "developer", "summary": "Run the sample command through its engine.", diff --git a/tests/test_feature_workflows.py b/tests/test_feature_workflows.py index 66629c9..a52792a 100644 --- a/tests/test_feature_workflows.py +++ b/tests/test_feature_workflows.py @@ -31,6 +31,7 @@ def _write_valid_response(mini_repo, graph, **feature_overrides): feature = { "id": "submit_case", + "area_id": "case_management", "title": "Submit Case", "audience": "user", "summary": "Submit and persist a case.", @@ -54,6 +55,13 @@ def _write_valid_response(mini_repo, graph, **feature_overrides): return write_payload(str(mini_repo.tldrgraph_dir / RESPONSE_FILENAME), { "schema": RESPONSE_SCHEMA, "graph_hash": graph_hash(graph), + "areas": [{ + "id": "case_management", + "title": "Case management", + "summary": "Submit and manage pension cases.", + "perspective": "product", + "order": 0, + }], "features": [feature], }) @@ -72,9 +80,11 @@ def test_missing_feature_artifacts_create_host_subagent_request(loader, mini_rep request = yaml.safe_load(open(stats["request_path"], encoding="utf-8")) assert request["schema"] == REQUEST_SCHEMA assert request["graph_hash"] == stats["graph_hash"] - assert request["candidates"] - assert request["candidates"][0]["root"]["node_id"] - assert request["candidates"][0]["evidence_nodes"] + assert request["investigation_leads"] + assert request["investigation_leads"][0]["root"]["node_id"] + assert request["investigation_leads"][0]["evidence_nodes"] + assert "not the feature list" in "\n".join(request["instructions"]) + assert request["repository_discovery"]["graph_file"] == ".tldrgraph/graph.json" assert "delegate this entire request" in "\n".join(request["instructions"]) @@ -89,6 +99,126 @@ def test_feature_workflow_validation_rejects_steps_without_evidence(): }) +def test_partial_and_pending_workflow_validation_requires_honest_coverage(mini_repo): + from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow + + partial_step = _step(1, "frontend", "ui_page", mini_repo) + assert validate_workflow({ + "schema": WORKFLOW_SCHEMA, + "status": "partial", + "missing_coverage": "The backend handoff is not present in the graph.", + "steps": [partial_step], + }) + assert not validate_workflow({ + "schema": WORKFLOW_SCHEMA, "status": "partial", "steps": [partial_step], + }) + assert validate_workflow({ + "schema": WORKFLOW_SCHEMA, + "status": "pending", + "missing_coverage": "No reliable sequence can be drawn.", + "steps": [], + }) + assert not validate_workflow({ + "schema": WORKFLOW_SCHEMA, + "status": "pending", + "missing_coverage": "A sequence is not proven.", + "steps": [partial_step], + }) + + +def test_v2_catalog_normalizes_areas_and_incomplete_capabilities(loader, mini_repo): + from tldrgraph.feature_workflow_handoff import RESPONSE_SCHEMA, normalize_feature_workflow_response + from tldrgraph.feature_workflows import graph_hash + + graph = loader.load_or_extract(enrich_llm=False) + evidence = _step(1, "frontend", "ui_page", mini_repo)["evidence"] + payload = { + "schema": RESPONSE_SCHEMA, + "graph_hash": graph_hash(graph), + "areas": [ + {"id": "operations", "title": "Operations", "summary": "Internal operation.", + "perspective": "technical", "order": 0}, + {"id": "case_management", "title": "Case management", "summary": "Manage cases.", + "perspective": "product", "order": 0}, + ], + "features": [ + {"id": "review_case", "area_id": "case_management", "title": "Review a case", + "audience": "admin", "summary": "Review an existing case.", "evidence": evidence, + "workflow": {"status": "partial", "summary": "Open the review page.", + "missing_coverage": "The save response is not proven.", + "steps": [_step(1, "frontend", "ui_page", mini_repo)]}}, + {"id": "operate_pipeline", "area_id": "operations", "title": "Operate the pipeline", + "audience": "operator", "summary": "Operate the internal pipeline.", "evidence": evidence, + "workflow": {"status": "pending", "summary": "Pipeline operation.", + "missing_coverage": "No reliable sequence can be drawn.", "steps": []}}, + ], + } + + manifest, workflows = normalize_feature_workflow_response(graph, graph_hash(graph), payload) + + assert [area["id"] for area in manifest["areas"]] == ["case_management", "operations"] + assert [feature["status"] for feature in manifest["features"]] == ["partial", "pending"] + assert workflows["review_case"]["missing_coverage"] + assert workflows["operate_pipeline"]["steps"] == [] + + +def test_v2_catalog_rejects_unknown_feature_area(loader, mini_repo): + from tldrgraph.feature_workflow_handoff import RESPONSE_SCHEMA, normalize_feature_workflow_response + from tldrgraph.feature_workflows import graph_hash + + graph = loader.load_or_extract(enrich_llm=False) + payload = { + "schema": RESPONSE_SCHEMA, "graph_hash": graph_hash(graph), + "areas": [{"id": "known", "title": "Known", "summary": "Known area.", + "perspective": "product", "order": 0}], + "features": [{"id": "orphan", "area_id": "missing", "title": "Orphan feature", + "audience": "user", "summary": "Has no valid area.", + "evidence": _step(1, "frontend", "ui_page", mini_repo)["evidence"], + "workflow": {"status": "pending", "summary": "Unknown.", + "missing_coverage": "No flow.", "steps": []}}], + } + + with pytest.raises(ValueError, match="unknown area"): + normalize_feature_workflow_response(graph, graph_hash(graph), payload) + + +def test_v1_manifest_loads_in_legacy_area(mini_repo): + from tldrgraph.cli_enrichment import write_payload + from tldrgraph.feature_workflows import LEGACY_FEATURE_SCHEMA, load_saved_feature_workflows + + write_payload(str(mini_repo.tldrgraph_dir / "features.yaml"), { + "schema": LEGACY_FEATURE_SCHEMA, "graph_hash": "old", + "features": [{"id": "old_service", "title": "Old Service", "audience": "developer", + "summary": "A legacy symbol-oriented feature.", "status": "pending", + "workflow_path": ".tldrgraph/workflows/old_service.yaml", "evidence": []}], + }) + + payload = load_saved_feature_workflows(str(mini_repo.root)) + + assert payload["state"] == "ready" + assert payload["legacy"] is True + assert payload["areas"][0]["id"] == "legacy_features" + assert payload["workflows"][0]["area_title"] == "Legacy features" + + +def test_v1_manifest_is_not_current_for_generation(loader, mini_repo): + from tldrgraph.cli_enrichment import write_payload + from tldrgraph.feature_workflow_handoff import REQUEST_SCHEMA + from tldrgraph.feature_workflows import LEGACY_FEATURE_SCHEMA, generate_feature_workflow_files, graph_hash + + graph = loader.load_or_extract(enrich_llm=False) + write_payload(str(mini_repo.tldrgraph_dir / "features.yaml"), { + "schema": LEGACY_FEATURE_SCHEMA, "graph_hash": graph_hash(graph), + "features": [{"id": "old", "title": "Old", "status": "generated"}], + }) + + stats = generate_feature_workflow_files(str(mini_repo.root), graph) + request = yaml.safe_load(open(stats["request_path"], encoding="utf-8")) + + assert stats["pending"] == 1 + assert request["schema"] == REQUEST_SCHEMA + + def test_user_workflow_validation_rejects_shallow_generated_steps(mini_repo): from tldrgraph.feature_workflows import WORKFLOW_SCHEMA, validate_workflow @@ -229,7 +359,7 @@ def test_feature_request_evidence_crosses_endpoint_context(tmp_path): stats = generate_feature_workflow_files(str(tmp_path), graph) request = yaml.safe_load(open(stats["request_path"], encoding="utf-8")) - candidate = next(item for item in request["candidates"] if item["root"]["node_id"] == "page") + candidate = next(item for item in request["investigation_leads"] if item["root"]["node_id"] == "page") node_ids = [node["evidence"]["node_id"] for node in candidate["evidence_nodes"]] assert node_ids[:6] == ["page", "api", "endpoint", "handler", "service", "checkpoint"] @@ -264,8 +394,13 @@ def test_missing_workflow_file_becomes_pending_state(mini_repo): write_payload(str(mini_repo.tldrgraph_dir / "features.yaml"), { "schema": FEATURE_SCHEMA, "graph_hash": "test", + "areas": [{ + "id": "operations", "title": "Operations", "summary": "Operational capabilities.", + "perspective": "technical", "order": 0, + }], "features": [{ "id": "not_generated", + "area_id": "operations", "title": "Not Generated", "audience": "developer", "summary": "A feature without a workflow file.", diff --git a/tests/test_visualizer.py b/tests/test_visualizer.py index 3c11a34..1885624 100644 --- a/tests/test_visualizer.py +++ b/tests/test_visualizer.py @@ -177,8 +177,13 @@ def test_workflows_payload_structure(mini_repo): write_payload(str(mini_repo.tldrgraph_dir / "features.yaml"), { "schema": FEATURE_SCHEMA, "graph_hash": "test", + "areas": [{ + "id": "case_management", "title": "Case management", + "summary": "Submit and manage cases.", "perspective": "product", "order": 0, + }], "features": [{ "id": "submit_case", + "area_id": "case_management", "title": "Submit Case", "audience": "user", "summary": "Send a case through the project.", @@ -231,6 +236,44 @@ def test_workflows_payload_structure(mini_repo): assert "node_id" in s +def test_feature_process_uses_human_titles_and_partial_end_marker(mini_repo): + from tldrgraph.cli_enrichment import write_payload + from tldrgraph.feature_workflows import FEATURE_SCHEMA, WORKFLOW_SCHEMA, load_saved_feature_workflows + + evidence = {"node_id": mini_repo.nid("ui_page"), "symbol": mini_repo.label("ui_page"), + "file": mini_repo.source_file("ui_page"), "line": 1} + write_payload(str(mini_repo.tldrgraph_dir / "features.yaml"), { + "schema": FEATURE_SCHEMA, "graph_hash": "test", "generator": "feature-catalog-subagent@2", + "areas": [{"id": "case_management", "title": "Case management", + "summary": "Manage cases.", "perspective": "product", "order": 0}], + "features": [{"id": "submit_case", "area_id": "case_management", "title": "Submit a case", + "audience": "user", "summary": "Submit a case for review.", "status": "partial", + "workflow_path": ".tldrgraph/workflows/submit_case.yaml", "evidence": [evidence]}], + }) + write_payload(str(mini_repo.tldrgraph_dir / "workflows" / "submit_case.yaml"), { + "schema": WORKFLOW_SCHEMA, "graph_hash": "test", "generator": "feature-workflow-subagent@2", + "feature_id": "submit_case", "title": "Submit a case", "summary": "Submit a case.", + "status": "partial", "missing_coverage": "The server response is not proven.", + "evidence": [evidence], "steps": [{"number": 1, "phase": "user_action", + "title": "Choose Submit case", "text": "The user chooses the submit action.", + "evidence": [evidence]}], + }) + + payload = load_saved_feature_workflows(str(mini_repo.root)) + workflow = payload["workflows"][0] + task = workflow["process"]["elements"][1] + + assert workflow["area_title"] == "Case management" + assert workflow["perspective"] == "product" + assert task["label"] == "Choose Submit case" + assert task["detail"] == "The user chooses the submit action." + assert task["source_symbol"] == mini_repo.label("ui_page") + assert task["phase"] == "user_action" + assert any(element.get("minor") for element in workflow["process"]["elements"]) + assert workflow["process"]["elements"][-1]["kind"] == "partial" + assert workflow["process"]["elements"][-1]["label"] == "Known flow ends here" + + def test_workflow_explorer_does_not_call_discovery_or_bpmn(monkeypatch, mini_repo): """The tab is file-driven, not discovered from routes, blueprints, or BPMN.""" from tldrgraph.visualizer import prepare_visualizer_data @@ -458,6 +501,6 @@ def test_saved_feature_generation_ignores_route_link_relations(): stats = generate_feature_workflow_files(root, graph) request = yaml.safe_load(open(stats["request_path"], encoding="utf-8")) - page = next(item for item in request["candidates"] if item["root"]["node_id"] == "page") + page = next(item for item in request["investigation_leads"] if item["root"]["node_id"] == "page") outgoing = page["evidence_nodes"][0]["outgoing"] assert all(item["target"]["node_id"] != "handler" for item in outgoing) diff --git a/tldrgraph/agent_commands.py b/tldrgraph/agent_commands.py index f42ce76..d4e2509 100644 --- a/tldrgraph/agent_commands.py +++ b/tldrgraph/agent_commands.py @@ -126,14 +126,18 @@ class AgentTarget: TLDRGraph never invents heuristic features or launches an AI process for feature generation. If `init` returns `needs_feature_workflows`, the agent running it must open `.tldrgraph/feature_workflows_request.yaml`, spawn a source-reading subagent, -and have that subagent write `.tldrgraph/feature_workflows_response.yaml` with both -features and complete workflows. Run `tldrgraph init` again to validate and apply +and have that subagent write `.tldrgraph/feature_workflows_response.yaml` with inferred +product/technical areas and source-backed capabilities. Ranked symbols are investigation +leads, not the feature list. Run `tldrgraph init` again to validate and apply the response; do not edit final feature/workflow files directly. Workflow Explorer reads only saved YAML, never curated blueprints, route-link workflow discovery, BPMN-derived generation, `discover_workflows()`, `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When writing feature workflows, start at the user's button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update. Do not skip proven steps. +Define outcomes as features: "Natural-language project creation" is a feature; +`OpencodeService` is supporting evidence. Use `partial` with `missing_coverage` for a +proven fragment and `pending` with no steps when no reliable sequence can be drawn. Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary, and do not use route-link relations as workflow evidence. @@ -174,17 +178,18 @@ class AgentTarget: ## Feature Workflow Explorer artifacts -An accepted subagent response writes `.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`. +An accepted subagent response writes a v2 area/capability catalog in `.tldrgraph/features.yaml` and flows in `.tldrgraph/workflows/.yaml`. Workflow Explorer reads only those files; missing or invalid workflows show pending states. Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. -Each plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. +Define human outcomes as features, not classes, hooks, services, or endpoints. "Natural-language project creation" is a feature; `OpencodeService` is evidence. Each short plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. +Use `partial` with `missing_coverage` for a proven fragment and `pending` with no steps when no reliable sequence can be drawn. Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. When `init` reports `status: needs_feature_workflows`: 1. Open `.tldrgraph/feature_workflows_request.yaml`. 2. Spawn a source-reading subagent and delegate the entire request to it. -3. Have the subagent inspect the referenced files and write both features and complete workflows to `.tldrgraph/feature_workflows_response.yaml`. +3. Have the subagent inspect the whole repository and write areas, capabilities, and generated/partial/pending workflows to `.tldrgraph/feature_workflows_response.yaml`. 4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. 5. Run `tldrgraph init` again. diff --git a/tldrgraph/feature_workflow_handoff.py b/tldrgraph/feature_workflow_handoff.py index ad7a0f3..e018304 100644 --- a/tldrgraph/feature_workflow_handoff.py +++ b/tldrgraph/feature_workflow_handoff.py @@ -3,7 +3,6 @@ from __future__ import annotations import os -import re import tempfile from datetime import datetime, timezone from typing import Any, Dict, List, Optional, Tuple @@ -12,14 +11,14 @@ import yaml from .cli_enrichment import read_payload, write_payload +from .feature_workflow_schema import ID_RE as FEATURE_ID_RE +from .feature_workflow_schema import RESPONSE_SCHEMA, normalize_response from .feature_workflow_validation import validate_workflow REQUEST_FILENAME = "feature_workflows_request.yaml" RESPONSE_FILENAME = "feature_workflows_response.yaml" APPLIED_RESPONSE_FILENAME = "feature_workflows_response.applied.yaml" -REQUEST_SCHEMA = "codechakra/feature-workflows-request@1" -RESPONSE_SCHEMA = "codechakra/feature-workflows-response@1" -FEATURE_ID_RE = re.compile(r"^[a-z0-9][a-z0-9_]{0,63}$") +REQUEST_SCHEMA = "codechakra/feature-workflows-request@2" def _state_path(root: str, filename: str) -> str: @@ -36,7 +35,7 @@ def clear_feature_workflow_request(root: str) -> None: def feature_workflow_status_lines(root: str, stats: Optional[Dict[str, Any]]) -> List[str]: values = stats or {} if not int(values.get("pending") or 0): - return ["Feature workflow files are complete."] + return ["Feature catalog and workflow files are complete."] request_path = str(values.get("request_path") or _state_path(root, REQUEST_FILENAME)) lines = [ "Feature discovery and workflow generation require the active coding agent:", @@ -80,21 +79,30 @@ def _response_shape() -> Dict[str, Any]: return { "schema": RESPONSE_SCHEMA, "graph_hash": "copy graph_hash from this request", + "areas": [{ + "id": "lowercase_snake_case", + "title": "Repository-specific capability area", + "summary": "Plain-language description of this area.", + "perspective": "product | technical", + "order": 0, + }], "features": [{ "id": "lowercase_snake_case", - "title": "Feature title", - "audience": "user | developer", - "summary": "Source-backed feature goal.", - "evidence": ["copy evidence objects from candidates/evidence_nodes"], + "area_id": "copy an id from areas", + "title": "Human capability title", + "audience": "user | admin | developer | operator", + "summary": "Source-backed outcome, not a code symbol description.", + "evidence": ["copy exact node IDs from investigation_leads or .tldrgraph/graph.json"], "workflow": { - "status": "generated", - "summary": "Complete end-to-end flow.", + "status": "generated | partial | pending", + "summary": "Complete or known portion of the flow.", + "missing_coverage": "required for partial or pending; omit for generated", "steps": [{ "number": 1, - "phase": "user_action | frontend | request | backend | persistence | response | ui_update", + "phase": "user_action | frontend | request | backend | persistence | external | response | ui_update", "title": "Short step title", "text": "Plain-language source-backed behavior.", - "evidence": ["copy evidence objects from candidates/evidence_nodes"], + "evidence": ["exact graph node IDs; every displayed step needs evidence"], }], }, }], @@ -111,145 +119,36 @@ def write_feature_workflow_request( "response_file": os.path.join(".tldrgraph", RESPONSE_FILENAME), "instructions": [ "The coding agent running tldrgraph init must delegate this entire request to a source-reading subagent.", - "The subagent must inspect the repository source and generate both the feature manifest and every workflow.", - "Select major end-to-end user-facing and developer-facing goals, not isolated helpers.", + "Build a repository-wide capability catalog; investigation_leads are starting points, not the feature list.", + "Inspect README and docs, UI routes/actions, API registration, services, persistence, integrations, deployment/configuration, and administration surfaces.", + "Infer repository-specific product areas and list concrete capabilities beneath them; put product areas before technical/internal areas.", + "A feature is a meaningful user, admin, developer, or operator outcome, never merely a class, hook, helper, service, endpoint, or page symbol.", + "Good: Natural-language project creation. Bad: OpencodeService. Use OpencodeService only as supporting evidence.", "Start each workflow at the exact user action or command and follow every proven hop through the final response or UI update.", - "Every feature and step must copy source evidence from candidates/evidence_nodes; do not guess missing flows.", + "Use generated only for a complete proven journey, partial for a proven fragment with missing_coverage, and pending when no reliable sequence can be drawn.", + "List meaningful source-backed capabilities even when their workflow is partial or pending; never fabricate missing steps.", + "Every feature and displayed step must use exact node IDs from investigation_leads or .tldrgraph/graph.json and be verified against source.", "Do not use curated workflows, BPMN, route-link discovery, llm_http_route_link, http_route_link, or calls_endpoint as evidence.", f"Write YAML matching response_shape to .tldrgraph/{RESPONSE_FILENAME}, then run tldrgraph init again.", ], + "repository_discovery": { + "root": ".", + "graph_file": ".tldrgraph/graph.json", + "note": "Read the repository and graph directly; do not limit the catalog to the ranked leads below.", + }, "banned_evidence_relations": ["llm_http_route_link", "http_route_link", "calls_endpoint"], "response_shape": _response_shape(), - "candidates": _candidate_payload(graph), + "investigation_leads": _candidate_payload(graph), } if error: payload["previous_response_error"] = error return write_payload(_state_path(root, REQUEST_FILENAME), payload) -def _canonical_evidence(graph: nx.DiGraph, raw: Any) -> Dict[str, Any]: - from .feature_workflows import _evidence - - if not isinstance(raw, dict): - raise ValueError("evidence entries must be objects") - node_id = str(raw.get("node_id") or "") - if not node_id or node_id not in graph: - raise ValueError(f"unknown evidence node_id: {node_id or ''}") - relation = str(raw.get("relation") or "") - if relation in {"llm_http_route_link", "http_route_link", "calls_endpoint"}: - raise ValueError(f"banned workflow evidence relation: {relation}") - evidence = _evidence(node_id, graph.nodes[node_id]) - if relation: - evidence["relation"] = relation - return evidence - - -def _canonical_evidence_list(graph: nx.DiGraph, raw: Any) -> List[Dict[str, Any]]: - if not isinstance(raw, list) or not raw: - raise ValueError("a non-empty evidence list is required") - return [_canonical_evidence(graph, item) for item in raw] - - -def _normalized_steps(graph: nx.DiGraph, raw: Any) -> List[Dict[str, Any]]: - if not isinstance(raw, list) or not raw: - raise ValueError("workflow steps must be a non-empty list") - steps: List[Dict[str, Any]] = [] - for index, item in enumerate(raw, 1): - if not isinstance(item, dict): - raise ValueError(f"workflow step {index} must be an object") - steps.append({ - "number": item.get("number"), - "phase": item.get("phase"), - "title": item.get("title"), - "text": item.get("text"), - "evidence": _canonical_evidence_list(graph, item.get("evidence")), - }) - return steps - - -def _feature_record(graph: nx.DiGraph, raw: Any, used_ids: set[str]) -> Dict[str, Any]: - from .feature_workflows import relative_workflow_path - - if not isinstance(raw, dict): - raise ValueError("each feature must be an object") - feature_id = str(raw.get("id") or "") - if not FEATURE_ID_RE.fullmatch(feature_id): - raise ValueError(f"invalid feature id: {feature_id or ''}") - if feature_id in used_ids: - raise ValueError(f"duplicate feature id: {feature_id}") - used_ids.add(feature_id) - title = str(raw.get("title") or "").strip() - summary = str(raw.get("summary") or "").strip() - audience = str(raw.get("audience") or "").strip() - if not title or not summary or audience not in {"user", "developer"}: - raise ValueError(f"feature {feature_id} requires title, summary, and user/developer audience") - evidence = _canonical_evidence_list(graph, raw.get("evidence")) - return { - "id": feature_id, - "title": title, - "audience": audience, - "summary": summary, - "status": "generated", - "workflow_path": relative_workflow_path(feature_id), - "evidence": evidence, - } - - -def _normalize_feature( - graph: nx.DiGraph, current_hash: str, raw: Any, used_ids: set[str] -) -> Tuple[Dict[str, Any], Dict[str, Any]]: - from .feature_workflows import ( - WORKFLOW_GENERATOR, - WORKFLOW_SCHEMA, - _workflow_evidence, - ) - - feature = _feature_record(graph, raw, used_ids) - feature_id = feature["id"] - title = feature["title"] - summary = feature["summary"] - evidence = feature["evidence"] - raw_workflow = raw.get("workflow") - if not isinstance(raw_workflow, dict) or raw_workflow.get("status") != "generated": - raise ValueError(f"feature {feature_id} requires a generated workflow") - workflow = { - "schema": WORKFLOW_SCHEMA, - "graph_hash": current_hash, - "generator": WORKFLOW_GENERATOR, - "feature_id": feature_id, - "title": title, - "summary": str(raw_workflow.get("summary") or summary).strip(), - "status": "generated", - "evidence": evidence, - "evidence_nodes": _workflow_evidence(graph, feature), - "steps": _normalized_steps(graph, raw_workflow.get("steps")), - } - if not validate_workflow(workflow): - raise ValueError(f"feature {feature_id} has an incomplete or invalid workflow") - return feature, workflow - - def normalize_feature_workflow_response( graph: nx.DiGraph, current_hash: str, payload: Any ) -> Tuple[Dict[str, Any], Dict[str, Dict[str, Any]]]: - from .feature_workflows import FEATURE_SCHEMA - - if not isinstance(payload, dict) or payload.get("schema") != RESPONSE_SCHEMA: - raise ValueError(f"response schema must be {RESPONSE_SCHEMA}") - if payload.get("graph_hash") != current_hash: - raise ValueError("response graph_hash does not match the current graph") - raw_features = payload.get("features") - if not isinstance(raw_features, list) or not raw_features: - raise ValueError("response must contain at least one feature") - features: List[Dict[str, Any]] = [] - workflows: Dict[str, Dict[str, Any]] = {} - used_ids: set[str] = set() - for raw in raw_features: - feature, workflow = _normalize_feature(graph, current_hash, raw, used_ids) - features.append(feature) - workflows[feature["id"]] = workflow - manifest = {"schema": FEATURE_SCHEMA, "graph_hash": current_hash, "features": features} - return manifest, workflows + return normalize_response(graph, current_hash, payload) def _atomic_write(path: str, payload: Dict[str, Any]) -> None: @@ -298,17 +197,33 @@ def current_manifest(root: str, current_hash: str) -> Optional[Dict[str, Any]]: manifest = read_payload(features_path(root)) if not isinstance(manifest, dict) or manifest.get("schema") != FEATURE_SCHEMA: return None + areas = manifest.get("areas") features = manifest.get("features") - if manifest.get("graph_hash") != current_hash or not isinstance(features, list) or not features: + if ( + manifest.get("graph_hash") != current_hash + or manifest.get("generator") != "feature-catalog-subagent@2" + or not isinstance(areas, list) or not areas + or not isinstance(features, list) or not features + ): + return None + area_ids = {str(area.get("id") or "") for area in areas if isinstance(area, dict)} + if len(area_ids) != len(areas) or any( + not FEATURE_ID_RE.fullmatch(str(area.get("id") or "")) + or area.get("perspective") not in {"product", "technical"} + or not isinstance(area.get("order"), int) + for area in areas if isinstance(area, dict) + ): return None feature_ids: set[str] = set() for feature in features: - if not isinstance(feature, dict) or feature.get("status") != "generated": + if not isinstance(feature, dict) or feature.get("status") not in {"generated", "partial", "pending"}: return None feature_id = str(feature.get("id") or "") if not FEATURE_ID_RE.fullmatch(feature_id) or feature_id in feature_ids: return None feature_ids.add(feature_id) + if feature.get("area_id") not in area_ids: + return None if feature.get("workflow_path") != os.path.join(".tldrgraph", "workflows", f"{feature_id}.yaml"): return None workflow = read_payload(workflow_path(root, feature_id)) @@ -317,7 +232,7 @@ def current_manifest(root: str, current_hash: str) -> Optional[Dict[str, Any]]: or workflow.get("graph_hash") != current_hash or workflow.get("generator") != WORKFLOW_GENERATOR or workflow.get("feature_id") != feature_id - or workflow.get("status") != "generated" + or workflow.get("status") != feature.get("status") or not validate_workflow(workflow) ): return None diff --git a/tldrgraph/feature_workflow_loader.py b/tldrgraph/feature_workflow_loader.py index 6f78d3d..a2233a6 100644 --- a/tldrgraph/feature_workflow_loader.py +++ b/tldrgraph/feature_workflow_loader.py @@ -8,6 +8,7 @@ from .cli_enrichment import read_payload from .feature_workflows import ( FEATURE_SCHEMA, + LEGACY_FEATURE_SCHEMA, WORKFLOW_SCHEMA, features_path, relative_workflow_path, @@ -15,6 +16,15 @@ workflow_path, ) +LEGACY_AREA = { + "id": "legacy_features", + "title": "Legacy features", + "summary": "Features generated with the previous symbol-oriented catalog.", + "perspective": "technical", + "order": 0, +} +VALID_PERSPECTIVES = {"product", "technical"} + def _humanize(text: str) -> str: return str(text or "").replace("_", " ").replace("-", " ").title() or "Project Feature" @@ -37,7 +47,10 @@ def load_feature_manifest(root: str) -> Tuple[Optional[Dict[str, Any]], str]: data = read_payload(features_path(root)) if data is None: return None, "missing_features" - if not isinstance(data, dict) or data.get("schema") != FEATURE_SCHEMA: + if ( + not isinstance(data, dict) + or data.get("schema") not in {FEATURE_SCHEMA, LEGACY_FEATURE_SCHEMA} + ): return None, "invalid_features" features = data.get("features") if not isinstance(features, list): @@ -51,26 +64,72 @@ def load_feature_manifest(root: str) -> Tuple[Optional[Dict[str, Any]], str]: return None, "stale_features" if not features: return data, "empty_features" + if data.get("schema") == LEGACY_FEATURE_SCHEMA: + data = _legacy_manifest(data) + elif not _valid_areas(data.get("areas")): + return None, "invalid_features" return data, "ready" +def _valid_areas(raw: Any) -> bool: + if not isinstance(raw, list) or not raw: + return False + ids = set() + for area in raw: + if not isinstance(area, dict) or not str(area.get("id") or ""): + return False + if area["id"] in ids or area.get("perspective") not in VALID_PERSPECTIVES: + return False + if not isinstance(area.get("order"), int) or not str(area.get("title") or "").strip(): + return False + ids.add(area["id"]) + return True + + +def _legacy_manifest(data: Dict[str, Any]) -> Dict[str, Any]: + """Adapt v1 files in memory while the next init requests v2 regeneration.""" + features = [] + for feature in data.get("features") or []: + if not isinstance(feature, dict): + continue + features.append({ + **feature, + "area_id": LEGACY_AREA["id"], + "status": feature.get("status") or "generated", + }) + return {**data, "areas": [LEGACY_AREA], "features": features, "legacy": True} + + def load_saved_feature_workflows(root: str) -> Dict[str, Any]: manifest, state = load_feature_manifest(root) if not manifest: return {"state": state, "workflows": []} - workflows = [_visualizer_workflow(root, f) for f in manifest.get("features", []) if isinstance(f, dict)] + areas = manifest.get("areas") or [] + area_by_id = { + str(area.get("id") or ""): area for area in areas if isinstance(area, dict) + } + workflows = [ + _visualizer_workflow( + root, feature, area_by_id.get(str(feature.get("area_id") or ""), LEGACY_AREA) + ) + for feature in manifest.get("features", []) if isinstance(feature, dict) + ] ready_count = sum(1 for wf in workflows if wf.get("status") == "generated") + partial_count = sum(1 for wf in workflows if wf.get("status") == "partial") return { "state": "ready" if workflows else "empty_features", "graph_hash": manifest.get("graph_hash"), + "areas": areas, + "legacy": bool(manifest.get("legacy")), "workflows": workflows, "ready_count": ready_count, - "pending_count": len(workflows) - ready_count, + "partial_count": partial_count, + "pending_count": len(workflows) - ready_count - partial_count, } -def _visualizer_workflow(root: str, feature: Dict[str, Any]) -> Dict[str, Any]: +def _visualizer_workflow(root: str, feature: Dict[str, Any], area: Dict[str, Any]) -> Dict[str, Any]: feature_id = str(feature.get("id") or "") path = workflow_path(root, feature_id) data = read_payload(path) @@ -84,7 +143,13 @@ def _visualizer_workflow(root: str, feature: Dict[str, Any]) -> Dict[str, Any]: return { "id": feature_id, "title": data.get("title") or feature.get("title") or _humanize(feature_id), - "category": feature.get("audience") or "Feature", + "area_id": area.get("id") or "legacy_features", + "area_title": area.get("title") or "Legacy features", + "area_summary": area.get("summary") or "", + "perspective": area.get("perspective") or "technical", + "area_order": int(area.get("order") or 0), + "audience": feature.get("audience") or "developer", + "category": area.get("title") or feature.get("audience") or "Feature", "root_node": feature.get("title") or feature_id, "root_id": _first_node_id(steps, feature), "file": _first_file(steps, feature), @@ -97,9 +162,11 @@ def _visualizer_workflow(root: str, feature: Dict[str, Any]) -> Dict[str, Any]: "steps": steps, "support": [], "status": status, - "pending_reason": data.get("pending_reason") or ("" if status == "generated" else "Workflow is pending."), + "missing_coverage": data.get("missing_coverage") or data.get("pending_reason") or "", + "pending_reason": data.get("missing_coverage") or data.get("pending_reason") + or ("" if status == "generated" else "Workflow coverage is incomplete."), "workflow_path": relative_workflow_path(feature_id), - "process": _simple_process(feature_id, steps, status), + "process": _simple_process(feature_id, feature.get("title") or feature_id, steps, status), } @@ -122,6 +189,7 @@ def _visualizer_steps(raw_steps: Any) -> List[Dict[str, Any]]: "layer": "Feature Workflow", "type": "feature_step", "intent": step.get("text") or "", + "phase": step.get("phase") or "backend", "code_start": int(ev.get("code_start") or ev.get("line") or 0), "code_end": int(ev.get("code_end") or ev.get("line") or 0), "evidence": evidence, @@ -129,29 +197,51 @@ def _visualizer_steps(raw_steps: Any) -> List[Dict[str, Any]]: return steps -def _simple_process(feature_id: str, steps: List[Dict[str, Any]], status: str) -> Dict[str, Any]: - elements = [_event(f"{feature_id}__start", "start", "Start the feature", 0)] +def _step_element(feature_id: str, step: Dict[str, Any]) -> Dict[str, Any]: + return { + "id": f"{feature_id}__step_{step['step_number']}", "kind": "task", + "label": step.get("display_label") or step.get("symbol"), "detail": step.get("intent") or "", + "source_symbol": step.get("symbol") or "", "phase": step.get("phase") or "backend", + "lane": "user" if step.get("phase") == "user_action" else "system", + "step": step["step_number"], "step_title": step.get("display_label") or step.get("symbol"), + "file": step.get("file") or "", "line": step.get("code_start") or 0, + "node_id": step.get("node_id") or None, "minor": False, + } + + +def _evidence_elements(feature_id: str, step: Dict[str, Any]) -> List[Dict[str, Any]]: + elements = [] + for index, evidence in enumerate(step.get("evidence") or [], 1): + where = str(evidence.get("file") or "") + line = int(evidence.get("code_start") or evidence.get("line") or 0) + elements.append({ + "id": f"{feature_id}__step_{step['step_number']}__evidence_{index}", "kind": "task", + "label": evidence.get("symbol") or "Source evidence", "detail": f"Source evidence: {where}:{line}", + "source_symbol": evidence.get("symbol") or "", "phase": step.get("phase") or "backend", + "lane": "system", "step": step["step_number"], "file": where, "line": line, + "node_id": evidence.get("node_id") or None, "minor": True, + }) + return elements + + +def _simple_process(feature_id: str, title: str, steps: List[Dict[str, Any]], status: str) -> Dict[str, Any]: + if status == "pending": + return {"lanes": [], "elements": [], "flows": []} + elements = [_event(f"{feature_id}__start", "start", f"Start: {title}", 0)] flows = [] previous = elements[0]["id"] for step in steps: - element_id = f"{feature_id}__step_{step['step_number']}" - elements.append({ - "id": element_id, - "kind": "task", - "label": step.get("intent") or step.get("display_label") or step.get("symbol"), - "detail": step.get("symbol") or "", - "lane": "system", - "step": step["step_number"], - "step_title": step.get("display_label") or step.get("symbol"), - "file": step.get("file") or "", - "line": step.get("code_start") or 0, - "node_id": step.get("node_id") or None, - "minor": False, - }) + element = _step_element(feature_id, step) + element_id = element["id"] + elements.append(element) flows.append({"source": previous, "target": element_id, "label": "", "kind": "sequence"}) + for evidence in _evidence_elements(feature_id, step): + elements.append(evidence) + flows.append({"source": element_id, "target": evidence["id"], "label": "evidence", "kind": "detail"}) previous = element_id - label = "Workflow pending" if status != "generated" else "Feature workflow complete" - elements.append(_event(f"{feature_id}__finish", "end", label, len(steps) + 1)) + label = "Feature workflow complete" if status == "generated" else "Known flow ends here" + kind = "end" if status == "generated" else "partial" + elements.append(_event(f"{feature_id}__finish", kind, label, len(steps) + 1)) flows.append({"source": previous, "target": elements[-1]["id"], "label": "", "kind": "sequence"}) return { "lanes": [ diff --git a/tldrgraph/feature_workflow_schema.py b/tldrgraph/feature_workflow_schema.py new file mode 100644 index 0000000..f6aa66d --- /dev/null +++ b/tldrgraph/feature_workflow_schema.py @@ -0,0 +1,197 @@ +"""Version 2 response normalization for source-backed feature catalogs.""" + +from __future__ import annotations + +import re +from typing import Any, Dict, List, Tuple + +import networkx as nx + +from .feature_workflow_validation import validate_workflow + +RESPONSE_SCHEMA = "codechakra/feature-workflows-response@2" +ID_RE = re.compile(r"^[a-z0-9][a-z0-9_]{0,63}$") +PERSPECTIVES = {"product", "technical"} +AUDIENCES = {"user", "admin", "developer", "operator"} +STATUSES = {"generated", "partial", "pending"} + + +def _canonical_evidence(graph: nx.DiGraph, raw: Any) -> Dict[str, Any]: + from .feature_workflows import _evidence + + if not isinstance(raw, dict): + raise ValueError("evidence entries must be objects") + node_id = str(raw.get("node_id") or "") + if not node_id or node_id not in graph: + raise ValueError(f"unknown evidence node_id: {node_id or ''}") + relation = str(raw.get("relation") or "") + if relation in {"llm_http_route_link", "http_route_link", "calls_endpoint"}: + raise ValueError(f"banned workflow evidence relation: {relation}") + evidence = _evidence(node_id, graph.nodes[node_id]) + if relation: + evidence["relation"] = relation + return evidence + + +def _evidence_list(graph: nx.DiGraph, raw: Any, *, required: bool = True) -> List[Dict[str, Any]]: + if not isinstance(raw, list) or (required and not raw): + raise ValueError("a non-empty evidence list is required") + return [_canonical_evidence(graph, item) for item in raw] + + +def _normalize_areas(raw: Any) -> List[Dict[str, Any]]: + if not isinstance(raw, list) or not raw: + raise ValueError("response must contain at least one feature area") + areas: List[Dict[str, Any]] = [] + used_ids: set[str] = set() + used_orders: set[Tuple[str, int]] = set() + for item in raw: + if not isinstance(item, dict): + raise ValueError("each feature area must be an object") + area_id = str(item.get("id") or "") + title = str(item.get("title") or "").strip() + summary = str(item.get("summary") or "").strip() + perspective = str(item.get("perspective") or "").strip() + order = item.get("order") + if not ID_RE.fullmatch(area_id) or area_id in used_ids: + raise ValueError(f"invalid or duplicate area id: {area_id or ''}") + if not title or not summary or perspective not in PERSPECTIVES: + raise ValueError( + f"area {area_id} requires title, summary, and product/technical perspective" + ) + if not isinstance(order, int) or order < 0 or (perspective, order) in used_orders: + raise ValueError( + f"area {area_id} requires a unique non-negative order within its perspective" + ) + used_ids.add(area_id) + used_orders.add((perspective, order)) + areas.append({ + "id": area_id, "title": title, "summary": summary, + "perspective": perspective, "order": order, + }) + rank = {"product": 0, "technical": 1} + return sorted(areas, key=lambda area: (rank[area["perspective"]], area["order"], area["id"])) + + +def _normalize_steps(graph: nx.DiGraph, raw: Any, status: str) -> List[Dict[str, Any]]: + if status == "pending": + if raw not in (None, []): + raise ValueError("pending workflows cannot contain steps") + return [] + if not isinstance(raw, list) or not raw: + raise ValueError(f"{status} workflows require at least one step") + steps: List[Dict[str, Any]] = [] + for index, item in enumerate(raw, 1): + if not isinstance(item, dict): + raise ValueError(f"workflow step {index} must be an object") + steps.append({ + "number": item.get("number"), + "phase": item.get("phase"), + "title": item.get("title"), + "text": item.get("text"), + "evidence": _evidence_list(graph, item.get("evidence")), + }) + return steps + + +def _normalize_feature( + graph: nx.DiGraph, + current_hash: str, + raw: Any, + area_ids: set[str], + used_ids: set[str], +) -> Tuple[Dict[str, Any], Dict[str, Any]]: + from .feature_workflows import relative_workflow_path + + feature_id, area_id, title, summary, audience = _feature_metadata( + raw, area_ids, used_ids + ) + evidence = _evidence_list(graph, raw.get("evidence")) + workflow = _workflow_record(graph, current_hash, raw, feature_id, title, summary, evidence) + feature = { + "id": feature_id, "area_id": area_id, "title": title, "audience": audience, + "summary": summary, "status": workflow["status"], + "workflow_path": relative_workflow_path(feature_id), "evidence": evidence, + } + if not validate_workflow(workflow): + raise ValueError(f"feature {feature_id} has an incomplete or invalid workflow") + return feature, workflow + + +def _feature_metadata( + raw: Any, area_ids: set[str], used_ids: set[str] +) -> Tuple[str, str, str, str, str]: + if not isinstance(raw, dict): + raise ValueError("each feature must be an object") + feature_id = str(raw.get("id") or "") + area_id = str(raw.get("area_id") or "") + title = str(raw.get("title") or "").strip() + summary = str(raw.get("summary") or "").strip() + audience = str(raw.get("audience") or "").strip() + if not ID_RE.fullmatch(feature_id) or feature_id in used_ids: + raise ValueError(f"invalid or duplicate feature id: {feature_id or ''}") + if area_id not in area_ids: + raise ValueError(f"feature {feature_id} references unknown area: {area_id or ''}") + if not title or not summary or audience not in AUDIENCES: + raise ValueError(f"feature {feature_id} requires title, summary, and a supported audience") + used_ids.add(feature_id) + return feature_id, area_id, title, summary, audience + + +def _workflow_record( + graph: nx.DiGraph, + current_hash: str, + raw: Dict[str, Any], + feature_id: str, + title: str, + summary: str, + evidence: List[Dict[str, Any]], +) -> Dict[str, Any]: + from .feature_workflows import WORKFLOW_GENERATOR, WORKFLOW_SCHEMA, _workflow_evidence + + raw_workflow = raw.get("workflow") + if not isinstance(raw_workflow, dict): + raise ValueError(f"feature {feature_id} requires a workflow object") + status = str(raw_workflow.get("status") or "") + if status not in STATUSES: + raise ValueError(f"feature {feature_id} has unsupported workflow status: {status or ''}") + missing = str(raw_workflow.get("missing_coverage") or "").strip() + if status in {"partial", "pending"} and not missing: + raise ValueError(f"feature {feature_id} requires missing_coverage for {status} status") + steps = _normalize_steps(graph, raw_workflow.get("steps"), status) + evidence_seed = {"evidence": evidence} + return { + "schema": WORKFLOW_SCHEMA, "graph_hash": current_hash, "generator": WORKFLOW_GENERATOR, + "feature_id": feature_id, "title": title, + "summary": str(raw_workflow.get("summary") or summary).strip(), + "status": status, "missing_coverage": missing, "evidence": evidence, + "evidence_nodes": _workflow_evidence(graph, evidence_seed), "steps": steps, + } + + +def normalize_response( + graph: nx.DiGraph, current_hash: str, payload: Any +) -> Tuple[Dict[str, Any], Dict[str, Dict[str, Any]]]: + from .feature_workflows import FEATURE_SCHEMA + + if not isinstance(payload, dict) or payload.get("schema") != RESPONSE_SCHEMA: + raise ValueError(f"response schema must be {RESPONSE_SCHEMA}") + if payload.get("graph_hash") != current_hash: + raise ValueError("response graph_hash does not match the current graph") + areas = _normalize_areas(payload.get("areas")) + raw_features = payload.get("features") + if not isinstance(raw_features, list) or not raw_features: + raise ValueError("response must contain at least one feature") + area_ids = {area["id"] for area in areas} + used_ids: set[str] = set() + features: List[Dict[str, Any]] = [] + workflows: Dict[str, Dict[str, Any]] = {} + for raw in raw_features: + feature, workflow = _normalize_feature(graph, current_hash, raw, area_ids, used_ids) + features.append(feature) + workflows[feature["id"]] = workflow + manifest = { + "schema": FEATURE_SCHEMA, "graph_hash": current_hash, + "generator": "feature-catalog-subagent@2", "areas": areas, "features": features, + } + return manifest, workflows diff --git a/tldrgraph/feature_workflow_validation.py b/tldrgraph/feature_workflow_validation.py index 29ba581..44aa999 100644 --- a/tldrgraph/feature_workflow_validation.py +++ b/tldrgraph/feature_workflow_validation.py @@ -5,7 +5,8 @@ from typing import Any, Dict, Iterable, List, Set -WORKFLOW_SCHEMA = "codechakra/feature-workflow@1" +WORKFLOW_SCHEMA = "codechakra/feature-workflow@2" +LEGACY_WORKFLOW_SCHEMA = "codechakra/feature-workflow@1" BANNED_WORKFLOW_RELATIONS = {"llm_http_route_link", "http_route_link", "calls_endpoint"} FRONTEND_MARKERS = ("frontend/", "/app/", "/pages/", "/components/") BACKEND_MARKERS = ("backend/", "/backend/", "server/", "/server/", "/routes/", "controller") @@ -16,6 +17,10 @@ "database": "persistence", "result": "response", } +ALLOWED_PHASES = { + "user_action", "frontend", "request", "backend", "persistence", + "external", "response", "ui_update", +} SCAFFOLD_MARKERS = ( "pending_reason", "instructions", @@ -24,17 +29,22 @@ def validate_workflow(workflow: Dict[str, Any]) -> bool: - if workflow.get("schema") != WORKFLOW_SCHEMA: + if workflow.get("schema") not in {WORKFLOW_SCHEMA, LEGACY_WORKFLOW_SCHEMA}: return False steps = workflow.get("steps") + status = workflow.get("status") + if status not in {"generated", "partial", "pending"}: + return False if not isinstance(steps, list) or not steps: - return workflow.get("status") == "pending" + return status == "pending" and _non_empty(workflow.get("missing_coverage")) + if status == "pending": + return False if not _steps_have_required_shape(steps): return False if _uses_banned_relation(steps): return False - if workflow.get("status") != "generated": - return True + if status == "partial": + return _non_empty(workflow.get("missing_coverage")) return _validate_generated_workflow(workflow, steps) @@ -71,12 +81,12 @@ def _validate_generated_workflow(workflow: Dict[str, Any], steps: List[Dict[str, def _steps_have_required_shape(steps: List[Dict[str, Any]]) -> bool: - for step in steps: + for expected_number, step in enumerate(steps, 1): if not isinstance(step, dict): return False - if not isinstance(step.get("number"), int): + if step.get("number") != expected_number: return False - if not _non_empty(step.get("phase")): + if _normalize_phase(step.get("phase")) not in ALLOWED_PHASES: return False if not _non_empty(step.get("title")) or not _non_empty(step.get("text")): return False diff --git a/tldrgraph/feature_workflows.py b/tldrgraph/feature_workflows.py index a4e423c..20bb2fc 100644 --- a/tldrgraph/feature_workflows.py +++ b/tldrgraph/feature_workflows.py @@ -15,9 +15,11 @@ FEATURES_FILENAME = "features.yaml" WORKFLOWS_DIRNAME = "workflows" -FEATURE_SCHEMA = "codechakra/features@1" -WORKFLOW_SCHEMA = "codechakra/feature-workflow@1" -WORKFLOW_GENERATOR = "feature-workflow-subagent@1" +FEATURE_SCHEMA = "codechakra/features@2" +LEGACY_FEATURE_SCHEMA = "codechakra/features@1" +WORKFLOW_SCHEMA = "codechakra/feature-workflow@2" +LEGACY_WORKFLOW_SCHEMA = "codechakra/feature-workflow@1" +WORKFLOW_GENERATOR = "feature-workflow-subagent@2" BANNED_WORKFLOW_RELATIONS = {"llm_http_route_link", "http_route_link", "calls_endpoint"} SKIP_DIRS = (".tldrgraph/", "tests/", "test/", "spec/", "__tests__/", "node_modules/", "dist/", "build/", "vendor/", "migrations/") @@ -197,8 +199,12 @@ def generate_feature_workflow_files( manifest = manifest or current_manifest(root, current_hash) if manifest is not None: clear_feature_workflow_request(root) - count = len(manifest.get("features") or []) - return {"features": count, "generated": count, "pending": 0, + features = manifest.get("features") or [] + generated = sum(1 for feature in features if feature.get("status") == "generated") + partial = sum(1 for feature in features if feature.get("status") == "partial") + catalog_pending = sum(1 for feature in features if feature.get("status") == "pending") + return {"features": len(features), "generated": generated, "partial": partial, + "workflow_pending": catalog_pending, "pending": 0, "graph_hash": current_hash, "agent_reason": "", "request_path": ""} request_path = write_feature_workflow_request(root, graph, current_hash, error) return {"features": 0, "generated": 0, "pending": 1, diff --git a/tldrgraph/installer_contract.py b/tldrgraph/installer_contract.py index 7044b81..098ce1d 100644 --- a/tldrgraph/installer_contract.py +++ b/tldrgraph/installer_contract.py @@ -42,6 +42,8 @@ def generate_layers_prose(registry: Optional[LayerRegistry] = None) -> str: heuristic features. It writes `.tldrgraph/feature_workflows_request.yaml`; the host coding agent must delegate that request to a source-reading subagent and rerun init after `.tldrgraph/feature_workflows_response.yaml` is written. +The response is a repository-wide catalog: inferred product/technical areas contain +human capabilities, while ranked symbols are investigation leads rather than features. `--agent-cli` is now explicit opt-in for architecture layer design and enrichment. Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless @@ -71,7 +73,9 @@ def generate_layers_prose(registry: Optional[LayerRegistry] = None) -> str: on every run. - **Complete `needs_feature_workflows` through a subagent.** Delegate the entire `.tldrgraph/feature_workflows_request.yaml` file to a source-reading subagent, - have it write both features and workflows to the named response file, then rerun init. + have it inspect the whole repository and write areas plus generated, partial, or pending + capability workflows to the named response file, then rerun init. A feature is an outcome + such as "Natural-language project creation," not a symbol such as `OpencodeService`. - **Complete `needs_llm_links` when shown.** Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend files, and write `.tldrgraph/llm_links_response.yaml` with `{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. diff --git a/tldrgraph/visualizer/assets/app.js b/tldrgraph/visualizer/assets/app.js index 47930a1..a599097 100644 --- a/tldrgraph/visualizer/assets/app.js +++ b/tldrgraph/visualizer/assets/app.js @@ -2650,10 +2650,10 @@ function initWorkflowsExplorer() { const workflows = DATA.workflows || []; const badgeEl = document.getElementById('flows-badge-count'); const readyCount = workflows.filter(w => (w.status || 'generated') === 'generated').length; - if (badgeEl) badgeEl.textContent = readyCount; + if (badgeEl) badgeEl.textContent = workflows.length ? `${readyCount}/${workflows.length}` : '0'; const countEl = document.getElementById('flows-list-count'); - if (countEl) countEl.textContent = `Workflows (${readyCount}/${workflows.length})`; + if (countEl) countEl.textContent = `Features (${readyCount}/${workflows.length} complete)`; // Setup search input const searchInput = document.getElementById('flows-search-input'); @@ -2704,72 +2704,6 @@ function initWorkflowsExplorer() { renderWorkflowsList(); } -function renderWorkflowsList() { - const listEl = document.getElementById('flows-list'); - if (!listEl) return; - - const allWorkflows = DATA.workflows || []; - const workflows = (DATA.workflows || []).filter(w => { - if (!flowSearchQuery) return true; - const matchTitle = (w.title || '').toLowerCase().includes(flowSearchQuery); - const matchRoot = (w.root_node || '').toLowerCase().includes(flowSearchQuery); - const matchFile = (w.file || '').toLowerCase().includes(flowSearchQuery); - const matchSummary = (w.summary || '').toLowerCase().includes(flowSearchQuery); - const matchSteps = (w.steps || []).some(s => - (s.symbol || '').toLowerCase().includes(flowSearchQuery) || - (s.file || '').toLowerCase().includes(flowSearchQuery) - ); - return matchTitle || matchRoot || matchFile || matchSummary || matchSteps; - }); - - const countEl = document.getElementById('flows-list-count'); - const readyCount = workflows.filter(w => (w.status || 'generated') === 'generated').length; - if (countEl) countEl.textContent = `Workflows (${readyCount}/${workflows.length})`; - - if (workflows.length === 0) { - const state = (DATA.workflow_state || {}).state || 'missing_features'; - const messages = { - missing_features: 'No feature manifest found. Run tldrgraph init to create .tldrgraph/features.yaml.', - invalid_features: '.tldrgraph/features.yaml is invalid. Run tldrgraph init to refresh it.', - stale_features: 'Saved feature workflows are stale. Complete the host-agent subagent handoff from .tldrgraph/feature_workflows_request.yaml.', - empty_features: 'No features were saved for this project yet.', - ready: allWorkflows.length ? 'No matching saved workflows found.' : 'No saved feature workflows found.', - }; - listEl.innerHTML = `
${escapeHtml(messages[state] || messages.ready)}
`; - selectWorkflow(null); - return; - } - - listEl.innerHTML = workflows.map(w => { - const isActive = w.id === activeWorkflowId; - const lColor = getLayerColor(w.layer_id); - const layerBadges = (w.layers_involved || []).slice(0, 3).map(lname => { - return `${escapeHtml(lname)}`; - }).join(''); - - return ` -
-
- ${escapeHtml(w.title)} - ${(w.status || 'generated') === 'generated' ? `${w.step_count} steps` : 'pending'} -
-
- ${escapeHtml(w.category || w.layer || 'Feature')} -
-
${escapeHtml(w.summary || '')}
-
${(w.status || 'generated') === 'generated' ? layerBadges : `${escapeHtml(w.pending_reason || 'Workflow file pending')}`}
-
- `; - }).join(''); - - listEl.querySelectorAll('.flow-card').forEach(card => { - card.addEventListener('click', () => { - const flowId = card.getAttribute('data-flow-id'); - selectWorkflow(flowId); - }); - }); -} - let flowCanvas = null; let flowCtx = null; let flowWidth = 800; @@ -3025,7 +2959,7 @@ let flowRowWidth = 0; function shapeSize(el) { if (el.kind === 'step') return { w: TASK_W + 44, h: TASK_H }; if (el.kind === 'gateway') return { w: GATE_SIZE, h: GATE_SIZE }; - if (el.kind === 'start' || el.kind === 'end' || el.kind === 'handoff' || el.kind === 'error') { + if (el.kind === 'start' || el.kind === 'end' || el.kind === 'partial' || el.kind === 'handoff' || el.kind === 'error') { return { w: EVENT_SIZE, h: EVENT_SIZE }; } return { w: TASK_W, h: TASK_H }; @@ -3193,7 +3127,8 @@ function buildWorkflowLayout(w) { // the decisions belong underneath. const head = group.head; const stepMeta = (w.steps || [])[group.step - 1] || {}; - const bare = head.kind === 'start' || (head.kind === 'end' && group.members.length === 1); + const bare = head.kind === 'start' || + ((head.kind === 'end' || head.kind === 'partial') && group.members.length === 1); const lineNode = bare ? head : { ...head, id: 'step__' + group.step, @@ -3300,14 +3235,19 @@ function showFlowTooltip(node, mouseX, mouseY) { const source = (node.node_id && typeof nodesById !== 'undefined') ? nodesById[node.node_id] : null; const layer = layerById[(source || {}).layer_id] || FALLBACK_COLOR; const where = node.file ? node.file + (node.line ? ':' + node.line : '') : ''; - const code = node.detail && node.detail !== node.label ? node.detail : ''; + const detail = node.detail && node.detail !== node.label ? node.detail : ''; + const symbol = node.source_symbol || ''; flowTooltipEl.innerHTML = '
*' + '' + escapeHtml(node.label) + '
' + (where ? '
' + escapeHtml(where) + '
' : '') + - (code ? '
' + - escapeHtml(code) + '
' : '') + + (detail ? '
' + + escapeHtml(detail) + '
' : '') + + (node.phase ? '
' + + escapeHtml(phaseLabel(node.phase)) + '
' : '') + + (symbol ? '
' + + escapeHtml(symbol) + '
' : '') + (node.external ? '
Leaves the tool: ' + escapeHtml(node.external) + '
' : ''); @@ -3349,10 +3289,21 @@ const KIND_COLORS = { loop: { fill: '#141829', border: '#38bdf8' }, start: { fill: '#0f2417', border: '#34d399' }, end: { fill: '#241318', border: '#f87171' }, + partial: { fill: '#2a2110', border: '#fbbf24' }, handoff: { fill: '#161d2e', border: '#94a3b8' }, error: { fill: '#2a1a12', border: '#fb923c' }, }; +const PHASE_COLORS = { + user_action: '#38bdf8', frontend: '#818cf8', request: '#a78bfa', + backend: '#22c55e', persistence: '#f59e0b', external: '#c084fc', + response: '#14b8a6', ui_update: '#06b6d4', +}; + +function phaseLabel(phase) { + return String(phase || 'system').replaceAll('_', ' '); +} + // Who does the work, shown on the card itself now that there are no lanes. const ACTOR_MARKS = { user: { dot: '#38bdf8', text: 'You' }, @@ -3396,10 +3347,14 @@ function drawRoundedTask(c, n, active, hovered) { hovered: hovered, emphasis: n.kind === 'step', dead: false, - metaText: '', + metaText: phaseLabel(n.phase), badge: n.kind === 'step' && n.step ? ('#' + n.step) : (n.kind === 'loop' ? '\u21bb' : (n.external ? n.external : null)), }); + c.save(); + c.fillStyle = PHASE_COLORS[n.phase] || '#64748b'; + c.fillRect(n.x - n.w / 2 + 1, n.y - n.h / 2 + 1, n.w - 2, 4); + c.restore(); } // Decisions and events keep their BPMN outline but wear the card's colours and @@ -3678,7 +3633,13 @@ function selectWorkflow(flowId) { const emptyState = document.getElementById('flows-empty-state'); const headerCard = document.getElementById('flow-header-card'); if (!w) { - if (emptyState) emptyState.style.display = 'flex'; + if (emptyState) { + emptyState.style.display = 'flex'; + const heading = emptyState.querySelector('h3'); + const copy = emptyState.querySelector('p'); + if (heading) heading.textContent = 'Select a capability to explore'; + if (copy) copy.textContent = 'Follow a source-backed feature from the initiating action to its visible result.'; + } if (headerCard) headerCard.style.display = 'none'; flowNodes = []; flowEdges = []; @@ -3700,20 +3661,34 @@ function selectWorkflow(flowId) { if (titleEl) titleEl.textContent = w.title; if (badgeEl) { badgeEl.textContent = w.category || w.layer || 'Architecture'; - badgeEl.style.background = getLayerColor(w.layer_id); + badgeEl.style.background = w.perspective === 'product' ? '#0369a1' : '#6d28d9'; } if (summaryEl) summaryEl.textContent = w.summary; - if (stepsMetaEl) stepsMetaEl.textContent = (w.status || 'generated') === 'generated' - ? `${w.step_count} Saved Steps` - : 'Workflow Pending'; - if (entryMetaEl) entryMetaEl.textContent = (w.status || 'generated') === 'generated' - ? `Starts with: ${w.root_node || w.title}` - : (w.pending_reason || 'The workflow file has not been generated yet.'); + const status = w.status || 'generated'; + if (stepsMetaEl) stepsMetaEl.textContent = status === 'generated' + ? `${w.step_count} Proven Steps` + : (status === 'partial' ? `${w.step_count} Known Steps · Partial` : 'Workflow Pending'); + if (entryMetaEl) entryMetaEl.textContent = status === 'generated' + ? `Starts with: ${(w.steps[0] || {}).display_label || w.title}` + : (w.missing_coverage || w.pending_reason || 'The reliable sequence is not yet known.'); if (layersMetaEl) { - layersMetaEl.innerHTML = (w.layers_involved || []).map(lname => { - return `${escapeHtml(lname)}`; - }).join(''); + layersMetaEl.innerHTML = `${escapeHtml(w.audience || 'developer')}` + + `${escapeHtml(status)}`; + } + + if (status === 'pending') { + if (emptyState) { + emptyState.style.display = 'flex'; + const heading = emptyState.querySelector('h3'); + const copy = emptyState.querySelector('p'); + if (heading) heading.textContent = 'Workflow evidence is pending'; + if (copy) copy.textContent = w.missing_coverage || w.pending_reason || 'No reliable sequence can be drawn yet.'; + } + flowNodes = []; + flowEdges = []; + requestFlowFrame(); + return; } // Build Layout and Render on Workflow Canvas diff --git a/tldrgraph/visualizer/assets/index.html b/tldrgraph/visualizer/assets/index.html index ea3dc19..db18442 100644 --- a/tldrgraph/visualizer/assets/index.html +++ b/tldrgraph/visualizer/assets/index.html @@ -6,6 +6,7 @@ TLDRGraph 🌐 Architecture Visualizer @@ -75,13 +76,13 @@
🔍 - +
Workflows (0) - Architecture Flows + Capability Catalog
@@ -238,6 +239,7 @@

Select a workflow to explore

const HIERARCHY = DATA; const LAYERS_CONFIG = /*__LAYERS_JSON__*/; /*__SOURCEVIEW_JS__*/ +/*__WORKFLOW_CATALOG_JS__*/ /*__APP_JS__*/ diff --git a/tldrgraph/visualizer/assets/workflow-catalog.css b/tldrgraph/visualizer/assets/workflow-catalog.css new file mode 100644 index 0000000..9056410 --- /dev/null +++ b/tldrgraph/visualizer/assets/workflow-catalog.css @@ -0,0 +1,85 @@ +.flows-list-message { + padding: 20px 16px; + text-align: center; + font-size: 12px; + line-height: 1.5; + color: var(--text-dim); +} + +.flow-perspective { margin-bottom: 18px; } +.flow-perspective-title { + padding: 10px 16px 7px; + color: #94a3b8; + font-size: 10px; + font-weight: 800; + letter-spacing: .12em; + text-transform: uppercase; +} + +.flow-area { + margin: 0 10px 10px; + border: 1px solid rgba(148, 163, 184, .14); + border-radius: 12px; + overflow: hidden; + background: rgba(15, 23, 42, .34); +} + +.flow-area-header { + width: 100%; + display: grid; + grid-template-columns: 18px 1fr auto; + gap: 8px; + align-items: center; + border: 0; + padding: 11px 12px; + color: #e2e8f0; + background: rgba(30, 41, 59, .62); + cursor: pointer; + text-align: left; +} + +.flow-area-chevron { color: #7dd3fc; transition: transform .16s ease; } +.flow-area.collapsed .flow-area-chevron { transform: rotate(-90deg); } +.flow-area-copy { display: flex; min-width: 0; flex-direction: column; gap: 3px; } +.flow-area-title { font-size: 13px; font-weight: 750; } +.flow-area-summary { + overflow: hidden; + color: #7f8da7; + font-size: 10.5px; + line-height: 1.3; + text-overflow: ellipsis; + white-space: nowrap; +} +.flow-area-count { color: #94a3b8; font-size: 10px; } +.flow-area-features { padding: 8px; } +.flow-area.collapsed .flow-area-features { display: none; } + +.flow-area .flow-card { + width: 100%; + margin: 0 0 7px; + color: inherit; + font: inherit; + text-align: left; +} +.flow-area .flow-card:last-child { margin-bottom: 0; } + +.flow-status-badge, +.flow-audience-badge, +.flow-coverage-note { + flex: 0 0 auto; + border-radius: 999px; + padding: 3px 7px; + font-size: 9.5px; + font-weight: 750; + letter-spacing: .02em; +} +.flow-status-badge.generated { color: #6ee7b7; background: rgba(16, 185, 129, .14); } +.flow-status-badge.partial { color: #fcd34d; background: rgba(245, 158, 11, .15); } +.flow-status-badge.pending { color: #cbd5e1; background: rgba(100, 116, 139, .18); } +.flow-audience-badge { color: #bae6fd; background: rgba(14, 165, 233, .13); text-transform: capitalize; } +.flow-coverage-note { color: #fcd34d; } + +@media (max-width: 760px) { + .flows-sidebar { width: min(86vw, 360px); } + .flow-area-summary { display: none; } +} diff --git a/tldrgraph/visualizer/assets/workflow-catalog.js b/tldrgraph/visualizer/assets/workflow-catalog.js new file mode 100644 index 0000000..d23016a --- /dev/null +++ b/tldrgraph/visualizer/assets/workflow-catalog.js @@ -0,0 +1,132 @@ +/* Grouped, searchable feature catalog for Workflow Explorer. */ + +const collapsedWorkflowAreas = new Set(); + +function workflowMatches(w, area, query) { + if (!query) return true; + const values = [ + w.title, w.root_node, w.file, w.summary, w.audience, + area.title, area.summary, area.perspective, + ]; + (w.steps || []).forEach(step => values.push( + step.display_label, step.intent, step.symbol, step.file, step.phase + )); + return values.some(value => String(value || '').toLowerCase().includes(query)); +} + +function workflowAreas() { + const configured = DATA.workflow_areas || []; + if (configured.length) return configured; + const seen = new Map(); + (DATA.workflows || []).forEach(w => { + const id = w.area_id || 'legacy_features'; + if (!seen.has(id)) seen.set(id, { + id: id, + title: w.area_title || 'Legacy features', + summary: w.area_summary || '', + perspective: w.perspective || 'technical', + order: w.area_order || 0, + }); + }); + return Array.from(seen.values()); +} + +function statusLabel(workflow) { + const status = workflow.status || 'generated'; + if (status === 'generated') return `${workflow.step_count} steps`; + if (status === 'partial') return `${workflow.step_count} known`; + return 'pending'; +} + +function featureCardHtml(w) { + const status = w.status || 'generated'; + const isActive = w.id === activeWorkflowId; + return ` + `; +} + +function areaHtml(area, workflows, searching) { + const collapsed = !searching && collapsedWorkflowAreas.has(area.id); + const generated = workflows.filter(w => (w.status || 'generated') === 'generated').length; + return ` +
+ +
${workflows.map(featureCardHtml).join('')}
+
`; +} + +function perspectiveHtml(perspective, areas, workflows, searching) { + const label = perspective === 'product' ? 'Product capabilities' : 'Technical capabilities'; + const body = areas.map(area => { + const members = workflows.filter(w => w.area_id === area.id); + return members.length ? areaHtml(area, members, searching) : ''; + }).join(''); + return body ? `
${label}
${body}
` : ''; +} + +function renderWorkflowsList() { + const listEl = document.getElementById('flows-list'); + if (!listEl) return; + + const allWorkflows = DATA.workflows || []; + const areas = workflowAreas(); + const areaById = new Map(areas.map(area => [area.id, area])); + const workflows = allWorkflows.filter(w => workflowMatches( + w, areaById.get(w.area_id) || {}, flowSearchQuery + )); + const countEl = document.getElementById('flows-list-count'); + const complete = workflows.filter(w => (w.status || 'generated') === 'generated').length; + if (countEl) countEl.textContent = `Features (${complete}/${workflows.length} complete)`; + + if (!workflows.length) { + const state = (DATA.workflow_state || {}).state || 'missing_features'; + const messages = { + missing_features: 'No feature catalog found. Run tldrgraph init to create it.', + invalid_features: 'The saved feature catalog is invalid. Run tldrgraph init to refresh it.', + stale_features: 'The saved catalog is stale. Complete the feature-workflow handoff.', + empty_features: 'No source-backed capabilities were saved for this project.', + ready: allWorkflows.length ? 'No matching capabilities found.' : 'No saved capabilities found.', + }; + listEl.innerHTML = `
${escapeHtml(messages[state] || messages.ready)}
`; + selectWorkflow(null); + return; + } + + const orderedAreas = [...areas].sort((a, b) => (a.order || 0) - (b.order || 0)); + listEl.innerHTML = ['product', 'technical'].map(perspective => perspectiveHtml( + perspective, + orderedAreas.filter(area => (area.perspective || 'technical') === perspective), + workflows, + Boolean(flowSearchQuery) + )).join(''); + + listEl.onclick = event => { + const toggle = event.target.closest('[data-toggle-area]'); + if (toggle) { + const id = toggle.getAttribute('data-toggle-area'); + if (collapsedWorkflowAreas.has(id)) collapsedWorkflowAreas.delete(id); + else collapsedWorkflowAreas.add(id); + renderWorkflowsList(); + return; + } + const card = event.target.closest('[data-flow-id]'); + if (card) selectWorkflow(card.getAttribute('data-flow-id')); + }; +} diff --git a/tldrgraph/visualizer/data.py b/tldrgraph/visualizer/data.py index 0dca344..ec2a027 100644 --- a/tldrgraph/visualizer/data.py +++ b/tldrgraph/visualizer/data.py @@ -383,6 +383,7 @@ def prepare_visualizer_data(root_dir: str) -> Dict[str, Any]: "modules": modules, "nodes": list(nodes_by_id.values()), "workflows": workflows, + "workflow_areas": workflow_payload.get("areas", []), "workflow_state": {k: v for k, v in workflow_payload.items() if k != "workflows"}, "module_edges": module_edges, "child_edges": child_edges, diff --git a/tldrgraph/visualizer/render.py b/tldrgraph/visualizer/render.py index 6407ae3..9230485 100644 --- a/tldrgraph/visualizer/render.py +++ b/tldrgraph/visualizer/render.py @@ -43,9 +43,11 @@ def render_html(data: Dict[str, Any], layers_config: List[Dict[str, Any]]) -> st template = _read_asset("index.html") replacements = { "/*__STYLES__*/": _read_asset("app.css"), + "/*__WORKFLOW_CATALOG_STYLES__*/": _read_asset("workflow-catalog.css"), "/*__DATA_JSON__*/": _json_for_script(data), "/*__LAYERS_JSON__*/": _json_for_script(layers_config), "/*__SOURCEVIEW_JS__*/": _read_asset("sourceview.js"), + "/*__WORKFLOW_CATALOG_JS__*/": _read_asset("workflow-catalog.js"), "/*__APP_JS__*/": _read_asset("app.js"), } pattern = re.compile("|".join(re.escape(k) for k in replacements.keys())) From b5789e314c8611d45cc29f9410d9aee93de6cfa0 Mon Sep 17 00:00:00 2001 From: Ashwani Kharwar Date: Mon, 14 Sep 2026 16:54:31 +0530 Subject: [PATCH 04/13] optional rendering --- tldrgraph/visualizer/assets/app.js | 6 +++++- tldrgraph/visualizer/assets/workflow-catalog.js | 10 +++++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/tldrgraph/visualizer/assets/app.js b/tldrgraph/visualizer/assets/app.js index a599097..e1e8b19 100644 --- a/tldrgraph/visualizer/assets/app.js +++ b/tldrgraph/visualizer/assets/app.js @@ -2653,7 +2653,11 @@ function initWorkflowsExplorer() { if (badgeEl) badgeEl.textContent = workflows.length ? `${readyCount}/${workflows.length}` : '0'; const countEl = document.getElementById('flows-list-count'); - if (countEl) countEl.textContent = `Features (${readyCount}/${workflows.length} complete)`; + if (countEl) { + // Do not present a completion summary until there is a completed feature. + countEl.hidden = readyCount === 0; + countEl.textContent = `Features (${readyCount}/${workflows.length} complete)`; + } // Setup search input const searchInput = document.getElementById('flows-search-input'); diff --git a/tldrgraph/visualizer/assets/workflow-catalog.js b/tldrgraph/visualizer/assets/workflow-catalog.js index d23016a..52081e8 100644 --- a/tldrgraph/visualizer/assets/workflow-catalog.js +++ b/tldrgraph/visualizer/assets/workflow-catalog.js @@ -93,7 +93,15 @@ function renderWorkflowsList() { )); const countEl = document.getElementById('flows-list-count'); const complete = workflows.filter(w => (w.status || 'generated') === 'generated').length; - if (countEl) countEl.textContent = `Features (${complete}/${workflows.length} complete)`; + if (countEl) { + const completedFeatures = allWorkflows.filter( + w => (w.status || 'generated') === 'generated' + ).length; + // Search results must not make the summary disappear; only the absence of + // completed features does. + countEl.hidden = completedFeatures === 0; + countEl.textContent = `Features (${complete}/${workflows.length} complete)`; + } if (!workflows.length) { const state = (DATA.workflow_state || {}).state || 'missing_features'; From caeb30c472ac9bb3ddf4fed3f65553420ca438d2 Mon Sep 17 00:00:00 2001 From: Ashwani Kharwar Date: Mon, 14 Sep 2026 17:02:17 +0530 Subject: [PATCH 05/13] flow diagram show in vertical way --- tldrgraph/visualizer/assets/app.js | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/tldrgraph/visualizer/assets/app.js b/tldrgraph/visualizer/assets/app.js index e1e8b19..cd950fd 100644 --- a/tldrgraph/visualizer/assets/app.js +++ b/tldrgraph/visualizer/assets/app.js @@ -2935,17 +2935,16 @@ function fitWorkflowView(animate) { } // ------------------------------------------------------------- -// Layout: the journey runs as one straight line of steps, and what happens -// inside a step - its decisions, each outcome, its failure paths - hangs -// underneath it. Read the line to follow the story; look down from a step to -// see how it decides. +// Layout: the journey runs top to bottom, and what happens inside a step - its +// decisions, each outcome, its failure paths - hangs underneath it. Read down +// the page to follow the story; look below a step to see how it decides. // ------------------------------------------------------------- const NODE_GAP = 96; // space between two steps on the line const BRANCH_TOP = 118; // drop from the line to the first shape below const BRANCH_STEP = 92; // drop between shapes down a branch const BRANCH_GAP = 30; // space between two shapes side by side const BRANCH_PER_LEVEL = 2; // branch shapes across before stacking down -const SPINE_PER_ROW = 5; // steps on the line before it wraps +const SPINE_PER_ROW = 1; // one journey step per vertical row const ROW_DROP = 150; // space under a row's deepest branch const STEP_LABEL_H = 18; // breathing room above a row const TASK_W = 268; // wider than a symbol card: the titles are sentences @@ -3193,8 +3192,8 @@ function buildWorkflowLayout(w) { closeRow(groups.length); - // The line joins step to step; each step drops into its own internals; the - // shapes inside a step keep the flows the extractor found between them. + // The line joins steps vertically; each step drops into its own internals; + // the shapes inside a step keep the flows the extractor found between them. const placedById = new Map(flowNodes.map(n => [n.id, n])); const stepOf = new Map(visible.map(e => [e.id, e.step === undefined || e.step === null ? 0 : e.step])); const rebuilt = []; @@ -3212,12 +3211,7 @@ function buildWorkflowLayout(w) { rebuilt.push({ ...f, source: source }); }); - flowEdges = rebuilt.map(f => { - const a = placedById.get(f.source); - const b = placedById.get(f.target); - if (a && b && a.row !== b.row && f.kind !== 'loop_back') return { ...f, kind: 'wrap' }; - return f; - }); + flowEdges = rebuilt; flowRowWidth = widest; flowBounds = { @@ -3576,7 +3570,7 @@ function drawWorkflowCanvas() { flowCtx.translate(flowPanX, flowPanY); flowCtx.scale(flowScale, flowScale); - // 1. The main line each row runs along, drawn behind everything else. + // 1. The main journey is connected by the vertical sequence connectors. flowRows.forEach(row => { const onRow = flowNodes.filter(n => n.onSpine && n.row === row.index); if (onRow.length < 2) return; From b59324c2d0df9c9f4783180e65ad8ebc8dec377f Mon Sep 17 00:00:00 2001 From: Ashwani Kharwar Date: Tue, 15 Sep 2026 13:49:24 +0530 Subject: [PATCH 06/13] refactor: replace static graph analysis with source-backed workflow catalog - simplify TLDRGraph around agent-driven feature and workflow extraction - add source inventory, workflow contracts, validation, and visualizer support - remove legacy graph, BPMN, embedding, and benchmark pipelines - update CLI, installer, docs, agent integrations, and tests --- .agents/skills/tldrgraph-init/SKILL.md | 90 +- .claude/commands/tldrgraph-init.md | 90 +- .cursor/commands/tldrgraph-init.md | 90 +- .gitignore | 4 +- .tldrgraph/AGENT_CONTRACT.md | 359 +- AGENTS.md | 71 +- AGENT_CONTRACT.md | 333 +- LICENSE | 5 - README.md | 259 +- benchmarks/benchmark_swebench.py | 533 - benchmarks/enrich_real_corpus_manually.py | 227 - benchmarks/fetch_real_ast_corpus.py | 223 - benchmarks/swebench_lite_cache.json | 902 - benchmarks/swebench_real_ast_corpus.json | 17947 ---------------- docs/agents/contract-spec.md | 94 +- docs/agents/integrations.md | 52 +- docs/assets/architecture_map.png | Bin 485783 -> 0 bytes docs/benchmarks/swe-bench.md | 41 - docs/cli-reference/analysis-tools.md | 67 - docs/cli-reference/init-and-scan.md | 57 - docs/cli-reference/query-and-trace.md | 68 - docs/cli-reference/workflow-explorer.md | 16 + docs/concepts/dynamic-layers.md | 84 - docs/concepts/hash-gating.md | 33 - docs/concepts/seams-and-routes.md | 62 - docs/concepts/vector-retrieval.md | 40 - docs/getting-started/installation.md | 71 +- docs/getting-started/quickstart.md | 96 +- docs/index.md | 109 +- docs/llms-full.txt | 936 +- docs/llms.txt | 37 +- docs/visualizer/architecture-map.md | 34 - docs/visualizer/workflows-explorer.md | 43 +- mkdocs.yml | 17 +- pyproject.toml | 20 +- tests/conftest.py | 485 +- tests/test_agent_loop.py | 1054 - tests/test_auto_agent.py | 1174 - tests/test_bpmn.py | 440 - tests/test_code_health.py | 11 +- tests/test_dynamic_layers.py | 272 - tests/test_embeddings.py | 634 - tests/test_extractors.py | 923 - tests/test_feature_workflows.py | 536 - tests/test_flow_engine.py | 496 - tests/test_hash_gate.py | 344 - tests/test_hierarchy.py | 567 - tests/test_index_write_through.py | 219 - tests/test_installer.py | 29 + tests/test_layer_config.py | 366 - tests/test_layers.py | 331 - tests/test_llm_route_inference.py | 181 - tests/test_persistence.py | 274 - tests/test_scan_smoke.py | 98 - tests/test_source_inventory.py | 44 + tests/test_visualizer.py | 506 - tests/test_visualizer_source.py | 262 - tests/test_workflow_pipeline.py | 80 + tests/test_workflow_schema.py | 40 + tests/test_workflow_visualizer.py | 44 + tldrgraph/AGENT_CONTRACT.md | 46 + tldrgraph/__init__.py | 11 +- tldrgraph/agent_commands.py | 368 +- tldrgraph/agent_runner.py | 325 - tldrgraph/bpmn_enrichment.py | 215 - tldrgraph/bpmn_externals.py | 67 - tldrgraph/bpmn_extract.py | 304 - tldrgraph/bpmn_languages.py | 122 - tldrgraph/bpmn_process.py | 123 - tldrgraph/bpmn_treesitter.py | 302 - tldrgraph/call_resolver.py | 139 - tldrgraph/classifier.py | 75 - tldrgraph/cli.py | 409 +- tldrgraph/cli_agent_loop.py | 154 - tldrgraph/cli_bpmn.py | 52 - tldrgraph/cli_commands.py | 293 - tldrgraph/cli_enrichment.py | 387 - tldrgraph/cli_llm_links.py | 124 - tldrgraph/cli_pipeline.py | 408 +- tldrgraph/deadcode.py | 238 - tldrgraph/dense_embedder.py | 164 - tldrgraph/extractors.py | 190 - tldrgraph/extractors_client.py | 158 - tldrgraph/extractors_prisma.py | 251 - tldrgraph/extractors_route.py | 398 - tldrgraph/feature_workflow_bridges.py | 163 - tldrgraph/feature_workflow_handoff.py | 225 +- tldrgraph/feature_workflow_loader.py | 312 +- tldrgraph/feature_workflow_schema.py | 241 +- tldrgraph/feature_workflow_validation.py | 167 +- tldrgraph/feature_workflows.py | 222 +- tldrgraph/flow_engine.py | 254 - tldrgraph/flow_traversal.py | 222 - tldrgraph/graph_loader.py | 379 - tldrgraph/hash_gate.py | 128 - tldrgraph/hierarchy.py | 336 - tldrgraph/hierarchy_builder.py | 247 - tldrgraph/init_policy.py | 87 - tldrgraph/installer.py | 247 +- tldrgraph/installer_contract.py | 195 - tldrgraph/intent_quality.py | 21 - tldrgraph/labels.py | 118 - tldrgraph/layer_config.py | 180 - tldrgraph/layer_evidence.py | 231 - tldrgraph/layers.py | 390 - tldrgraph/llm_enricher.py | 352 - tldrgraph/llm_route_inference.py | 292 - tldrgraph/node_registrar.py | 310 - tldrgraph/paths.py | 89 - tldrgraph/payload.py | 35 + tldrgraph/propose_layers.py | 305 - tldrgraph/rules.py | 151 - tldrgraph/snapshot_sync.py | 286 - tldrgraph/source_inventory.py | 117 + tldrgraph/vector_store.py | 382 - tldrgraph/vector_tfidf.py | 137 - tldrgraph/visualizer/__init__.py | 30 +- tldrgraph/visualizer/action_labels.py | 81 - tldrgraph/visualizer/assets/app.css | 1545 +- tldrgraph/visualizer/assets/app.js | 3805 +--- tldrgraph/visualizer/assets/index.html | 275 +- tldrgraph/visualizer/assets/sourceview.js | 653 +- .../visualizer/assets/workflow-catalog.css | 85 - .../visualizer/assets/workflow-catalog.js | 140 - tldrgraph/visualizer/bpmn_data.py | 377 - tldrgraph/visualizer/bpmn_phrasebook.py | 353 - tldrgraph/visualizer/bpmn_phrasing.py | 70 - tldrgraph/visualizer/data.py | 402 +- tldrgraph/visualizer/flows_blueprints.py | 159 - tldrgraph/visualizer/flows_data.py | 325 - tldrgraph/visualizer/flows_discover.py | 289 - tldrgraph/visualizer/palette.py | 66 - tldrgraph/visualizer/render.py | 61 +- tldrgraph/visualizer/source.py | 385 - uv.lock | 1661 +- 135 files changed, 1550 insertions(+), 53876 deletions(-) delete mode 100644 benchmarks/benchmark_swebench.py delete mode 100644 benchmarks/enrich_real_corpus_manually.py delete mode 100644 benchmarks/fetch_real_ast_corpus.py delete mode 100644 benchmarks/swebench_lite_cache.json delete mode 100644 benchmarks/swebench_real_ast_corpus.json delete mode 100644 docs/assets/architecture_map.png delete mode 100644 docs/benchmarks/swe-bench.md delete mode 100644 docs/cli-reference/analysis-tools.md delete mode 100644 docs/cli-reference/init-and-scan.md delete mode 100644 docs/cli-reference/query-and-trace.md create mode 100644 docs/cli-reference/workflow-explorer.md delete mode 100644 docs/concepts/dynamic-layers.md delete mode 100644 docs/concepts/hash-gating.md delete mode 100644 docs/concepts/seams-and-routes.md delete mode 100644 docs/concepts/vector-retrieval.md delete mode 100644 docs/visualizer/architecture-map.md delete mode 100644 tests/test_agent_loop.py delete mode 100644 tests/test_auto_agent.py delete mode 100644 tests/test_bpmn.py delete mode 100644 tests/test_dynamic_layers.py delete mode 100644 tests/test_embeddings.py delete mode 100644 tests/test_extractors.py delete mode 100644 tests/test_feature_workflows.py delete mode 100644 tests/test_flow_engine.py delete mode 100644 tests/test_hash_gate.py delete mode 100644 tests/test_hierarchy.py delete mode 100644 tests/test_index_write_through.py create mode 100644 tests/test_installer.py delete mode 100644 tests/test_layer_config.py delete mode 100644 tests/test_layers.py delete mode 100644 tests/test_llm_route_inference.py delete mode 100644 tests/test_persistence.py delete mode 100644 tests/test_scan_smoke.py create mode 100644 tests/test_source_inventory.py delete mode 100644 tests/test_visualizer.py delete mode 100644 tests/test_visualizer_source.py create mode 100644 tests/test_workflow_pipeline.py create mode 100644 tests/test_workflow_schema.py create mode 100644 tests/test_workflow_visualizer.py create mode 100644 tldrgraph/AGENT_CONTRACT.md delete mode 100644 tldrgraph/agent_runner.py delete mode 100644 tldrgraph/bpmn_enrichment.py delete mode 100644 tldrgraph/bpmn_externals.py delete mode 100644 tldrgraph/bpmn_extract.py delete mode 100644 tldrgraph/bpmn_languages.py delete mode 100644 tldrgraph/bpmn_process.py delete mode 100644 tldrgraph/bpmn_treesitter.py delete mode 100644 tldrgraph/call_resolver.py delete mode 100644 tldrgraph/classifier.py delete mode 100644 tldrgraph/cli_agent_loop.py delete mode 100644 tldrgraph/cli_bpmn.py delete mode 100644 tldrgraph/cli_commands.py delete mode 100644 tldrgraph/cli_enrichment.py delete mode 100644 tldrgraph/cli_llm_links.py delete mode 100644 tldrgraph/deadcode.py delete mode 100644 tldrgraph/dense_embedder.py delete mode 100644 tldrgraph/extractors.py delete mode 100644 tldrgraph/extractors_client.py delete mode 100644 tldrgraph/extractors_prisma.py delete mode 100644 tldrgraph/extractors_route.py delete mode 100644 tldrgraph/feature_workflow_bridges.py delete mode 100644 tldrgraph/flow_engine.py delete mode 100644 tldrgraph/flow_traversal.py delete mode 100644 tldrgraph/graph_loader.py delete mode 100644 tldrgraph/hash_gate.py delete mode 100644 tldrgraph/hierarchy.py delete mode 100644 tldrgraph/hierarchy_builder.py delete mode 100644 tldrgraph/init_policy.py delete mode 100644 tldrgraph/installer_contract.py delete mode 100644 tldrgraph/intent_quality.py delete mode 100644 tldrgraph/labels.py delete mode 100644 tldrgraph/layer_config.py delete mode 100644 tldrgraph/layer_evidence.py delete mode 100644 tldrgraph/layers.py delete mode 100644 tldrgraph/llm_enricher.py delete mode 100644 tldrgraph/llm_route_inference.py delete mode 100644 tldrgraph/node_registrar.py delete mode 100644 tldrgraph/paths.py create mode 100644 tldrgraph/payload.py delete mode 100644 tldrgraph/propose_layers.py delete mode 100644 tldrgraph/rules.py delete mode 100644 tldrgraph/snapshot_sync.py create mode 100644 tldrgraph/source_inventory.py delete mode 100644 tldrgraph/vector_store.py delete mode 100644 tldrgraph/vector_tfidf.py delete mode 100644 tldrgraph/visualizer/action_labels.py delete mode 100644 tldrgraph/visualizer/assets/workflow-catalog.css delete mode 100644 tldrgraph/visualizer/assets/workflow-catalog.js delete mode 100644 tldrgraph/visualizer/bpmn_data.py delete mode 100644 tldrgraph/visualizer/bpmn_phrasebook.py delete mode 100644 tldrgraph/visualizer/bpmn_phrasing.py delete mode 100644 tldrgraph/visualizer/flows_blueprints.py delete mode 100644 tldrgraph/visualizer/flows_data.py delete mode 100644 tldrgraph/visualizer/flows_discover.py delete mode 100644 tldrgraph/visualizer/palette.py delete mode 100644 tldrgraph/visualizer/source.py diff --git a/.agents/skills/tldrgraph-init/SKILL.md b/.agents/skills/tldrgraph-init/SKILL.md index 94f15de..57bd8e4 100644 --- a/.agents/skills/tldrgraph-init/SKILL.md +++ b/.agents/skills/tldrgraph-init/SKILL.md @@ -1,92 +1,22 @@ --- name: tldrgraph-init -description: Build or continue this repository's TLDRGraph architecture graph (layers, extraction, enrichment) +description: Build or refresh source-backed feature workflows --- -# TLDRGraph: build this repository's architecture graph +# TLDRGraph: build the source-backed workflow catalog -In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select -`tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. - -One command handles extraction, feature-workflow handoff, and embeddings: +Run exactly: ```bash tldrgraph init ``` -TLDRGraph never launches an AI process for feature generation and never invents -heuristic features. When feature artifacts are missing or stale, it writes -`.tldrgraph/feature_workflows_request.yaml`. The coding agent that ran -`tldrgraph init` must delegate that request to a source-reading subagent. - -Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent -`needs_confirmation` response explicitly asks for approval. `--batch 200` means -all nodes in chunks; `--limit 200` means stop after only 200 nodes. -Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless the user -explicitly requests it. - -## Feature Workflow Explorer artifacts - -An accepted subagent response writes the v2 capability catalog in `.tldrgraph/features.yaml` -and flows in `.tldrgraph/workflows/.yaml`. The manifest groups concrete -capabilities into inferred product or technical areas; ranked symbols in the request are -investigation leads, not the feature list. -Workflow Explorer reads only those files; missing or invalid workflows show pending states. -Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. -Define features as user, admin, developer, or operator outcomes, not code symbols. Good: -"Natural-language project creation." Bad: "OpencodeService"; that service is evidence. -Each short plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. -Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. -Use `partial` with `missing_coverage` for a proven fragment and `pending` when no reliable -sequence can be drawn. List source-backed capabilities in either state without inventing steps. -Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. -When `init` reports `status: needs_feature_workflows`: - -1. Open `.tldrgraph/feature_workflows_request.yaml`. -2. Spawn a source-reading subagent and delegate the entire request to it. -3. Have the subagent inspect the whole repository and write areas, features, and generated/partial/pending workflows to `.tldrgraph/feature_workflows_response.yaml`. -4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. -5. Run `tldrgraph init` again. - -## `status: needs_layers` - -This should only appear when the user explicitly opted into architecture AI with -`--agent-cli` or an older TLDRGraph build is running. Do not complete this -handoff unless the user asked for architecture layer design. - -## `status: needs_confirmation` - -This belongs to explicit `--agent-cli` enrichment. Ask before continuing. +When the status is `needs_feature_workflows`: -## `status: needs_enrichment` - -This should only appear when the user explicitly opted into architecture -enrichment with `--agent-cli` or an older TLDRGraph build is running. Do not -process enrichment batches unless the user asked for full graph enrichment. - -**Copy every `id` verbatim.** A constructed id matches nothing, is dropped, and -gets reported back to you -- but the work is wasted. - -**Never invent `fields` or `calls`.** Omit what you cannot verify in the code: an -empty list is a correct answer, a wrong `calls` entry becomes a real wrong edge. - -## `status: needs_llm_links` - -Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend -files, then write `.tldrgraph/llm_links_response.yaml` as a YAML list of -`{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. -Only include source-backed links with file and line evidence. Run `tldrgraph init` -again. This continuation state only appears when the user explicitly opted into -route-link inference with `--llm-links`. - -## Once it says DONE - -```bash -tldrgraph query "" -tldrgraph trace "" "" -tldrgraph layers -tldrgraph ui --serve -``` +1. Read `.tldrgraph/feature_workflows_request.yaml` completely. +2. Spawn a source-reading subagent and delegate the entire request. +3. Have it inspect the repository and write the requested v3 response to + `.tldrgraph/feature_workflows_response.yaml`. +4. Run `tldrgraph init` again. -Read-only, and they never trigger enrichment. Full schema: -`.tldrgraph/AGENT_CONTRACT.md`. +Continue until the status is `done`. diff --git a/.claude/commands/tldrgraph-init.md b/.claude/commands/tldrgraph-init.md index 94f15de..57bd8e4 100644 --- a/.claude/commands/tldrgraph-init.md +++ b/.claude/commands/tldrgraph-init.md @@ -1,92 +1,22 @@ --- name: tldrgraph-init -description: Build or continue this repository's TLDRGraph architecture graph (layers, extraction, enrichment) +description: Build or refresh source-backed feature workflows --- -# TLDRGraph: build this repository's architecture graph +# TLDRGraph: build the source-backed workflow catalog -In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select -`tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. - -One command handles extraction, feature-workflow handoff, and embeddings: +Run exactly: ```bash tldrgraph init ``` -TLDRGraph never launches an AI process for feature generation and never invents -heuristic features. When feature artifacts are missing or stale, it writes -`.tldrgraph/feature_workflows_request.yaml`. The coding agent that ran -`tldrgraph init` must delegate that request to a source-reading subagent. - -Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent -`needs_confirmation` response explicitly asks for approval. `--batch 200` means -all nodes in chunks; `--limit 200` means stop after only 200 nodes. -Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless the user -explicitly requests it. - -## Feature Workflow Explorer artifacts - -An accepted subagent response writes the v2 capability catalog in `.tldrgraph/features.yaml` -and flows in `.tldrgraph/workflows/.yaml`. The manifest groups concrete -capabilities into inferred product or technical areas; ranked symbols in the request are -investigation leads, not the feature list. -Workflow Explorer reads only those files; missing or invalid workflows show pending states. -Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. -Define features as user, admin, developer, or operator outcomes, not code symbols. Good: -"Natural-language project creation." Bad: "OpencodeService"; that service is evidence. -Each short plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. -Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. -Use `partial` with `missing_coverage` for a proven fragment and `pending` when no reliable -sequence can be drawn. List source-backed capabilities in either state without inventing steps. -Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. -When `init` reports `status: needs_feature_workflows`: - -1. Open `.tldrgraph/feature_workflows_request.yaml`. -2. Spawn a source-reading subagent and delegate the entire request to it. -3. Have the subagent inspect the whole repository and write areas, features, and generated/partial/pending workflows to `.tldrgraph/feature_workflows_response.yaml`. -4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. -5. Run `tldrgraph init` again. - -## `status: needs_layers` - -This should only appear when the user explicitly opted into architecture AI with -`--agent-cli` or an older TLDRGraph build is running. Do not complete this -handoff unless the user asked for architecture layer design. - -## `status: needs_confirmation` - -This belongs to explicit `--agent-cli` enrichment. Ask before continuing. +When the status is `needs_feature_workflows`: -## `status: needs_enrichment` - -This should only appear when the user explicitly opted into architecture -enrichment with `--agent-cli` or an older TLDRGraph build is running. Do not -process enrichment batches unless the user asked for full graph enrichment. - -**Copy every `id` verbatim.** A constructed id matches nothing, is dropped, and -gets reported back to you -- but the work is wasted. - -**Never invent `fields` or `calls`.** Omit what you cannot verify in the code: an -empty list is a correct answer, a wrong `calls` entry becomes a real wrong edge. - -## `status: needs_llm_links` - -Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend -files, then write `.tldrgraph/llm_links_response.yaml` as a YAML list of -`{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. -Only include source-backed links with file and line evidence. Run `tldrgraph init` -again. This continuation state only appears when the user explicitly opted into -route-link inference with `--llm-links`. - -## Once it says DONE - -```bash -tldrgraph query "" -tldrgraph trace "" "" -tldrgraph layers -tldrgraph ui --serve -``` +1. Read `.tldrgraph/feature_workflows_request.yaml` completely. +2. Spawn a source-reading subagent and delegate the entire request. +3. Have it inspect the repository and write the requested v3 response to + `.tldrgraph/feature_workflows_response.yaml`. +4. Run `tldrgraph init` again. -Read-only, and they never trigger enrichment. Full schema: -`.tldrgraph/AGENT_CONTRACT.md`. +Continue until the status is `done`. diff --git a/.cursor/commands/tldrgraph-init.md b/.cursor/commands/tldrgraph-init.md index 94f15de..57bd8e4 100644 --- a/.cursor/commands/tldrgraph-init.md +++ b/.cursor/commands/tldrgraph-init.md @@ -1,92 +1,22 @@ --- name: tldrgraph-init -description: Build or continue this repository's TLDRGraph architecture graph (layers, extraction, enrichment) +description: Build or refresh source-backed feature workflows --- -# TLDRGraph: build this repository's architecture graph +# TLDRGraph: build the source-backed workflow catalog -In Claude Code or Cursor, invoke `/tldrgraph-init`; in Codex CLI, select -`tldrgraph-init` from `/skills` or mention `$tldrgraph-init`. - -One command handles extraction, feature-workflow handoff, and embeddings: +Run exactly: ```bash tldrgraph init ``` -TLDRGraph never launches an AI process for feature generation and never invents -heuristic features. When feature artifacts are missing or stale, it writes -`.tldrgraph/feature_workflows_request.yaml`. The coding agent that ran -`tldrgraph init` must delegate that request to a source-reading subagent. - -Use exactly `tldrgraph init` for this workflow. Add `--yes` only if a non-agent -`needs_confirmation` response explicitly asks for approval. `--batch 200` means -all nodes in chunks; `--limit 200` means stop after only 200 nodes. -Never add `--limit`, `--agent-cli`, `--llm-links`, or `--embeddings off` unless the user -explicitly requests it. - -## Feature Workflow Explorer artifacts - -An accepted subagent response writes the v2 capability catalog in `.tldrgraph/features.yaml` -and flows in `.tldrgraph/workflows/.yaml`. The manifest groups concrete -capabilities into inferred product or technical areas; ranked symbols in the request are -investigation leads, not the feature list. -Workflow Explorer reads only those files; missing or invalid workflows show pending states. -Do not restore discovery through `discover_workflows()`, curated blueprints, route-link workflow discovery, route-link relations, or BPMN generation. -Define features as user, admin, developer, or operator outcomes, not code symbols. Good: -"Natural-language project creation." Bad: "OpencodeService"; that service is evidence. -Each short plain-language step needs source evidence. Start at the button/menu/form action and continue through client request, backend work, response payload, client handling, and final UI update when proven. -Do not set `status: generated` unless the saved steps cover the end-to-end flow for the proven feature boundary. -Use `partial` with `missing_coverage` for a proven fragment and `pending` when no reliable -sequence can be drawn. List source-backed capabilities in either state without inventing steps. -Do not use route-link relations as workflow evidence: `llm_http_route_link`, `http_route_link`, or `calls_endpoint`. -When `init` reports `status: needs_feature_workflows`: - -1. Open `.tldrgraph/feature_workflows_request.yaml`. -2. Spawn a source-reading subagent and delegate the entire request to it. -3. Have the subagent inspect the whole repository and write areas, features, and generated/partial/pending workflows to `.tldrgraph/feature_workflows_response.yaml`. -4. Do not edit `features.yaml` or `workflows/*.yaml` directly; TLDRGraph validates and applies the response. -5. Run `tldrgraph init` again. - -## `status: needs_layers` - -This should only appear when the user explicitly opted into architecture AI with -`--agent-cli` or an older TLDRGraph build is running. Do not complete this -handoff unless the user asked for architecture layer design. - -## `status: needs_confirmation` - -This belongs to explicit `--agent-cli` enrichment. Ask before continuing. +When the status is `needs_feature_workflows`: -## `status: needs_enrichment` - -This should only appear when the user explicitly opted into architecture -enrichment with `--agent-cli` or an older TLDRGraph build is running. Do not -process enrichment batches unless the user asked for full graph enrichment. - -**Copy every `id` verbatim.** A constructed id matches nothing, is dropped, and -gets reported back to you -- but the work is wasted. - -**Never invent `fields` or `calls`.** Omit what you cannot verify in the code: an -empty list is a correct answer, a wrong `calls` entry becomes a real wrong edge. - -## `status: needs_llm_links` - -Read `.tldrgraph/llm_links_request.yaml`, open the referenced frontend/backend -files, then write `.tldrgraph/llm_links_response.yaml` as a YAML list of -`{source, target, confidence, frontend_evidence, backend_evidence, explanation}`. -Only include source-backed links with file and line evidence. Run `tldrgraph init` -again. This continuation state only appears when the user explicitly opted into -route-link inference with `--llm-links`. - -## Once it says DONE - -```bash -tldrgraph query "" -tldrgraph trace "" "" -tldrgraph layers -tldrgraph ui --serve -``` +1. Read `.tldrgraph/feature_workflows_request.yaml` completely. +2. Spawn a source-reading subagent and delegate the entire request. +3. Have it inspect the repository and write the requested v3 response to + `.tldrgraph/feature_workflows_response.yaml`. +4. Run `tldrgraph init` again. -Read-only, and they never trigger enrichment. Full schema: -`.tldrgraph/AGENT_CONTRACT.md`. +Continue until the status is `done`. diff --git a/.gitignore b/.gitignore index cff9126..53961f3 100644 --- a/.gitignore +++ b/.gitignore @@ -85,9 +85,7 @@ Thumbs.db !.vscode/extensions.json # BEGIN TLDRGRAPH -# TLDRGraph analysis state. Generated artifacts are ignored; the agent -# contract and layer map are committed so the whole team shares them. +# TLDRGraph generated workflow state. .tldrgraph/* !.tldrgraph/AGENT_CONTRACT.md -!.tldrgraph/layers.config.yaml # END TLDRGRAPH diff --git a/.tldrgraph/AGENT_CONTRACT.md b/.tldrgraph/AGENT_CONTRACT.md index 2512fac..5ff660b 100644 --- a/.tldrgraph/AGENT_CONTRACT.md +++ b/.tldrgraph/AGENT_CONTRACT.md @@ -1,342 +1,55 @@ # TLDRGraph Agent Contract -**Audience: the coding agent with this repository open** (Codex, Claude Code, Cursor, Antigravity). +TLDRGraph builds a source-backed feature catalog and Workflow Explorer. It does +not construct an architecture graph and does not launch an AI process itself. -TLDRGraph builds an architectural graph from the graphify AST export. By default, -`tldrgraph init` does not ask AI to design architecture layers, enrich every -symbol, infer route links, or generate BPMN workflows. The only default AI-shaped -artifact is the saved Feature Workflow Explorer YAML, and it must stay backed by -real source evidence. +## Initialization handshake ---- +Run `tldrgraph init`. A missing or stale catalog produces: -## Start here: `tldrgraph init` - -One command handles extraction, saved feature workflows, and embeddings: - -```bash -tldrgraph init +```text +.tldrgraph/feature_workflows_request.yaml ``` -Use `--agent-cli` only when the user explicitly asks for AI-assisted architecture -layer design and all-symbol enrichment. Saved feature workflow generation is -independent and does not require `--agent-cli`. - -| status | what it wants | -| --- | --- | -| `needs_layers` | Only process if the user explicitly opted into architecture AI with `--agent-cli`. | -| `needs_confirmation` | Only belongs to explicit `--agent-cli` enrichment. Ask before continuing. | -| `needs_enrichment` | Only process if the user explicitly asked for full graph enrichment. | -| `needs_embeddings` | Enrichment finished but the required dense model/index could not be built. Fix model access and rerun init. | -| `done` | Nothing left. Use `query` / `trace` / `layers`. | - -`--json` gives you the same thing machine-readably. The sections below document the file -formats `init` reads and writes; the underlying `queue-enrichment` / `apply-enrichment` -commands remain available for scripting. - -**Copy every `id` verbatim from the request.** A constructed id matches nothing, is -dropped, and will be reported back to you — but the work is wasted. - ---- - -## Explicit Enrichment Loop - -The legacy enrichment files below are for explicit `--agent-cli`, -`queue-enrichment`, or `apply-enrichment` work. Do not process them as part of -default Feature Workflow Explorer generation. Never add `--limit`, -`--agent-cli`, `--llm-links`, or `--embeddings off` unless the user explicitly -requests that behavior. - -Request and response are **separate files**. Never write your answer back into -`enrichment_request.yaml`; it is regenerated on every run and your work would be lost. - -| File | Written by | Read by | -| --- | --- | --- | -| `.tldrgraph/enrichment_request.yaml` (or `enrichment_request.json`) | `queue-enrichment` | you | -| `.tldrgraph/enrichment_response.yaml` (or `enrichment_response.json`) | **you** | `apply-enrichment` | -| `.tldrgraph/enrichment_cursor.json` | both commands | both commands | -| `.tldrgraph/enrichment_approval.json` | `init --yes` | later `init` runs | -| `.tldrgraph/pending_enrichment.json` | *(legacy)* | `apply-enrichment`, only if no response file exists | - ---- - -## Feature Workflow Explorer artifacts - -`tldrgraph init` also creates the v2 capability catalog and saved flows used by the visualizer: - -| File | Written by | Read by | -| --- | --- | --- | -| `.tldrgraph/features.yaml` | `tldrgraph init` | Workflow Explorer | -| `.tldrgraph/workflows/.yaml` | `tldrgraph init` | Workflow Explorer | - -`features.yaml` contains ordered repository-specific areas (`product` before -`technical`) and concrete capabilities for user, admin, developer, or operator audiences. -Ranked symbols in the handoff request are investigation leads, not the feature list. -Good feature: **Natural-language project creation**. Bad feature: `OpencodeService`; -that service belongs in the capability's source evidence. - -The Workflow Explorer tab is intentionally file-driven. It must read only -`.tldrgraph/features.yaml` and `.tldrgraph/workflows/.yaml`; if those -files are missing, invalid, incomplete, or a feature has no generated workflow -yet, show an explicit empty or pending state. - -Do **not** reintroduce Workflow Explorer fallback discovery through -`discover_workflows()`, curated workflow blueprints, route-link workflow -discovery, `llm_http_route_link`, `http_route_link`, `calls_endpoint`, or -BPMN-derived workflow generation. Graph views elsewhere may still show route -links or BPMN data, but saved feature workflows must remain independent. +The coding agent must delegate that complete request to a source-reading +subagent. The subagent writes the requested +`tldrgraph/feature-workflows-response@3` document to: -Every saved workflow step must have a short title simple enough for non-technical users and vibe -coders, and each step must carry source evidence: `node_id`, symbol, file, and -line/range. If the evidence is absent, mark the workflow pending instead of -guessing. Use `status: partial` plus `missing_coverage` for a proven fragment and -`status: pending` when no reliable sequence can be drawn. Pending workflows contain no steps. - -A saved feature workflow should describe the complete flow when evidence exists: -the exact user button/menu/form action, event handler, validation, client -request code, request payload construction, API route/controller, -middleware/auth, service or use-case logic, persistence/database, background -job, external system, response payload creation, client response parsing, state -update, navigation/toast/rendered result, and visible success or error handling. -Do not stop at only the frontend or only the backend when the source proves the -handoff, and do not collapse multiple proven source hops into one vague step. - -The v2 response shape is: - -```yaml -schema: codechakra/feature-workflows-response@2 -graph_hash: "copy from request" -areas: - - id: ai_builder - title: AI application builder - summary: Create and revise applications with AI. - perspective: product - order: 0 -features: - - id: natural_language_project_creation - area_id: ai_builder - title: Natural-language project creation - audience: user - summary: Turn a prompt into a new application project. - evidence: [{node_id: "copy exact graph node id"}] - workflow: - status: generated # or partial / pending - summary: Create the project and show its result. - steps: [{number: 1, phase: user_action, title: Submit a prompt, - text: The user submits the project request., - evidence: [{node_id: "copy exact graph node id"}]}] -``` - -`partial` and `pending` require `missing_coverage`; `pending` requires an empty -`steps` list. Allowed audiences are `user`, `admin`, `developer`, and `operator`. - ---- - -## Request schema (`enrichment_request.yaml`) - -```yaml -schema: codechakra/enrichment-request@1 -generated_at: "2026-08-19T00:00:00+00:00" -response_file: .tldrgraph/enrichment_response.yaml -contract: .tldrgraph/AGENT_CONTRACT.md -progress: - total_candidates: 1873 # un-enriched, non-utility nodes - already_enriched: 12 # nodes that already carry an intent - queued_now: 200 # entries in "nodes" below - remaining_after: 1673 # still waiting after this batch is applied -nodes: - - id: backend_src_applications_applications_controller_applicationscontroller - label: ApplicationsController - layer_id: api - layer: "Layer 2: API Gateway" - file: backend/src/applications/applications.controller.ts - source_location: L31 - degree: 41 # in + out edges in the AST graph - cross_layer_degree: 17 # of those, how many cross a layer boundary - rank: 1 # 1 = highest priority in this batch - existing_intent_source: heuristic # "" when the node has no intent at all +```text +.tldrgraph/feature_workflows_response.yaml ``` -`file` is repo-relative. `source_location` is graphify's line hint and may be `null`. -`layer_id` is the stable machine key (e.g. `cli`, `engine`, `storage`, `api`, `ui`). - -`existing_intent_source` is `"heuristic"` when the node already carries an intent written -by the offline template enricher. That text was generated from the label and layer alone -— it has not read a line of source — so the node is still a candidate and your answer -should overwrite it. Applied answers are stamped `"agent"` and are never re-queued. - ---- +Run `tldrgraph init` again. TLDRGraph validates the response against the current +source inventory and atomically writes `.tldrgraph/features.yaml` plus one file +per capability under `.tldrgraph/workflows/`. -## Response schema (`enrichment_response.yaml` or `enrichment_response.json`) +## Evidence -A **YAML list** (preferred) or **JSON array** of objects: +Every feature and every displayed workflow step must have at least one evidence +record: ```yaml -- id: backend_src_applications_applications_controller_applicationscontroller - intent: | - ### Pension Application Lifecycle Gateway - REST gateway for the pension application lifecycle. Authorizes DEO/AAO/AO/DAG roles, - dispatches cases to ApplicationsService and records status transitions. - input_fields: - - caseId - - transitionPayload - - remarks - - sanctionOrderNo - output_fields: - - applicationStatus - - disposition - calls: - - ApplicationsService - - JwtAuthGuard - - RolesGuard - - pension_cases +file: relative/path/to/source.py +symbol: verified_symbol +line: 12 +code_start: 12 +code_end: 28 ``` -Equivalent JSON format (also accepted from `.tldrgraph/enrichment_response.json` or `.tldrgraph/pending_enrichment.json`): -```json -[ - { - "id": "backend_src_applications_applications_controller_applicationscontroller", - "intent": "### Pension Application Lifecycle Gateway\nREST gateway for the pension application lifecycle. It authorizes roles and dispatches source-backed status transitions.", - "input_fields": ["caseId", "transitionPayload", "remarks", "sanctionOrderNo"], - "output_fields": ["applicationStatus", "disposition"], - "calls": ["ApplicationsService", "JwtAuthGuard", "RolesGuard", "pension_cases"] - } -] -``` - -| Key | Type | Meaning | -| --- | --- | --- | -| `id` | string, **required** | The node id, copied **verbatim** from the request. An id that is not in the graph is skipped silently. | -| `intent` | string (Markdown) | Markdown formatted 2-3 sentence explanation: what this symbol does, why it exists, and its source-backed behavior. Headings and list markers do not count as sentences. This is the text semantic search matches against. | -| `input_fields` | array of strings | Input parameters, arguments, request body payload attributes, query filters. | -| `output_fields` | array of strings | Return types, response models, emitted event names, or mutated state attributes. | -| `fields` | array of strings (legacy) | Supported for backwards compatibility (maps to input fields). | -| `calls` | array of strings or objects | Downstream symbols, files (`file:symbol`), or node IDs this symbol calls. Cross-layer bridges are created with 100% confidence. | -| `layer_id` | string (optional) | Explicitly reassign the architectural layer ID if the AST classification miscategorized it. | - -`input_fields`, `output_fields`, and `calls` may be omitted or empty. An object with only `id` and `intent` is -valid and useful. - ---- - -## Hard rules - -1. **Open and read the actual source file before writing an intent.** You have the repo - checked out; that is the entire reason this path exists. Read `file` (use - `source_location` to find the symbol), and read enough of its imports and callees to - describe what it really does. An intent paraphrased from the label is worse than no - intent, because it poisons search with confident-sounding noise. - -2. **Write every intent in 2-3 complete sentences.** Cover what the symbol does, why it - exists, and its source-backed behavior. Markdown headings and list markers do not count - as sentences. - -3. **Do not invent fields or calls. Omit what you cannot verify in the code.** If you - read the file and it handles three params, list three. Do not pad the list with what a - symbol of that name "usually" has. `"fields": []` is a correct, honest answer. - A wrong `calls` entry creates a real, wrong edge in the graph that later queries will - follow. - -4. **`calls` entries are resolved with 2-tier high precision.** - - **Tier 1 (Exact Match, 100% confidence):** Exact symbol names (`ApplicationsService`), - function names, node IDs, file paths (`calc.ts`), or database table names (`pension_cases`). - - **Tier 2 (Vector Fallback):** Semantic search with a calibrated 0.35 score floor. - - | Good | Bad | - | --- | --- | - | `ApplicationsService` | `the application service` | - | `calc.ts` | `some calculation helper` | - | `pension_cases` | `the database` | - | `JwtAuthGuard` | `auth stuff` | - - Prefer the exact symbol name, file name, or table/model name as it appears in the source. - -5. **Copy `id` verbatim.** Do not normalize, shorten or re-case it. - -6. **Answer only the nodes in the request.** Extra ids are ignored; missing ids just come - back in a later batch. - -7. **After full approval, never ask again for the same campaign.** Continue processing - `needs_enrichment` batches until `status: done`. Do not silently add `--limit` or - `--embeddings off`. - ---- - -## Priority order in the queue - -The queue is not arbitrary — a node that many things depend on is worth more of your -attention than a leaf. A node is a **candidate** when it sits outside `General / Utility` -and either has no intent at all, or has one that came from the offline template heuristic -(`enrichment_source: "heuristic"`, i.e. nobody read the source). Candidates are sorted by: - -1. **`cross_layer_degree` descending** — neighbours that sit in a *different* layer. - These are the seams TLDRGraph exists to describe, and they are exactly where the AST - alone is weakest. -2. **`degree` descending** — total in + out edges. Hub nodes first. -3. **node id ascending** — only to make the ordering deterministic. - -Both degrees are computed from the live graph. (The `degree` key that graphify emits is -absent, so anything reading `node["degree"]` from the raw export sees `0`; TLDRGraph -recomputes it and stamps it back into `.tldrgraph/graph.json`.) - ---- - -## Paging and progress - -`queue-enrichment` remembers what it has handed out in `.tldrgraph/enrichment_cursor.json`: - -- `applied` — ids successfully merged by `apply-enrichment`. Never re-queued. -- `queued` — ids handed out but not yet applied ("in flight"). Skipped by default. - -So running `queue-enrichment` twice in a row **advances** to the next batch instead of -repeating. Two escape hatches: - -- `--requeue` — also hand out in-flight ids again (use when a batch was abandoned). -- `--reset` — clear all progress and start again from the highest-priority node. -- `--limit 0` — no cap; queue every remaining candidate at once. - ---- - -## What `apply-enrichment` does with your answer - -For each object it can match to a node: - -1. sets `intent`, rewrites `summary` to `":