feat: adf-ctl direct dispatch (#1875), FffIndexer migration (#1873), local .terraphim config (#1862)

[AlexMikhalev]

Summary

Multi-feature branch consolidating three related improvements:

1. adf-ctl direct dispatch (#1875)

• Unix domain socket listener for low-latency local agent dispatch
• adf-ctl --local trigger --direct bypasses HTTP webhook + HMAC
• Socket permissions 0600, bounded reads (8 KiB), agent name validation
• Separate LoopEvent::DirectDispatch variant (no mention config required)
• #[cfg(unix)] gated for cross-platform compilation

2. FffIndexer migration (#1873)

• Replaces RipgrepIndexer with pure-Rust fff-search middleware
• KG scorer helper for TerraphimGraph relevance function
• 722-line integration test suite

3. Local .terraphim config (#1862)

• ProjectConfig::load_from_dir() for .terraphim/ directory scanning
• Role file discovery (role-*.json), thesaurus/KG path helpers
• MCP server and terraphim-agent integration

Housekeeping

• Cargo.toml metadata (description, readme, homepage) for ~20 crates
• .terraphim/learnings/ added to .gitignore (auto-generated artefacts)

Test plan

• cargo test -p terraphim_orchestrator --lib direct_dispatch -- 13 tests pass
• cargo test -p terraphim_orchestrator --bin adf-ctl -- 26 tests pass
• cargo check -p terraphim_orchestrator --target x86_64-pc-windows-gnu -- cross-compile passes
• cargo clippy -p terraphim_orchestrator -- zero warnings
• cargo test -p terraphim_middleware --test fff_indexer -- FffIndexer tests
• cargo test -p terraphim_config -- project config tests

Generated with Terraphim AI

[github-actions]

Documentation Preview

Your documentation changes have been deployed to:
https://54436606.terraphim-docs.pages.dev

This preview will be available until the PR is closed.

[AlexMikhalev]

Summary

This PR is a large consolidation branch that lands three related but distinct features at once:

• adf-ctl direct dispatch (#1875): A new Unix domain socket path (adf-ctl --local trigger --direct) that bypasses the HTTP webhook + HMAC roundtrip for low-latency local agent spawning on the same machine.
• FffIndexer migration (#1873): Replacement of the external ripgrep-based RipgrepIndexer with a pure-Rust fff-search implementation (FffIndexer) in terraphim_middleware, including optional KG path scoring and frecency support.
• Local .terraphim config (#1862): New ProjectConfig discovery logic for .terraphim/ directories (role files, thesauri, KG paths) plus related MCP/terraphim-agent integration.

What was done well

• Direct dispatch correctly uses 0600 socket permissions, bounded reads (8 KiB), and agent name allow-list validation before emitting WebhookDispatch::SpawnAgent.
• FffIndexer is a clean, extensible pure-Rust replacement with proper builder-pattern extension points (with_kg_scorer) and caching discipline.
• Excellent test coverage in the new direct_dispatch module (including adversarial oversized input and unknown-agent cases).
• Significant investment in design/research documentation.

What remains structurally risky

• This is a 65-file, ~7.7k LOC net change that bundles three non-trivial features. The risk of one feature masking problems in another is high.
• The new direct-dispatch path hardcodes issue_number: 0 and comment_id: 0. This creates a second, structurally different code path that downstream consumers of WebhookDispatch may not handle correctly.
• The PR deletes a large number of .terraphim/learnings/ artefacts while simultaneously introducing new local config discovery — the interaction between "generated artefacts" and the new persistence/discovery model is under-specified.
• FffIndexer integration tests are not yet green according to the PR checklist.

Confidence Score: 2/5

Address issues before merging.

Zero P0s identified so far, but multiple P1-class structural and integration risks exist across the three features (particularly the dual dispatch paths and the FffIndexer migration). The combination of a new privileged local dispatch mechanism + a search engine replacement + new configuration discovery in one PR makes the blast radius too large for safe landing without further targeted review and test evidence.

Files requiring special attention:

• crates/terraphim_orchestrator/src/direct_dispatch.rs
• crates/terraphim_orchestrator/src/bin/adf-ctl.rs
• crates/terraphim_middleware/src/indexer/fff.rs
• crates/terraphim_orchestrator/src/lib.rs (wiring + WebhookDispatch handling)
• All consumers of WebhookDispatch::SpawnAgent

Important Files Changed

Filename	Overview
crates/terraphim_orchestrator/src/direct_dispatch.rs (new)	Core Unix socket listener. Good local security posture but introduces a second dispatch path with impoverished context (issue/comment IDs = 0). Needs cross-check against all WebhookDispatch handlers.
crates/terraphim_orchestrator/src/bin/adf-ctl.rs	Large expansion of the CLI to support --local + --direct. Primary entry point for the new fast path.
crates/terraphim_middleware/src/indexer/fff.rs (new)	Pure-Rust replacement for RipgrepIndexer. Well-structured but the migration impact on relevance (TerraphimGraph + KG scoring) is not yet fully validated in the PR.
crates/terraphim_orchestrator/src/lib.rs	Orchestrator wiring changes. Shows how the new direct dispatch listener is started and how WebhookDispatch is extended.
crates/terraphim_orchestrator/src/config.rs + project_adf.rs	New local .terraphim/ discovery logic.
Many .docs/design-*.md and .docs/research-*.md files	Large volume of design artefacts. Useful for context but increase review surface area.
.terraphim/learnings/* (many deletions)	Mass deletion of generated learning files. Needs explicit justification tied to the new local config feature.

Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    autonumber
    participant CLI as adf-ctl
    participant Socket as Unix Socket (0600)
    participant Listener as direct_dispatch_listener
    participant Orchestrator as AgentOrchestrator
    participant Webhook as Existing Webhook Path

    alt Direct (local, fast path)
        CLI->>Socket: {"agent": "X", "context": "..."}
        Socket->>Listener: accept + bounded read (8KiB)
        Listener->>Listener: validate agent against HashSet
        Listener->>Orchestrator: WebhookDispatch::SpawnAgent { issue_number:0, comment_id:0, ... }
        Orchestrator-->>CLI: {"status":"ok"}
    else Traditional (remote)
        Webhook->>Orchestrator: WebhookDispatch::SpawnAgent { real issue/comment ids, HMAC verified }
    end

    Note over Orchestrator: Both paths converge on the same SpawnAgent handling.<br/>Downstream code must tolerate zero IDs.

Loading

Inline Findings

P1 crates/terraphim_orchestrator/src/direct_dispatch.rs:140-150 + lib.rs (multiple call sites): Direct dispatch emits WebhookDispatch::SpawnAgent with synthetic zero IDs

The new fast path constructs:

WebhookDispatch::SpawnAgent {
    agent_name: cmd.agent,
    detected_project: None,
    issue_number: 0,
    comment_id: 0,
    context: ...
}

Any code that branches on issue_number > 0 / comment_id > 0, uses these for Gitea API calls, deduplication, or audit logging will behave differently (or fail) on the direct path versus the webhook path. This is a structural API contract violation introduced by the new dispatch mechanism.

P1 crates/terraphim_middleware/src/indexer/fff.rs + integration in search path: FffIndexer migration lacks demonstrated parity on TerraphimGraph relevance

The PR replaces the production indexer but the checklist shows the dedicated FffIndexer test (cargo test -p terraphim_middleware --test fff_indexer) is still pending. Given that TerraphimGraph relevance depends heavily on document indexing quality and the new indexer adds KG path scoring + frecency as optional layers, there is a material risk of regression in result quality for roles that use the graph scorer.

P2 crates/terraphim_orchestrator/src/direct_dispatch.rs + adf-ctl.rs: New local dispatch path has no rate limiting or backpressure visible at the socket layer

While the orchestrator has rate limiting elsewhere, the Unix socket listener accepts connections and immediately tries to send on dispatch_tx with no visible bounding or shedding. Under load from multiple adf-ctl invocations this could exert backpressure on the main orchestrator loop.

P2 Mass deletion of .terraphim/learnings/ files in the same PR as new local .terraphim/ config discovery

The deletions are presented as "housekeeping", but they coincide with the introduction of ProjectConfig::load_from_dir(). If any of the deleted learnings were intended to be user- or agent-managed artefacts under the new local config scheme, this is a data-loss surface. The interaction is not clearly documented in the PR description or the new design docs.

Comments Outside Diff

None at this time. All material findings are in the changed files.

_{Last reviewed commit: pr-888-review | Reviews (1)}