Skip to content

feat: add lazy thread sync #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
galligan merged 2 commits into from
Jun 5, 2026
Merged

feat: add lazy thread sync #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ccc1b5e
docs: plan lazy thread sync
galligan Jun 5, 2026
a9514a5
feat: add lazy thread sync
galligan Jun 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .agents/plans/lazy-thread-sync/GOAL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Lazy Thread Sync — goal prompt

```text
/goal From `/Users/mg/Developer/outfitter/dispatch`, implement lazy sync for existing Codex threads so pickup is instant and honest.

Read first: `AGENTS.md`, `.agents/plans/PLANNING.md`, `.agents/plans/lazy-thread-sync/PLAN.md`, `.agents/plans/lazy-thread-sync/REFS.md`, `.agents/plans/lazy-thread-sync/RETRO.md`, `docs/adrs/0005-lane-authority-capability-ladder.md`, `docs/adrs/0011-codex-session-registration-is-explicit.md`, and current attach/discover/sync-adjacent code/tests. Verify live branch/state before editing.

Objective: ship metadata-only `dispatch lane attach <thread-id>` plus explicit progressive `dispatch lane sync <lane>` using compact App Server metadata and bounded Codex JSONL top+tail indexing. Update registry state, CLI/MCP/schema projections, docs, skills, and tests. Preserve observe-only attached-lane authority.

Constraints: work on `feat/lazy-thread-sync` or create it from main if missing. Follow contract-first/no-drift patterns; add ops through the registry, not hand-written surface forks. Do not unlock attached-lane writes, copy whole transcripts by default, index all unattached threads by default, merge, publish, mutate releases, commit secrets, or disturb live Codex threads. If live Codex smoke is useful, use temp `DISPATCH_HOME`, read-only metadata/sync only, no sends/stops/renames/archives.

Loop: define success before each slice; make small reversible changes; run focused tests first; update `RETRO.md`; do local review; fix P0/P1/P2; repeat until the feature is correct and you would ship it. Use bounded subagents for parser/schema/docs/review scouting, but verify their claims and keep synthesis, edits, commits, PRs, and tracker/source-control writes centralized.

Verification: parser/index fixture tests; attach tests proving default attach does not call `thread/resume`; registry migration tests; CLI/schema/MCP parity tests for `lane sync`; docs/skills checks; safe local runtime smoke; final `just check`.

Source control: commit coherent slices with conventional messages. Submit draft PR(s) only after local checks/review are clean. Do not mark ready, merge, or publish without explicit user approval.

Stop if unsafe ambiguity, missing credentials, unauthorized external side effects, impossible verification, or the same blocker repeats 3 times.

Done only when metadata-only attach, explicit sync, sync state surfaces, docs/skills/schema parity, tests, safe local smoke, green `just check`, and local review with no unresolved P0/P1/P2 are all complete; `RETRO.md` must record changed files, checks, review rounds, risks, forbidden-action audit, PR/branch state, and exact proof the completion condition is satisfied or blocked.
```

189 changes: 189 additions & 0 deletions .agents/plans/lazy-thread-sync/PLAN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
# Lazy Thread Sync — implementation plan

Goal packet for making existing Codex thread pickup instant without eager
`thread/resume` or transcript copying. Pasteable goal: [`GOAL.md`](./GOAL.md).
Execution ledger: [`RETRO.md`](./RETRO.md). References: [`REFS.md`](./REFS.md).

## Objective

Ship progressive sync for attached Codex lanes:

- `dispatch lane attach <thread-id>` registers existing Codex threads quickly
from compact metadata, without `thread/resume` by default.
- `dispatch lane sync <lane>` builds or refreshes dispatch's local indexed view
with bounded top+tail parsing of Codex JSONL artifacts.
- `lane get`, `lane list`, and unmanaged discovery expose honest sync state.
- The daemon can keep attached lanes warm without whole-home indexing.
- CLI, MCP, schemas, docs, and first-party skills stay derived/aligned.

## Current Evidence

The local investigation found the right split of responsibilities:

- `thread/list(useStateDbOnly:true)` is the official cheap discovery path.
- `thread/read(includeTurns:false)` verifies a thread id and returns compact
official metadata in a few milliseconds.
- `thread/resume` is too heavy and side-effectful for default attach: on the
sampled thread it returned about 87 KB, emitted notifications, loaded the
thread, and took 1-3 seconds.
- Local JSONL top+tail parsing is cheap enough for progressive sync, but must
cap bytes as well as lines because early records can be large.
- `Thread.path` is explicitly unstable in the current App Server schema, so
dispatch should treat it as a cached source pointer with file identity, not a
durable id.

The packet summarizes the local notes in [`REFS.md`](./REFS.md) so execution
does not depend on chat history.

## Product Contract

Expected operator shape:

```bash
dispatch lane list --unmanaged --limit 20
dispatch lane attach <thread-id>
dispatch lane attach <thread-id> --sync
dispatch lane sync <lane>
dispatch lane sync <lane> --full
dispatch lane get <lane>
dispatch lane list
dispatch schema "lane sync"
```

Semantics:

- `attach` defaults to metadata-only and uses `thread/read(includeTurns:false)`.
- `attach --sync` performs a quick sync after registration.
- `sync` means progressive indexing, not hydration and not transcript copying.
- `sync --full` may backfill more aggressively if implemented, but should still
be bounded and explicit; do not block the main product slice on perfect full
archival indexing.
- Existing attached lanes remain observe-only. Do not unlock send/stop/archive
or history writes for attached desktop lanes.
- `lane tail` may keep using official `thread/read(includeTurns:true)` for
persisted turn summaries, but should not be described as a fast tail API.

## Implementation Slices

### Slice 1 — Codex source adapters

- Add a focused module for Codex thread metadata and JSONL source parsing.
- Wrap official App Server metadata reads/lists instead of making handlers
inspect raw schema blobs.
- Add a JSONL top+tail parser with:
- line and byte caps;
- complete-line handling;
- partial-final-line tolerance;
- file identity capture: path, device, inode, size, mtime;
- sanitized metadata extraction, not raw transcript dumping.
- Add fixture tests for large first records, tail offsets, partial final lines,
missing/moved files, and path identity changes.

### Slice 2 — Registry sync index

- Bump the registry schema and add the smallest useful sync tables.
- Start with source/snapshot facts before adding item-level history.
- Candidate fields:
- thread id, source path, device, inode, size, mtime;
- sync state: `metadata`, `partial`, `complete`, `error`;
- last synced at, latest event at, latest turn id, error;
- display name, preview/excerpt policy, cwd, source, model/provider, session id;
- top cursor/tail cursor/backfill cursor.
- Add migration/open tests, model validation tests, and older-db/newer-db checks.

### Slice 3 — Metadata-only attach and explicit sync op

- Change `attach` to verify with `thread/read(includeTurns:false)`, register the
lane, store metadata sync state, and never call `thread/resume` by default.
- Add `sync` as a first-class op projected to CLI/MCP/schema.
- Route `dispatch lane sync` through the op registry; do not hand-write a
special surface.
- Make idempotency and error behavior explicit:
- missing thread -> clean App Server/not-found style error;
- missing source file -> registered lane can still exist with sync error;
- repeated sync -> updates snapshot/cursors.
- Update tests proving attach no longer resumes.

### Slice 4 — Read surfaces and daemon warming

- Include sync state in `lane get` and `lane list`.
- Keep `lane list --unmanaged` official and cheap, using `thread/list` state DB.
- Add bounded daemon background sync for attached lanes if it stays simple:
- quick sync after attach when configured;
- poll/watch attached source file size/mtime;
- append-only parsing from stored cursor;
- conservative defaults.
- Add config only where needed:

```toml
[sync]
quick_on_attach = true
watch_attached = true
unattached = "off" # off | cwd | recent | all
tail_lines = 200
tail_bytes = 262144
top_bytes = 262144
max_backfill_bytes_per_tick = 1048576
```

Do not implement broad automatic unattached indexing unless the slice remains
small and private by default.

### Slice 5 — Docs, skills, schemas, and local proving

- Update README/docs/usage/skills/plugin docs for the new attach/sync semantics.
- Update ADRs or add one if a durable decision changed:
- likely: progressive sync/index cache and metadata-only attach.
- Ensure `dispatch schema "lane sync"` and MCP grouped tools expose the new op.
- Run focused tests first, then `just check`.
- Put the feature through local paces:
- isolated registry/`DISPATCH_HOME`;
- fixture JSONL sync;
- real App Server metadata read if available;
- optional real existing Codex thread attach/sync smoke in a temp dispatch
home, read-only, no sends/stops/renames/archives.

## Review Loop

Use local review between meaningful slices:

- request/perform review with score out of 5;
- P0/P1/P2 block completion;
- P3 can be fixed if cheap or recorded in `RETRO.md`;
- update `RETRO.md` after each slice, check run, review round, and material
decision.

Subagents are useful for:

- fixture/parser audit;
- App Server schema/current-doc check;
- docs/skill drift review;
- final code quality review.

Subagents must not commit, push, mutate PRs, merge, publish, or touch live user
agent state.

## Source Control

- Work from branch `feat/lazy-thread-sync`.
- Use Graphite if submitting PRs.
- Keep commits coherent. One commit is fine if the final diff stays reviewable;
split only when it genuinely helps review.
- PRs stay draft until local checks and local review are clean.
- Do not merge, publish, or alter release state without explicit user approval.

## Done

Done only when:

- metadata-only attach, `lane sync`, sync index, read surfaces, and docs/skills
are implemented or explicitly pared down with evidence;
- the code path no longer relies on eager `thread/resume` for default attach;
- schema/MCP/CLI parity tests pass;
- parser/index fixtures cover the failure modes above;
- local runtime smoke demonstrates attach/sync behavior safely;
- `just check` is green;
- local review has no unresolved P0/P1/P2;
- `RETRO.md` records final state, checks, review result, remaining risks, and
any deferred follow-up issues.

104 changes: 104 additions & 0 deletions .agents/plans/lazy-thread-sync/REFS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Lazy Thread Sync — references

## Local Investigation Summary

The local investigation note is gitignored at
`.agents/notes/2026-06-05-codex-storage-investigation.md`. This file captures
the load-bearing findings so the goal packet does not depend on chat history or
ignored notes.

Evidence gathered on 2026-06-05:

- Codex CLI: `codex-cli 0.137.0-alpha.4`.
- dispatch CLI: `dispatch 0.2.1`.
- Sample thread id: `019e9598-9214-7ed1-ac40-52d6d675d3e7`.
- Sample JSONL artifact size: about 1.49 MB, 812 records.
- `~/.codex` symlinked to `/Users/mg/.config/codex`.

Observed stores:

- `session_index.jsonl`: display thread names and updated timestamps.
- `state_5.sqlite`: thread metadata, rollout path, cwd, title/preview,
model/provider, archive state, dynamic tools, spawn edges, agent jobs.
- `goals_1.sqlite`: native goals.
- `logs_2.sqlite`: logs/telemetry, not transcript structure.
- `sessions/YYYY/MM/DD/rollout-...<thread-id>.jsonl`: append-oriented JSONL.
- `shell_snapshots/<thread-id>.<timestamp>.sh`: shell snapshots.

Important mismatch:

- `state_5.sqlite.threads.title` and `preview` can be the raw opening prompt.
- `session_index.jsonl.thread_name` and App Server `Thread.name` are better
user-facing display-name sources.

Schema findings from `codex app-server generate-json-schema --out <tmp>`:

- `thread/list` supports `useStateDbOnly`, pagination, cwd/search/source/model
filters, and returns rows under `result.data`.
- `thread/read` only accepts `threadId` and `includeTurns`.
- `thread/resume` does not currently expose a usable turn-page parameter.
- `Thread.path` is marked `[UNSTABLE]`; store path plus file identity as a
pointer, not the durable id.
- `Thread.turns` is populated only for resume/fork/rollback/read with
`includeTurns:true`.

Timing on the sample thread:

- `thread/list(useStateDbOnly:true, limit=10)`: about 10 ms, about 33.5 KB.
- `thread/list(useStateDbOnly:false, limit=10)`: about 133 ms.
- `thread/read(includeTurns:false)`: about 3 ms, about 4 KB.
- `thread/read(includeTurns:true)`: about 10-15 ms, about 86 KB.
- `thread/resume`: about 1.1-3.0 seconds, about 87 KB, loaded the thread and
emitted notifications.
- local first 8 JSONL records: about 0.46 ms, about 114 KB.
- local tail 256 KiB window: about 0.88 ms, 126 complete records.
- local full 1.49 MB parse: about 5 ms.

JSONL shape on the sample:

- top-level types: `response_item`, `event_msg`, `session_meta`, `turn_context`.
- useful payload kinds: `thread_goal_updated`, `function_call`,
`function_call_output`, `token_count`, `message`, `agent_message`,
`reasoning`, `custom_tool_call`, `custom_tool_call_output`,
`patch_apply_end`, `task_complete`, `user_message`.
- first `session_meta` starts at byte 0 and includes thread id, cwd, source,
thread source, model provider, cli version, dynamic tools, and git keys.
- `turn_context` includes model, effort, approval policy, sandbox policy, cwd.
- `task_complete` includes duration, turn id, completion timestamp, and final
message length.

## Repo References

- `AGENTS.md`
- `.agents/plans/PLANNING.md`
- `docs/development/design.md`
- `docs/adrs/0005-lane-authority-capability-ladder.md`
- `docs/adrs/0011-codex-session-registration-is-explicit.md`
- `docs/adrs/0016-history-goals-and-bounded-watch.md`
- `docs/usage/README.md`
- `skills/dispatch/SKILL.md`
- `skills/dm/SKILL.md`
- `src/outfitter/dispatch/core/handlers.py`
- `src/outfitter/dispatch/core/models.py`
- `src/outfitter/dispatch/core/ops.py`
- `src/outfitter/dispatch/client/client.py`
- `src/outfitter/dispatch/client/models.py`
- `src/outfitter/dispatch/registry/store.py`
- `src/outfitter/dispatch/contracts/derive_cli.py`
- `src/outfitter/dispatch/contracts/derive_mcp.py`
- `tests/core/test_handlers.py`
- `tests/client/test_client.py`
- `tests/surfaces/test_parity.py`
- `tests/surfaces/test_mcp_routing.py`

## Follow-Up Unknowns

- Active-write behavior: verify JSONL append behavior while a thread is running.
- Rename propagation: verify display-name update order across App Server,
`session_index.jsonl`, state DB, and JSONL.
- Archived threads: verify `thread/list(archived:true)` behavior.
- Subagent source projection: state DB source can be JSON-shaped strings.
- Rotation/rewrite: detect file shrink, inode change, move, or compact.
- Privacy policy: decide whether excerpts are stored by default, capped, or
opt-in.

Loading