Add an opt-in LLM review mode with SSOT diffs and separate review comments #18

Goal

Introduce a separate interactive review mode in which an LLM creates several plots, a person edits and comments on them, and the LLM updates the source code before presenting the next complete set.

Existing plotting and manual editing workflows remain available unchanged. Automated review uses an explicit session-oriented API rather than changing the meaning of ordinary show(), run_events() or Copy Python.

Review workflow

1. A runner executes plotting code and registers the resulting Figures in a review round.
2. The reviewer adjusts chart layout/styles through the existing editor and enters natural-language feedback in a dedicated review-comment input.
3. Approval of an individual Figure flushes pending edits and durably checkpoints its baseline, approved SSOT, exact diff, separate review comments and binary data references. Close the approved window only after the complete checkpoint is published; failure leaves it available for retry. Approval alone does not start LLM source modification.
4. Once every required Figure has an explicit disposition, deliver the complete approved batch to the LLM. Coordinate source changes in one staging workspace, with idempotent work IDs and stale-round checks.
5. Validate the updated source and the complete candidate set before displaying the next set together. Retain the previous complete checkpoint until the replacement is valid.
6. Repeat until the set is accepted. Support retry, cancellation and recovery without losing the previous reviewable set.

Approval with requested changes submits feedback; acceptance without changes is a distinct disposition. Closing a window must not silently count as approval.

Feedback contract

Keep two separate, machine-readable channels:

• SSOT diff: exact committed human edits relative to an identified baseline chart document.
• Review comments: separately entered natural-language requests, optionally targeted at a stable Figure/panel/layer/series/annotation identity.

Chart annotations that belong in the rendered output remain part of chart SSOT. Review comments belong to session metadata and are not inserted into chart images or settings automatically.

The versioned feedback envelope should include session/round/submission IDs, stable object IDs, baseline document revision/hash, model schema version and the approved revision. Define deterministic patch semantics, explicit additions/removals and stale-base preconditions. Select a documented patch representation, such as JSON Patch with identity resolution, or an equivalent typed patch with before/after values.

Exclude transient hover/selection, GPU handles and OS window placement from persistent chart diffs. Document the distinction between document layout edits and arranging separate desktop windows.

An LLM can consume the SSOT diff directly to understand exact edits. Provide the baseline snapshot or a resolvable reference when required; screenshots and Copy Python output are supplementary context. A diff of visual settings alone does not capture source-data computation.

Architecture to define before implementation

• Separate API: introduce an explicit review namespace/session lifecycle with registration, event retrieval, submission, completion, cancellation and round publication. Final names should follow an API design review. Ordinary plotting must not start an LLM session or acquire an LLM runtime dependency.
• Stable document addressing: share Figure/Axes/layer/series/annotation identities and validated mutations with the feature and editor work. Translate positional renderer addresses without exposing unstable indices as persistent identities.
• Source provenance and replay: associate Figures with a runnable entry point, source revision/digest, execution arguments and required input-data references. Define reproducibility requirements for generated/random data. Preserve existing GPU-resident data ownership rather than adding unconditional CPU array copies.
• Change reconciliation: carry exact approved edits into executable source or a reproducible configuration artifact. Retain them in subsequent rounds unless explicitly superseded; detect conflicting source edits or stale submissions rather than blindly applying patches.
• Concurrency and threading: keep window events on the host's main thread; perform LLM work and reruns outside the UI loop through a documented event/IPC boundary. Serialize source writes and track which submissions each candidate build contains.
• Round lifecycle: define the review set at round creation, track Figure-level dispositions across multiple views, invalidate approval after further edits and reject results for obsolete round/source revisions.
• Publication and recovery: prepare and validate the complete candidate set before publishing the next round. Approved windows may already be closed; preserve their complete checkpoints and data on agent disconnect or execution/render failure, and expose recovery, retry and cancellation.
• LLM adapter: provide an explicit tool/event interface for supported agent hosts. Define how feedback reaches the agent and how code/results return; a UI button alone does not establish that connection. Document local-window execution and remote-agent connection requirements.
• Integration skill distribution: evaluate shipping a versioned companion skill with installation/discovery guidance, compatible API/schema versions, session lifecycle instructions and runnable examples. The skill should teach agents how to consume SSOT diffs and separate comments, update source, validate results and publish a round through the adapter. Keep the protocol independently usable by other agents and direct tooling.

Durable binary checkpoints

This section incorporates the agreed recovery design. Review is turn-based; individual approval never starts source modification while the remainder of the batch is under review.

• Create binary data checkpoints only in the opt-in review lifecycle, while source buffers are available. Preserve every referenced column, matrix and bounded-family value with dtype, shape and stable identity. Reuse immutable payloads across settings-only revisions; do not add unconditional native CPU copies or full GPU readback.
• Publish a versioned manifest containing session/batch/document/turn and work IDs, source entry point/revision and inputs, baseline/approved revisions and hashes, settings, exact diff, separately entered comments, and binary file references with integrity metadata.
• Write and flush payloads before atomically publishing the manifest. A partial new revision cannot replace the latest complete recoverable checkpoint.
• Signal the agent with the work ID and manifest location. The agent consumes structured feedback; raw binary arrays stay outside its text context. Recover pending work by scanning durable manifests after reconnect/restart, and make duplicate delivery idempotent.
• Application code owns automatic cleanup. A complete higher revision of the same document lineage reclaims superseded checkpoints and unreferenced payloads. Repeat maintenance on startup and cache access; do not depend on LLM cleanup commands.
• Retain the latest complete checkpoint, all data it references, and baselines/approved feedback needed by unfinished work. Each retained checkpoint must remain interpretable without deleted deltas. Coordinate active readers/writers and cleanup; never compare unrelated Figures' revision counters.

Acceptance criteria

• A documented session API and versioned feedback/event schemas are reviewed before implementation.

• A demonstration creates multiple Figures, captures exact human edits plus separately entered comments, updates runnable code and presents a complete next round.

• Re-executing the updated source reproduces the approved chart edits.

• Approval freezes and checkpoints individual Figures, but LLM source modification starts only after every required Figure in the round has an explicit disposition.

• All-Figure completion, stale revisions, repeated submissions, conflicting code edits, failed reruns, closed windows and cancellation have defined and tested behavior.

• Review metadata is excluded from normal chart export and images.

• Manual plotting/editing remains fully usable without the review mode or any LLM integration.

• Companion-skill packaging and supported agent integration are evaluated and documented.

• Termination during payload/manifest writing preserves the previous complete checkpoint.

• Agent disconnect and duplicate delivery resume the same work without loss or duplicate rounds.

• A complete higher revision triggers cleanup without an LLM action; restart repairs missed cleanup.

• Settings-only revisions reuse data and cleanup preserves retained references and active readers.

• Approval/checkpoint/close ordering and the all-Figure batch barrier work in native and notebook hosts.

Available foundations

PR #19 completed #15/#16; merged PR #21 completed #17. CI #50 passed all 593 integration tests. The merged source provides stable document identities, revision validation, shared editor commands, accepted SSOT operations and Undo/Redo/Reset. The in-memory edit journal is bounded and is not a durable session log; freeze explicit baseline/approved snapshots when constructing review feedback.

Use the integration design and editor contracts as the implementation baseline. Draft PR #22 implements review sessions, separate comment input, durable binary checkpoints, the local CLI/filesystem agent adapter and a packaged companion skill. Required CI #60 passed all 785 tests. API review remains pending; this issue stays open until the implementation is accepted.

Dependencies and coordination

• Completed foundations: compatible rendering in Restore compatibility with figgy-renderer 0.18.0 and figgy-model 0.12.0 #15, document identities in Expose all renderer 0.18 features through the Python API and rendering hosts #16, and validated editing/history in Complete interactive editor controls for existing and new chart capabilities #17.
• Review this issue's API, identity, revision and feedback contracts early, alongside Expose all renderer 0.18 features through the Python API and rendering hosts #16 and Complete interactive editor controls for existing and new chart capabilities #17; do not defer those decisions until all UI work is complete.

Draft implementation and validation

PR #22 documents the explicit Session lifecycle and protocol-1 feedback/event schemas. Approval checkpoints each Figure before closing its view, emits work only after the complete set has dispositions, and publishes a validated successor after leased source execution and exact-edit reconciliation. UI and targeted API comments are preserved as separate metadata channels. Recovery uses immutable binary renderer inputs and complete settings; original computation inputs and seeds remain the builder's responsibility.

Review mode currently supports one interactive view per Figure. Duplicate native windows are rejected and duplicate notebook views block approval until resolved. Agent commands execute on the rendering host through an existing execution bridge; no vendor SDK or automatic remote-chat connection is implied.

Validation: CI #60 passed all 785 tests on 5c96e6a. The four shards passed 197/196/196/196 cases; their pytest runtimes were 868.22/817.64/458.05/413.72 seconds. The required PR / Linux integration check audited actual passing JUnit node IDs against every full collection plan: zero omissions, duplicates or nonpassing cases. A separate log audit also matched all four execution orders and retained all 642 original IDs. Local checks passed 209 focused cases, nine displayed WebGPU visual cases, 14 execution-contract cases with repository-local temporary files, and 131 preflight cases. Runtime files, correctness assertions, per-test deadlines and the 45-minute per-GPU-job limit are unchanged.

Execution evidence: CI #58 passed 470 of 771 cases before the 45-minute job deadline cancelled it; no assertion failure was reported and this is not a full-suite result. The same existing radial-detail WebGPU test's first GPU submission increased from 11.622 seconds in CI #56 to 42.342 seconds, while pipeline preparation increased from 0.501 to 0.660 seconds. The nine new browser visual checks occupied approximately 125 seconds of the logged interval. Four independent runners now execute disjoint quarters of the identical full ordered collection. Assertions and per-test deadlines are preserved. Measurements.

Execution-audit regression: CI #59 caught a sample pytest subprocess inheriting the parent collection root when its temporary directory was inside the checkout. The same layout reproduced the failure locally; an explicit temporary root now isolates the sample run. All 14 execution-contract cases pass with repository-local temporary files, and preflight passes all 131 cases. The complete execution audit was retained.

Regression evidence: the original nine feature cases accepted a single black pixel on white, and ten original restoration cases accepted zero arrays. They now exercise independent authored visual expectations and complete cache/upload/device-data verification. 112 deliberately wrong results must be rejected: 45 visual faults, nine substitutions through the actual replay test, 42 zero/reversed/wrong-dtype restore payloads, 14 incorrect device/transfer contents and two cache-writer input mutations. Device/transfer controls keep settings, cache and observed upload arguments correct. The original baseline test accepted all ten original resident-data faults. A later source-capture check also accepted both writer mutations when the expectation was recorded after the write; original snapshots now precede the cache writer and renderer. The strengthened checks reject these faults. Contract, before/after evidence and local results. PR #22 remains a draft and this issue stays open pending acceptance.

Validation scope

original API/combination replay fixtures are retained. Nine independent scenes verify authored locations, colors, shapes and empty regions on native and displayed WebGPU output. Recovery compares complete column identities, dtype, shape, values and payload bits at cache/upload/transfer boundaries, plus actual native GPU sample readback; coverage includes matrix edges, mapped sizes/colors, point/grid vectors, shared-X fills and panel/overlay data. Four precision cases cover float32/float64, signed zero, a custom NaN payload and infinities. Native finite readback uses the packed pair-of-f32 representation's 2e-14 relative bound; native selection JSON reports nonfinite values as null, so exact nonfinite bits are checked at the lossless boundaries rather than claimed for that JSON observation. These finite scenes do not establish every feature combination, exact font rasterization or general usability.