Agent-led TUI tests of the real dashboard against the live fixture #42

Posted by AI 🤖

Problem

The live suite from #23 (PR #41) validates the gh/gh stack adapter against fixture PRs, but nothing drives the real stack-stat dashboard against them. Dashboard behavior on real stacks (rendering, the link-then-act prompt, notices driven by real failures) is covered only by TestBackend tests with fake providers.

Goal

Let a local coding agent run full TUI tests of the real binary against the live fixture stacks, observe the screen, and report pass or fail. This is a local, agent-led workflow, not a CI gate.

Starting point

• Fixture stacks in this repo: s1 (three PRs), s2 (conflict), s3 (merged, branches deleted), s4 (unstacked), and fork PRs reusing head branch names. See docs/architecture/07-live-validation.md.
• scripts/live-fixture.sh reset|conflict|status and make live-fixture-reset restore start state; fixture-start/* tags hold baselines.
• VHS 0.11.0, ttyd and ffmpeg run locally. An exploratory VHS run against a fixture clone rendered the real dashboard, including CI check rows.

Scenarios to cover

• Dashboard renders each fixture stack: s1 layers and draft state, merged s3, not-in-stack for a branch with no PR or an unstacked PR.
• Fork PR sharing a head branch name resolves to the right PR.
• Push or sync from a clone with no local tracking shows the link confirmation; y links then retries; declining cancels.
• Sync against the conflicting trunk (live-fixture.sh conflict) shows the rebase-conflict notice and leaves the TUI open.
• Draft/ready toggle (m) updates the row.
• Missing auth and missing extension notices.

Approaches to evaluate

• VHS tapes: Wait+Screen regex assertions plus Output *.txt text captures and .png screenshots the agent can inspect. Already installed; also produces artifacts for review.
• PTY harness: a Rust or script driver that reads the terminal buffer. More precise assertions, more code.
• Agent-driven session: the agent launches the TUI in a PTY or VHS, sends keys, and judges screenshots against a written scenario checklist.

Constraints

• AGENTS.md currently says "Do not add PTY-based integration tests or visual snapshots". Either keep the work as an agent-run skill or script outside the test suite, or revisit the rule explicitly with an ADR that supersedes the relevant part of ADR-0003.
• Mutating scenarios must reset the fixture before and after and must not run concurrently with the nightly live workflow.
• The fixture can only be restored for s1 and s2; s3 is read-only.

Deliverables

• A documented one-command or one-skill way for an agent to run the scenarios locally and report results, with captured text or screenshots as evidence.
• A decision on tooling (VHS vs PTY) and on the AGENTS.md rule, recorded as an ADR if the rule changes.
• Scenario list with covered and uncovered cases.

Done when

An agent can run the scenarios against the fixture on a developer machine and report, with evidence, whether the real dashboard behaves correctly for each; a change to dashboard behavior that breaks a scenario is visible in that run.

Refs: #23