Skip to content

feat(cli): add replay — record once, verify forever - #125

Open
totigm wants to merge 1 commit into
feat/cli-packagefrom
feat/cli-replay
Open

totigm wants to merge 1 commit into
feat/cli-packagefrom
feat/cli-replay

Conversation

@totigm

@totigm totigm commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Stacked on #118. Base is feat/cli-package, so the diff shows only the replay work. Merge #118 first and this retargets to main cleanly.

replayTimeline has returned per-step pass/fail since the recorder shipped. Nothing outside the library could call it. This exposes it:

npx @humanjs/cli demo https://app.example --record flow.json   # capture
npx @humanjs/cli replay flow.json --headless                   # verify
replaying flow.json — 6 steps (careful, human)

  ✓  1  goto
  ✓  2  sleep
  ✗  3  hover
        locator.boundingBox: Timeout 3000ms exceeded.

  FAIL  step 3 of 6 (hover) · 5.3s

Exit 1 on the first failed step. That is the whole feature — a recorded .json becomes a regression check that drops into CI with no test framework and no code. The .spec.ts export is the other route; this is the one that needs nothing else installed.

process.exitCode is set rather than process.exit() called, so stdout flushes. A truncated final line in a CI log is worse than useless.

Replays reproduce what was recorded

A timeline stores the personality, speed and seed it was captured under. replayTimeline ignores all three and always starts from careful/human — so a run recorded as distracted replays as something else, which is the wrong default for a regression check. replay resolves each setting as flag → recorded value → default, and ignores a recorded value that is not a known option, since timelines are files on disk and may be hand-edited.

This is why --personality and --speed became optional in the parser: the command needs to tell "not specified" from "explicitly set to the default".

Two bugs found by actually running the round trip

Neither is visible from reading the code — both took recording a timeline and replaying it.

--record lied about four of its five formats. The recorder's own output handling only distinguishes .gif from video, so .json, .ts and .spec.ts were all routed to the video encoder and written as mp4s wearing the wrong extension — while the help text advertised all five. The CLI now owns the mapping, and checks the extension before launching a browser: a typo fails in 0.18s instead of after a full run.

Timelines recorded by demo could not be replayed. demo passed a Locator to hover, and the recorder serialises a target with String(), so the timeline contained locator('a[href]').first() — not a selector. The replay died on it every time. Targets are now narrowed with Playwright's >> nth=0, which avoids the strict-mode hang that motivated the Locator and survives the round trip.

Also: --timeout <ms>

The failing replay above took 32.2 seconds on Playwright's 30s default to report something it knew in two. With --timeout 3000 the same failure lands in 5.3s. Applies to all three commands.

Verification

80 unit tests in @humanjs/cli (was 31): timeline parsing and its rejections, format dispatch including .spec.ts beating bare .ts, setting resolution, summary wording, --timeout parsing.

End to end from dist/:

Case Result
demo --record flow.json 6 events, real selectors in the file
replay flow.json PASS 6 steps · 3.2s, exit 0
replay with a broken target FAIL step 5 of 6 (hover), exit 1
--timeout 3000 on that failure 32.2s → 5.3s
--record out.avi rejected in 0.18s, no browser

Whole suite: lint, typecheck (11/11), test (10/10), build (8/8), check:exports (15/15).

Next up is compare <url> — the side-by-side robotic-vs-humanized asset, generated for the user's own site.

replayTimeline has returned per-step pass/fail since the recorder
shipped, and nothing outside the library could call it. `replay` exposes
it and exits 1 on the first failed step, so a recorded .json becomes a
regression check that needs no test framework and no code.

The personality, speed and seed the timeline was captured with are
reused unless overridden -- replayTimeline on its own always starts from
careful/human, and a run recorded as distracted replays differently.
@vercel

vercel Bot commented Sep 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
humanjs Ready Ready Preview Sep 7, 2026 4:39am UTC

@totigm

totigm commented Sep 7, 2026

Copy link
Copy Markdown
Owner Author

Note on the checks above: only Vercel ran. ci.yml triggers on pull_request: branches: [main], and this PR targets feat/cli-package, so the test matrix does not fire on a stacked PR. It is not a failure — it never started.

The full suite was run locally against this branch: lint, typecheck (11/11), test (10/10, 80 of them in @humanjs/cli), build (8/8), check:exports (15/15). CI will run for real once #118 merges and this retargets to main.

This branch was successfully deployed

1 active deployment
Preview — 6d3e4362 Deployed Sep 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant