fix(flow): run a device-less flow without a device, and say why its counts are zero - #649
Open
filip131311 wants to merge 1 commit into
Open
fix(flow): run a device-less flow without a device, and say why its counts are zero#649filip131311 wants to merge 1 commit into
filip131311 wants to merge 1 commit into
Conversation
A flow of nothing but `echo` steps still went through device resolution, so it failed outright when nothing was booted — and picked whichever device happened to be running when something was, attributing a run to hardware it never touched. Whether a device is needed is now decided from the flow's own steps, before resolution. The classification is per step kind and defaults to needing one, so a kind added later inherits today's behaviour rather than quietly running against no device; the compiler rejects leaving a new kind unclassified. A `when` block needs a device whatever its body contains, since the guard reads one itself, and a `run` step counts as needing one without the fragment being read here — resolving it twice could disagree with the run-time resolution. `ExecState.device` becomes nullable so every site that acts on a device has to say so, and the executor re-checks each step against the same predicate: if the two decisions ever disagree the step reports it, rather than failing obscurely deeper in. A run with no device reports an empty `device` rather than borrowing one. The summary for such a run said `PASS — 0 passed, 0 failed, 0 errored, 0 skipped`, which reads as though nothing happened. Narration deliberately isn't counted — and should not be, since after a hard stop every remaining step including narration is reported skipped, so counting it would inflate that number on real failing runs. The count is right; what was missing was saying why it is zero. Both the CLI and the MCP renderer now add `(no test steps)`, and only on a pass, where the counts are what needs explaining. Fixes #644
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #644.
Before
and when a device was booted:
After
Symptom 1 — the device
Whether a device is needed is decided from the flow's own steps, before resolution.
Classification is per step kind and defaults to needing one, so a kind added later inherits
today's behaviour instead of silently running against no device — and the
neverbinding makesleaving a new kind unclassified a compile error, backed by a test keyed on the step union so it
fails to build rather than to run.
Two classifications worth calling out:
whenalways needs a device, whatever its body contains, because the guard reads one itself(its platform, or its view tree). A
when: {platform: ios}block containing only narration stillresolves.
runneeds one without the fragment being read here. The named flow is resolved at run time;resolving it a second time would duplicate that lookup and could disagree with it if the file
changed in between. A deliberate over-approximation: composing a narration-only fragment still
resolves a device. Stated in the code and in the tool description.
For
tool:steps the tool's own schema decides, via a superset of the keys the runner injects —deviceis included because a nestedflow-executestep drives a device without being handed therun's own. A tool that declares no input at all (
stop-all-simulator-servers) is device-free, andreading its absent schema must not throw — it previously would have.
ExecState.deviceis nullable, which is the safety mechanism rather than just typing: the compilerenumerates every site that acts on a device, each of which now says so via one
deviceEnv()narrowing.
flow-actions.ts,flow-visual.tsandActionEnvare untouched. The executor alsore-checks each step against the same predicate, so if the scan and the executor ever disagree
the step reports it instead of failing obscurely deeper in.
A run that resolved no device reports
device: "", and neither renderer claims it ran somewhere.Symptom 2 — the counters
Reproducing this showed the issue's framing is slightly off:
--jsondoesn't only disagree with thetext, it disagrees with itself — counters all zero while its own
steps[]marks both steps"pass". So the CLI text was a faithful rendering of the server's counters.The counters are right and stay as they are. Narration is deliberately excluded, and it should
be: after a hard stop every remaining step including narration is reported skipped, so counting it
would inflate
skippedon every real failing run — a worse and far more common lie than the vacuousone being reported here. It would also contradict the JUnit model in #578, where echo is
<system-out>and not a<testcase>.What was missing was saying why zero. Both the CLI and the MCP renderer now append
(no test steps), and only on a pass — a cancelled run can be a FAIL with all four counters zero,where "no test steps" would read as though the failure had no cause.
No wire format changes:
steps[].kind === "echo"already distinguishes counted from uncounted steps,so the JSON is self-describing without widening the status enum.
Conflicts, for whoever lands second
This is a hot area — seven open PRs touch
flow-run.ts.independent — resolution is keep-both), and its
attachFailureDiagnosticsdereferencesenv.device.platform. It will need a null-device guard, or every failure in a device-less flowsilently degrades to
read-faileddiagnostics. Happy to rebase onto it rather than the reverse.resolveRunDeviceat the same seam and edits the same description sentence.flow-device.ts. Worth noting it makesresolveFlowDevicereject a physicaliPhone for flows — skipping resolution means a narration-only flow now passes on a
physical-iPhone-only host, which I think is desirable but is a second behaviour change on this seam.
All new tests are in new files, since every existing flow test file in the blast radius is rewritten
by an open PR.
Behaviour changes a reviewer should weigh
PASS on <id>even when a device is booted. That is thepoint: otherwise an identical flow reports differently in CI (no simulator) than on a laptop with
one open. An explicit
--deviceis still honoured and still named.--platformis ignored for a device-free flow, where it previously errored with nothing booted.Flow "x" on— cosmetic version skew.Verification
Live against the built CLI: device-less flow with nothing booted (the reported bug) and with a device
booted (identical output, no ambient attribution);
--jsonshowingdevice: ""; explicit--devicestill naming it; a mixed flow and a composed flow both still demanding a device; a mixed flow with a
device booted still reporting
PASS on <id> — 1 passed.24 new tests. tool-server 3103 passed, CLI 284 passed, MCP 78 passed — no existing test modified.
🤖 Generated with Claude Code