Skip to content

Fix Follow Up UI tests timing out on a cold app.tsx import - #18

Merged
matthewdias merged 1 commit into
mainfrom
fix-follow-up-flaky-ui-tests
Oct 4, 2026
Merged

matthewdias merged 1 commit into
mainfrom
fix-follow-up-flaky-ui-tests

Conversation

@matthewdias

Copy link
Copy Markdown
Owner

Fixes the flaky Follow Up UI tests seen during #16. This PR changes tests only, so there is no version bump.

Cause

The flake was a cold import running inside a test's 5s timeout. It was not test order or state leaking between tests.

app.tsx imports @hugeicons/core-free-icons through src/handoff.tsx and components/ui/icon*.tsx. That package's ESM entry re-exports about 6,000 per-icon modules. Imported cold under vitest, it took 7.6s and 11.1s in two timed runs. Everything else app.tsx imports took under 2s together.

Three files loaded the app with loadPluginApp(() => import("../../app.tsx")) inside a test body: app.test.tsx, picker.test.tsx and record-draft.test.tsx. The first such test in each file paid the whole cost against vitest's 5s default and failed with Test timed out in 5000ms. pill-removal.test.tsx was never affected, because it imports banner.tsx, and with it the icons, at the top of the file.

This explains #16. The "three unrelated failures" were one timeout per file, and whether they appear depends on machine load. They were never related to the deliberate break.

Reproduction on a4da82e, with load average 34 to 73 on 16 cores

  • A plain npx vitest run failed 3 of 43 tests, every run. The failing tests took 10.4s, 10.1s and 10.6s.
  • Each file run alone still failed its first app-loading test. Those tests took 11.6s, 6.6s and 5.2s.
  • --maxWorkers=1 gave the same 3 failures.
  • Shuffled seeds 1 to 5 failed exactly one test per file each time. The test that failed was whichever ran first in that file. For example, seed 3 failed "offers the + menu row…", "reports a message action's refusal…" and "has the picker in its own thread-only customization".
  • As a control at a lower load average (about 9), the unfixed tests passed, but the first app-loading tests took 3.5s to 4.2s, close to the limit.

Fix

The three files now import app.tsx at the top and call loadPluginApp(pluginApp). The import cost moves to collection, where no per-test timeout applies, which is the same pattern pill-removal.test.tsx already uses.

The SDK supports this. loadPluginApp accepts a module as well as a loader function, and in SDK 0.6.15 definePluginApp and the hooks don't need the test runtime at import time. No timeouts, retries or assertions changed, and no test-only exports were added.

Evidence

  • The three tests that timed out now take 6ms, 7ms and 18ms, down from 3.5s to 11.6s.
  • The failing plain run and seeds 1 to 5 now pass 43 of 43.
  • 65 more runs all passed 43 of 43: 60 shuffled seeds (101 to 160) and 5 plain runs, at load averages between 10 and 64.
  • The original break still fails exactly one test. I made the + row's availability check always true (disabled: (composer) => false && …) and ran the suite 5 times. Each run failed only app.tsx > offers the + menu row whenever the thread has open rows, banner open or not (42 of 43 passed).
  • npm run check at the repo root passes.

Not changed

  • Two things depend on order in principle but have never failed. Two pill-removal tests share the thr_mark thread in the store, and both seed the same rows. src/rpc.ts remembers the RPC client at module level, and every record-draft test that reads it sets it first.
  • Importing icons by subpath in the shipped code (@hugeicons/core-free-icons/<Icon>) would make every load of app.tsx and banner.tsx fast. That changes shipped code and needs a release, so it is left for a separate PR.

🤖 Generated with Claude Code

… out

The first test in app, picker and record-draft to load the app paid for a
cold import of @hugeicons/core-free-icons, about 6,000 modules, inside its
own 5s timeout. On a busy machine that took 5-11s, so one test per file
failed, and which one moved with test order. Importing app.tsx at the top
moves the cost to collection, as pill-removal already does with banner.tsx.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matthewdias
matthewdias merged commit da09d93 into main Oct 4, 2026
1 check passed
matthewdias added a commit that referenced this pull request Oct 5, 2026
The registry checked `detail` (the first 8 rows, each needing a label
and a value) and `open` (an http(s) or app-path href, nothing else).
No surface draws either yet; the floating card would be the first.
Released copies freeze what they validate, so a card that turned out to
need 12 rows, a row without a value, or a click that runs a command
would have had those stripped by every older copy, for good.

Both are now reserved names that pass through like any field: copied as
JSON, deep-frozen, inside the size cap. The first surface that draws
them defines their shape, and it validates them before drawing, an href
above all. What v1 still checks is exactly what a real surface has
drawn: icon, label, tone, text and fraction.

Nothing uses either field today, and nothing has been released, so
COMPLICATIONS_IMPLEMENTATION stays at 1.

Replaces the three validation tests with two that pin the pass-through,
including an href arriving exactly as sent. Making either name
validated again turns them red.

Also moves the heavy imports in two new UI test files to the top of the
file, as #18 did for Follow Up's older ones: the first load of the
hugeicons package outlasted a test's timeout on a busy machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant