fix(cli): give wheels test its own timeout budget instead of the bridge default - #3359
Conversation
There was a problem hiding this comment.
Wheels Bot — Reviewer
TL;DR — This PR gives wheels test its own read-timeout budget (900s default, --timeout / WHEELS_TEST_TIMEOUT overridable) instead of inheriting the shared 120s bridge budget, applies the same budget to the browser-test runner, and hardens tools/test-local.sh against stale/overwritten result files. The CLI code change is correct, well-documented, and properly unit-tested. Verdict: comment — the timeout fix is solid and I'd merge it, but the PR also commits a stray rewrite.config at the repo root that is unrelated to the fix and should be dropped before merge.
Conventions
rewrite.config(new file, repo root) is an accidental inclusion and should be removed from this PR. It is byte-for-byte identical tocli/lucli/templates/app/rewrite.config— i.e. the artifact LuCLI'sCatalinaBaseConfigGeneratordrops into a project root when you runwheels server run(which the PR body says was used to run the suite). It has nothing to do with the timeout fix, and the PR body itself states the intent to keep "this PR stays one change" and lists two other items deliberately deferred to their own PRs. Committing it also creates a latent footgun: because a rootrewrite.configoverrides LuCLI's bundled template verbatim, a tracked copy will silently go stale if the template ever changes. Recommendgit rm rewrite.configfrom the branch (and consider adding it to.gitignore, since it's a per-environment generated file).
Cross-engine
No issues. This is CLI code on the bundled Lucee runtime; the new parameters (parsedTimeout, readTimeout, timeoutSeconds) avoid reserved scope names (anti-pattern #11), and makeHttpRequest/makeHttpRequestWithStatus are called with fully-named args, so no positional/named mixing. javacast("int", arguments.readTimeout) on 900 * 1000 = 900000 is well within int range.
Tests
Good coverage. TestCommandSpec.cfc::$resolveTestTimeout exercises the default ("", " "), explicit values ("1800", " 45 "), and the junk/non-positive fallback ("soon", "0", "-30") — matching the three documented branches of $resolveTestTimeout. New public method, new BDD specs extending the CLI BaseSpec; TDD expectation met.
Correctness
Verified the change does what the title claims:
Module.cfc:5823threadstimeoutSeconds * 1000into the test-runner request; thecatchat:5869matches(read timed out|SocketTimeout)—java.net.SocketTimeoutExceptionsurfaces asRead timed out, so the branch fires correctly and prints an actionable message instead of the raw engine error.- The shared
makeHttpRequestWithStatuskeeps itsreadTimeout = 120000default, so the other ~8makeHttpRequest(...)bridge callers are unaffected — confirmed each still calls the single-arg form. tools/test-local.sh:39keysRESULT_FILEonPROJECT_ROOT(defined at:28, before use) and:139clears it before the curl. This file'sRESULT_FILEis self-contained — CI reads results viatools/ci/run-tests.sh/compat-matrix.yml, which use their own paths, so no CI breakage.
Two minor, non-blocking notes:
tools/test-local.sh:39usesshasum, which is Perl-provided and not guaranteed on minimal Linux dev boxes;sha1sumis more universally present there. Cosmetic — the override env var is the real escape hatch.- The browser-runner
catch(Module.cfc:8087) still prints the generic "Failed to reach test runner… Is the server running?" on a timeout rather than the new timeout-specific guidance. Not a regression, just an inconsistency with the primary path.
Commits / Docs
- Commit
dbc93a51efix(cli): give wheels test its own timeout budget instead of the bridge defaultconforms to commitlint (valid type/scope, subject ≤100, not ALL-CAPS). - Changelog fragment
changelog.d/3352-test-timeout.fixed.mduses the correct<slug>.<type>.mdfragment form (not a directCHANGELOG.mdedit). Good.
Nice work on the fix itself — clear rationale in the code comments and honest scoping in the PR body. Just drop the stray rewrite.config and this is ready.
…ge default `wheels test` failed with `Read timed out` and NO result document on a suite of roughly 500 specs taking about 2.5 minutes — not a failure report, a crashed runner. Indistinguishable from a hung app to anyone who has not seen it before, and it scales IN: a suite works, then silently stops working as it grows. The cause is one line. `makeHttpRequestWithStatus()` hardcodes `conn.setReadTimeout(120000)` for every caller, which matches the reported ~140s threshold. That budget is correct for the short request/response bridge commands, but a test run is the one command here whose duration is expected to scale with the project, so it now gets its own. - `--timeout=<seconds>`, else WHEELS_TEST_TIMEOUT, else 900. - Non-numeric or non-positive input falls back to the default rather than throwing: a mistyped timeout should not be the thing that stops a test run. - The browser-test runner at the second call site makes the same long-running request over the same helper, so it gets the same budget. Fixing only one would have left the identical bug in a sibling. - On a timeout the message now says which side gave up, that the specs may well have passed, and how to give it longer or scope the run. The old output was the raw engine message. The shared helper keeps its 120s default, so no other command's behaviour changes. Two harness defects in the same family, both of which bit me while verifying tonight's other PRs, and both matching this issue's theme of a runner that reports something other than what happened: - tools/test-local.sh wrote results to a single fixed /tmp path shared by every checkout on the machine. Two working copies running the suite overwrite each other — which silently turned my first develop-vs-branch comparison into two copies of the same run, with identical totals that looked like a legitimate no-op result. Now keyed on the project root, overridable with WHEELS_TEST_RESULT_FILE. - When the request failed outright (HTTP 000, typically a server not yet up) the previous run's results were left in place and even printed. I read one of those as a current result before noticing the numbers were implausible. The file is now cleared before the request, so a crashed run leaves no result rather than a stale one. 3 specs on $resolveTestTimeout covering the default, an explicit value, and the junk-input fallback. Verification, CLI suite via /wheels/cli/tests: develop ab901cf 1143 pass / 0 fail / 0 error / 1203 specs this branch 1146 pass / 0 fail / 0 error / 1206 specs Exactly +3, the new specs. Core suite unaffected and unchanged at 4732. Note: tools/test-cli-local.sh could not be used — it invokes a `lucli` binary that does not exist on a normal install (cli/CLAUDE.md: `wheels` IS the binary). Both runs above went through the /wheels/cli/tests endpoint against a server started with `wheels server run`. Closes #3352 Signed-off-by: Peter Amiri <peter@alurium.com>
dbc93a5 to
ce125db
Compare
There was a problem hiding this comment.
Wheels Bot — Reviewer
TL;DR — This PR gives wheels test its own read-timeout budget (900s default, overridable via --timeout / WHEELS_TEST_TIMEOUT) instead of inheriting the shared 120s bridge budget, extends the same budget to the browser-test runner, adds a timeout-specific error message, and hardens tools/test-local.sh against stale/overwritten result files. The previous review's only blocker — a stray rewrite.config at the repo root — has been dropped; the file list is now exactly the four files the fix needs. The change is correct, well-documented, and properly unit-tested. Verdict: approve.
Previously-blocking finding, now resolved
rewrite.config(repo root) removed. The prior review flagged an accidental byte-for-byte copy ofcli/lucli/templates/app/rewrite.configcommitted at the repo root.gh pr view 3359 --json filesnow returns onlychangelog.d/3352-test-timeout.fixed.md,cli/lucli/Module.cfc,cli/lucli/tests/specs/commands/TestCommandSpec.cfc, andtools/test-local.sh— the stray file is gone. Resolved.
Correctness
Verified the change does what the title claims:
Module.cfc:5823threadsarguments.timeoutSeconds * 1000into the test-runner request;runTests's newnumeric timeoutSeconds = 900param (:5754) is threaded positionally in call order fromtestCommandand defaults correctly.- The
catchat:5869matches(read timed out|SocketTimeout);java.net.SocketTimeoutExceptionsurfaces its message asRead timed out, so the branch fires and prints actionable guidance rather than the raw engine error.runState.crashed = truestill propagates to theWheels.TestRunFailedthrow at:5888, so a timed-out run exits non-zero — no silent green. makeHttpRequest(:7643) gainsnumeric readTimeout = 120000and forwards it via named args tomakeHttpRequestWithStatus; every othermakeHttpRequest(...)bridge caller uses the single-arg form and keeps the 120s default, so no unrelated command changes behaviour.javacast("int", 900000)is well withinintrange.tools/test-local.sh:39keysRESULT_FILEonPROJECT_ROOT(set at:28, before use) and:139clears it before the curl, so a crashed run leaves no stale result to misread.
Conventions
Clean. $resolveTestTimeout is a public $-prefixed helper — the established CLI pattern for spec-reachable internals — and mcpHiddenTools()'s structural sweep (Module.cfc:200–215) hides every $-prefixed public function from the MCP surface, so it does not leak as a tool. Mirrors the existing $resolveTestBasePath / $normalizeTestFilter prior art the change explicitly follows.
Cross-engine
No issues. CLI code on the bundled Lucee runtime; new params (parsedTimeout, readTimeout, timeoutSeconds) avoid reserved scope names (anti-pattern #11), and the two forwarding calls use fully-named args (no positional/named mixing). val(" 45 ") trims correctly and isNumeric(trim(...)) guards the parse.
Tests
Good coverage. TestCommandSpec.cfc::$resolveTestTimeout exercises the default ("", " "), explicit values ("1800", " 45 "), and the junk/non-positive fallback ("soon", "0", "-30") — matching the three documented branches of $resolveTestTimeout. New public helper, new BDD specs extending the CLI BaseSpec; TDD expectation met.
Commits / Docs
ce125db99fix(cli): give wheels test its own timeout budget instead of the bridge defaultconforms to commitlint (valid type/scope, subject ≤100, not ALL-CAPS).- Changelog fragment
changelog.d/3352-test-timeout.fixed.mduses the correct<slug>.<type>.mdform (not a directCHANGELOG.mdedit). Good.
Minor (non-blocking) nits
tools/test-local.sh:39usesshasum(Perl-provided, not guaranteed on minimal Linux boxes);sha1sumis more universally present. Cosmetic —WHEELS_TEST_RESULT_FILEis the real escape hatch.- The browser-runner
catch(Module.cfc:8087) still prints the generic "Failed to reach test runner… Is the server running?" rather than the new timeout-specific guidance, even though it now passes the longer budget. Not a regression, just an inconsistency with the primary path — worth a follow-up.
Nice, tightly-scoped fix with honest rationale in the code comments and PR body. Approving.
Closes #3352.
The cause is one line
That matches the reported ~140s threshold exactly. The budget is right for the short request/response bridge commands, but a test run is the one command here whose duration is expected to scale with the project — and when it blew, the CLI produced no result document at all. Not a failure report, a crashed runner, indistinguishable from a hung app.
The fix
--timeout=<seconds>, elseWHEELS_TEST_TIMEOUT, else 900.On a timeout the message now names the real situation instead of echoing the engine:
Two harness defects in the same family
Both of these bit me while verifying tonight's other PRs, and both match this issue's theme — a runner reporting something other than what happened:
tools/test-local.shwrote results to a single fixed/tmppath shared by every checkout on the machine. Two working copies running the suite overwrite each other. This silently turned my first develop-vs-branch comparison into two copies of the same run — with identical totals, which read like a legitimate no-op result. Now keyed on the project root; override withWHEELS_TEST_RESULT_FILE.A request that failed outright left the previous run's results in place, and even printed them.
HTTP 000(typically a server not yet up) does not truncate the file. I read one of those as a current result before noticing the numbers were implausible. The file is now cleared before the request, so a crashed run leaves no result rather than a stale one.Verification
CLI suite via
/wheels/cli/tests:ab901cff7Exactly +3 — the new
$resolveTestTimeoutspecs (default, explicit value, junk-input fallback). Core suite unaffected and unchanged.A note on how I ran that:
tools/test-cli-local.shcould not be used — it invokes aluclibinary that does not exist on a normal install (cli/CLAUDE.md: "wheelsIS the binary… there is no separateluclibinary"). It fails withnohup: lucli: No such file or directory. Both runs above went through the endpoint against a server started withwheels server run. Worth its own issue; say the word.Not included, from the issue's "adjacent" section
Deliberately left out so this PR stays one change:
bundles=is accepted and silently ignored. You are right that this is worse than rejecting it. It is a server-side change in the test runner, so it wants its own PR and its own matrix run.?directory=is undocumented. It is the main workaround for this bug and deserves surfacing in the guides — a docs change, and arguably less urgent now that the timeout no longer forces people into it.Happy to take both next.