Skip to content

fix(ci): memoize Mission Control build cascade (stop 55-min timeout) - #21

Open
pilotmain wants to merge 4 commits into
mainfrom
fix/ci-mission-control-memoization
Open

fix(ci): memoize Mission Control build cascade (stop 55-min timeout)#21
pilotmain wants to merge 4 commits into
mainfrom
fix/ci-mission-control-memoization

Conversation

@pilotmain

Copy link
Copy Markdown
Owner

Summary

  • Root cause: Mission Control build_* services form a diamond dependency with no memoization, so top-level builders recompute the full stack exponentially. That made Python tests + lint hit the 55-minute job timeout (not an assertion failure).
  • Fix: request-scoped @scoped_build memoization across the cascade (build_memoization.py), plus pytest-timeout in CI so any future hang fails fast with a named stack instead of silently burning the budget.
  • Local check: hang-prone pilotos certification file now passes in ~7s (previously hung for minutes). Job name Python tests + lint is unchanged so the branch ruleset stays satisfied.

Test plan

  • Confirm Python tests + lint completes under 55 minutes on this PR
  • Confirm DCO sign-off is green
  • After merge, rebase/re-run PR #20 (launch discoverability) and Dependabot PRs

Made with Cursor

pilotmain and others added 4 commits August 25, 2026 11:49
The Python CI job hit the 55-minute timeout because diamond-shaped
build_* recomputation grew exponential. Add request-scoped memoization
and a pytest-timeout safety net so hangs fail fast with a named stack.

Signed-off-by: Raya Meresa <rayameha@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Unblocks the required Dependency audit + SBOM check (PYSEC-2026-3552/3553/3554).

Signed-off-by: Raya Meresa <rayameha@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Clears the high-severity GHSA-2v37-7h3g-55p8 finding that fails the
required Dependency audit + SBOM web check.

Signed-off-by: Raya Meresa <rayameha@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Guard in-progress keys so a cyclic build_* call cannot recurse forever, and
fall back when deepcopy hits circular graphs.

Signed-off-by: Raya Meresa <rayameha@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@pilotmain

Copy link
Copy Markdown
Owner Author

Status update

Timeout root cause is fixed. Python tests + lint now finishes the full suite in ~8–11 minutes (previously hit the 55-minute job timeout with no failure output).

Required checks already green on this PR:

  • DCO sign-off
  • Dependency audit + SBOM (after cryptography 50.0.0 + nanoid 3.3.18 bumps)

What CI now surfaces: ~400 real test failures / ~5436 passes. Spot-checked several failures with memoization disabled (no-op scoped_build) — same failures — so these look pre-existing on main, previously invisible because the job never completed.

Examples of pre-existing-looking failures:

  • VOICE_WAKE_ENABLED leaked in UI (test_ui_no_internal_flag_leak)
  • chat falling through to “generative intelligence is configured” when USE_REAL_LLM=false
  • Railway registry not_found for known aliases
  • 404s on /runtime/models endpoints

Next step after this PR merges: triage/fix the pre-existing failures so main can go fully green, then rebase launch PR #20 + Dependabot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant