Skip to content

feat(dashboard): model-search jobs and comparison view - #37

Merged
ng merged 1 commit into
mainfrom
feat/dashboard-search
Nov 16, 2026
Merged

ng merged 1 commit into
mainfrom
feat/dashboard-search

Conversation

@ng

@ng ng commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Plan B5 (training-11): model-search jobs and a comparison view.

On main d857949 (after #34 and #35).

Jobs. There are three new allow-listed actions, each with the usual schema, data/ path checks and argv-only execution:

  • search-plan: search.py plan SPEC --out NEW [--cache DIR]
  • search-run: search.py run RUN [--jobs N]
  • search-report: search.py report RUN

The run folder must hold a plan.json, and the run is linked into the run list when the job finishes. There is no final action. final reads the locked holdout once and exports model/set.json, so it stays a deliberate terminal step. Tests assert that:

  • no action argv contains final
  • no action exposes --candidate, --access-log or --holdout-final
  • search.py has no pod-write routes

View. Search in the sidebar lists runs of kind search. A run shows:

  • the pre-registered primary comparison: mean paired difference, night-bootstrap 95% interval, nights better/worse, sign test
  • the exploratory winner, with how many candidates were compared
  • a candidates table: pooled walk-forward macro F1, κ and per-stage F1, mean-night F1, and each exploratory comparison against the baseline, including the pod's live stages
  • vitals and movement layer scores
  • exclusions and limitations
  • Run search and Report buttons

With more than one search, an "All searches" table compares their primary results and winners.

Holdout. The view uses only report.json, which counts holdout reads but holds no holdout scores. final.json is never read or served. A test plants a sentinel score in final.json and checks it appears in no /api/runs* response. The view shows only "holdout reads so far" and whether this run has had its final read. Per-night rows aren't displayed at all: the vitals layer shows pooled MAE and ρ only.

Test plan

  • uv run --locked python scripts/validate_knowledge.py, check-vendor.py
  • uv run --locked python -m unittest discover -s scripts -p 'test_*.py': 290 tests, including the search action schemas and argv, the plan.json requirement, no final, and no final.json in served payloads
  • npm run typecheck, node --check scripts/ui/dashboard.js, node --test scripts/test-dashboard.mjs: 32 tests, including row ranking, comparison text, cross-search summary, and a render for scored, final, ready and blocked reports that never shows a planted final score
  • Synthetic make-synthetic-inputs.py + train.py smoke run
  • smoke-pod.py synthetic HTTP pod run
  • End to end on CI's synthetic search inputs (docs/search/synthetic.json, 14 nights), all through POST /api/jobs: search-plan, search-run and search-report each completed and linked the run. No final.json was written. Earlier, against the same inputs, in headless Chrome the Search view showed the primary comparison, the 4-candidate table, the layer scores, "Holdout reads so far: 0" and the limitations, and Report started a job and opened its log.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 43 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cdf12eca-8e25-4793-95bd-7a0b54d686aa
Commits

Reviewing files that changed from the base of the PR and between d857949 and 345ef9f.

Files selected for processing: 7

  • README.md
  • scripts/dashboard.py
  • scripts/test-dashboard.mjs
  • scripts/test_web_jobs.py
  • scripts/ui/dashboard.html
  • scripts/ui/dashboard.js
  • scripts/web_jobs.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@ng
ng force-pushed the feat/dashboard-search branch from 2763e1d to ecd078c Compare November 16, 2026 00:47
…ce and compare their reports without ever showing holdout scores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ng
ng force-pushed the feat/dashboard-search branch from ecd078c to 345ef9f Compare November 16, 2026 00:49
@ng
ng marked this pull request as ready for review November 16, 2026 00:49
@ng
ng merged commit b1cc690 into main Nov 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant