Skip to content

Charts: catch-model calibration and residual diagnostics - #151

Merged
Luen merged 5 commits into
mainfrom
codex/125-model-diagnostics
Nov 4, 2026
Merged

Luen merged 5 commits into
mainfrom
codex/125-model-diagnostics

Conversation

@Luen

@Luen Luen commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Adds expandable in-sample diagnostics to /drivers: observed versus expected catch rates, zero-catch shares by nursery/season, and Pearson residuals versus fitted count and season day. Each uses its exact species-specific fitted rows, design matrix and effort offsets. Failed/unsupported fits show an explicit unavailable state. Calibration rates are ratios of summed counts to summed net-hours; NB2 zero probabilities and residual variance use the fitted dispersion.

Closes #125.

Validation: lint; svelte-check (0 errors/warnings); 87 unit tests; opt-in real-data import; an additional opt-in real-data diagnostic reconciliation test; production build; 31 existing browser tests and the new diagnostics selector/light/dark axe check (passed on focused rerun after fixing its locator). The synthetic browser fixture exercises the unsupported-model state; numerical tests and the real-data check exercise supported fits. No private data committed.

@Luen Luen added enhancement New feature or request analytics Charts, statistics and analyses labels Nov 4, 2026
Luen and others added 4 commits November 4, 2026 14:21
- Page weight: the diagnostics sent every set as an object with eight
  unrounded fields for all three models, about 500 kB of JSON on every
  /drivers load. Only what the charts draw goes out now, rounded and in
  columns: 47 kB (12 kB with Brotli).
- Zero-catch charts show nursery names and formatted seasons instead of
  raw ids; shares are percentages in the chart and table.
- The section sits after the model it checks, inside the model's {#if},
  instead of above the single-variable correlations.
- Colours: the observed series takes the model's colour (species colours
  for the species models) and expected is muted, instead of slots 1 and
  2, which mean blacktip and lemon on this page.
- Plainer copy and axis labels; dates on the season axis; a log x axis
  for residuals against the expected catch, with a note on the bands.
- The 20-set threshold is one constant used by the filter and the copy.
- Tests: the Poisson limit, labelled groups and failed fits; the e2e test
  checks the section is absent without a model (the fixture can't fit
  one) and exercises it when there is one.
- AGENTS.md note moved from School holidays to the catch model.
- Merged main; IMPORTER_VERSION already 9 there, no bump needed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Luen

Luen commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Reviewed against the real data (aggregates only) and current main. The maths is right:

  • Expected counts include the log(net-hours) offset.
  • Pearson residuals use the NB2 variance μ + μ²/θ, and P(0) falls back to Poisson when θ is infinite.
  • Calibration rates are summed sharks over summed net-hours, as Analysis: add catch-model calibration and residual diagnostic plots #125 asks.
  • Each diagnostic uses its fit's own rows, design matrix, counts and offsets, and the sample size matches fit.n.

On the real data the model describes catches reasonably well:

All sharks
Sharks, observed vs expected 2,686 vs 2,796
Zero-catch sets, observed vs expected 303 vs 298
Pearson residuals mean 0.02, mean square 1.09

The tenth of sets with the highest expected catch gets fewer sharks than the model expects: 1.51 vs 1.88 per net-hour for all sharks. Sicklefin lemons show the same pattern (1.57 vs 1.93).

I pushed fixes in 3432eec, 3bf20cb and ef54789, after merging main in 4fde0e0.

1. Page weight. The diagnostics sent every set as an object with eight unrounded fields, for all three models: about 500 kB of JSON on every /drivers load, embedded in the HTML too. The rest of the model is 12 kB. Now only what the charts draw goes out, rounded and in columns: 47 kB, or 12 kB with Brotli. The AGENTS.md note says to keep it that way (#76).

2. Nursery ids. The zero-catch chart and table showed raw ids ("haapiti", "valorie"). They now show nursery names, seasons are formatted, and shares are percentages in both the chart and the table.

3. Placement. The section opened the page, above the single-variable correlations and before the model it checks. It now follows the model section, in its own {#if md} block. The fallback of three empty groups is gone: with no model there's nothing to check.

4. Colours. Observed and expected used series slots 1 and 2, which mean blacktip and sicklefin lemon on this page. Observed now takes the model's colour, which is the species colour for the species models, and expected is muted.

5. Copy and axes

  • The intro is plainer, and subtitles say what each point is.
  • The season axis shows dates.
  • Residuals against the expected catch use a log x axis, because the low end was squashed. A note explains that the curved bands are sets with 0, 1, 2… sharks.
  • The 20-set rule is a single constant used by both the filter and the copy.

6. Tests

  • Unit: the Poisson limit for residuals, null fits, labelled and sorted groups, and percentage shares.
  • e2e: the synthetic fixture can't fit the model (40 sets in one season). The old test only reached the "unavailable" text, with every model empty. It now checks that the section is absent without a model, and runs the model picker and axe in both themes when there is one. I filed Fixture: enough synthetic sets for the catch model to fit in CI #153 for a fixture that fits, so CI covers these charts.

7. Smaller fixes

Checks after the fixes:

  • pnpm lint, pnpm check and pnpm test (98 passed).
  • Both opt-in real-data tests: the import, and the reconciliation of the diagnostics with the fitted samples.
  • pnpm build and the full Playwright suite (34 passed). After the last /drivers change, I re-ran the diagnostics and pages specs (32 passed).
  • In the browser on the real data: all three models in light and dark. The console shows no errors from the app's code.

@Luen
Luen merged commit 49a84a2 into main Nov 4, 2026
5 checks passed
@Luen
Luen deleted the codex/125-model-diagnostics branch November 4, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

analytics Charts, statistics and analyses enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analysis: add catch-model calibration and residual diagnostic plots

1 participant