Skip to content

fix(lab): read per-model overrides in the report the way the runtime reads them - #2077

Open
ntdatt812 wants to merge 1 commit into
lidge-jun:devfrom
ntdatt812:fix/compat-behavior-model-overrides
Open

fix(lab): read per-model overrides in the report the way the runtime reads them#2077
ntdatt812 wants to merge 1 commit into
lidge-jun:devfrom
ntdatt812:fix/compat-behavior-model-overrides

Conversation

@ntdatt812

@ntdatt812 ntdatt812 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

#2059 fixed the list-shaped half of the behavior report. The per-model override maps are the other half, and they had the same shape of bug plus one more.

modelValue was a bare index:

function modelValue<T>(map: Record<string, T> | undefined, modelId: string): T | undefined {
  return map?.[modelId];
}

The runtime reads these maps through modelRecordValue (src/reasoning-effort.ts:73), which checks own properties, then the pre-colon family, then a case-folded key. Nine of the ten maps the report reads go through it at runtime — modelContextWindows, modelMaxInputTokens, modelMaxOutputTokens, modelInputModalities, modelReasoningEfforts, modelDefaultReasoningEfforts, modelReasoningEffortMap, modelSupportsReasoningSummaries, modelReasoningSummaryDelivery. The tenth, modelPreferHostedTools, is read in openai-responses.ts with its own explicit hasOwnProperty guard.

The three disagreements

1. Pre-colon family. ollama-cloud serves gpt-oss:120b. With modelMaxOutputTokens: {"gpt-oss": 1234}:

value
wire the adapter builds max_tokens: 1234
report limits.maxOutputTokens null

2. Case folding. A differently-cased key resolves at runtime and did not in the report.

3. Prototype chain — this one is not just wrong data.

Model ids are operator-controlled, so one can be constructor or toString. The bare index then returned an Object.prototype function:

report limits.contextWindow for 'constructor' = function Object() { [native code] }

jcsStringify rejects a function, so buildBehaviorFingerprintV1 threw unsupported value type function, and resolvePassiveRouteSubjectId swallows the throw — the subject silently never links and Lab loses that traffic with no diagnostic. The linker's contract states an implementation is "synchronous, free of side effects with respect to the request, and non-throwing"; the try/catch is described there as belonging to the mechanism so the guarantee is not restated by callers — a backstop, not a licence. openai-responses.ts:996-1001 already guards modelPreferHostedTools against exactly this, and says why in a comment.

Change

modelValue delegates to modelRecordValue. One helper, so the report cannot disagree with the runtime on any of the nine at once.

Blast radius, measured

Same config, before and after, first 16 hex of the behavior fingerprint:

model dev this branch
gpt-oss:120b 54154e19bd2c8ee4 5c992edff35bd7e9 was missing the 1234 override
gpt-oss 5c992edff35bd7e9 180a84b2d837619d was missing the case-folded 55555
glm-5.3 54154e19bd2c8ee4 54154e19bd2c8ee4 unchanged
constructor THROW 54154e19bd2c8ee4 now computable

Only subjects whose overrides were being missed move, so resolverVersion stays at 2 for the same reason as #2059 — say the word if you would rather draw a generation boundary.

Tests

Extends tests/routing-compatibility-model-matching.test.ts from #2059, same pattern: assert the wire the adapter really builds, then hold the report to it. Prototype-shaped ids get their own cases, including one asserting the fingerprint stays computable and that two such ids hash alike.

Seven of the nine new cases fail on current dev (9 pass / 7 fail); the two that pass there are the ground-truth wire assertion and the control.

Verification

On the exact head of this branch:

bun run typecheck                                     clean
bun test tests/routing-compatibility-model-matching.test.ts
                                                      16 pass / 0 fail
  same file with src reverted to dev:                  9 pass / 7 fail
bun scripts/test.ts <22 files touching compatibility, lab,
  passive-route-linker, fastwire, reasoning-effort>
                                                      696 pass / 6 skip / 0 fail

Not the full suite: this is a Windows machine and bun run test panics partway through on Bun 1.3.14 (index out of bounds: index 0, len 0), so a result from it would be a truncated log. The batch ran through scripts/test.ts so each file keeps its isolated home.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

  • All CI tests are green on my local testing.

  • I pushed my PR to the latest dev commit.

  • I resolved all correct Codex and CodeRabbit findings.

  • My PR is ready for review.

Summary by CodeRabbit

  • Bug Fixes

    • Improved per-model configuration matching for model names and families, including case-insensitive matches.
    • Prevented inherited or unsupported values from affecting behavior reporting and compatibility fingerprints.
  • Tests

    • Added coverage for model-family matching, case variations, unrelated models, prototype-shaped model IDs, and fingerprint generation.

…reads them

lidge-jun#2059 fixed the list-shaped half of the behavior report. The per-model maps
are the other half, and they had the same shape of bug plus one more.

`modelValue` was a bare index, `map?.[modelId]`. The runtime reads these maps
through `modelRecordValue`, which checks own properties, then the pre-colon
family, then a case-folded key. So the report disagreed three ways:

- ollama-cloud serves `gpt-oss:120b`. With `modelMaxOutputTokens: {"gpt-oss":
  1234}` the adapter puts `max_tokens: 1234` on the wire, while the report said
  `limits.maxOutputTokens: null`.
- a differently-cased key resolved at runtime and not in the report.
- the index walked the prototype chain. Model ids are operator-controlled, so
  one can be `constructor` or `toString`, and the row then held an
  Object.prototype *function*.

That last one is not merely wrong data. `jcsStringify` rejects a function, so
`buildBehaviorFingerprintV1` threw `unsupported value type function`, and
`resolvePassiveRouteSubjectId` swallows the throw -- the subject silently never
links, and Lab loses that traffic with no diagnostic. The linker's own contract
says a registered implementation is "synchronous, free of side effects with
respect to the request, and non-throwing"; the try/catch is the backstop, not a
licence. openai-responses.ts already guards `modelPreferHostedTools` against
exactly this, with a comment saying why.

Nine of the ten maps the report reads are read through `modelRecordValue` at
runtime; delegating to it makes the report agree with all of them at once.

Fingerprints, same config, before and after:

    gpt-oss:120b   54154e19bd2c8ee4 -> 5c992edff35bd7e9   (was missing 1234)
    gpt-oss        5c992edff35bd7e9 -> 180a84b2d837619d   (was missing 55555)
    glm-5.3        54154e19bd2c8ee4 -> 54154e19bd2c8ee4   (unchanged)
    constructor    THROW            -> 54154e19bd2c8ee4   (now computable)

Only subjects whose overrides were being missed move; `resolverVersion` stays
at 2 for the reason given in lidge-jun#2059.

Tests extend the file lidge-jun#2059 added: the adapter's wire is asserted first, then
the report is held to it, and the prototype-shaped ids get their own cases
including one that the fingerprint stays computable. Seven of the nine fail on
current dev.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added bug Something isn't working review-ready labels Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

✅ READY

  • all PR quality gates passed; the review readiness checklist is complete.

Review readiness checklist

  • ✅ All CI tests are green on my local testing.
  • ✅ I pushed my PR to the latest dev commit.
  • ✅ I resolved all correct Codex and CodeRabbit findings.
  • ✅ My PR is ready for review.

4/4 boxes ticked.

This pull request is already Ready for Review.
The review-ready label marks this PR as ready; review automation runs independently.
Maintainers: @lidge-jun @Ingwannu @Wibias

@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: cf496670-0e6b-498e-8b8f-6ca688089ac6

📥 Commits

Reviewing files that changed from the base of the PR and between abaa75a and f4213f2.

📒 Files selected for processing (2)
  • src/routing/compatibility/behavior.ts
  • tests/routing-compatibility-model-matching.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The change makes behavior reporting use shared per-model override resolution. It adds coverage for family matching, case-insensitive keys, unrelated models, prototype-shaped IDs, and equivalent behavior fingerprints.

Changes

Model override resolution

Layer / File(s) Summary
Shared model override lookup
src/routing/compatibility/behavior.ts
modelValue now uses modelRecordValue for own-property, model-family, and case-insensitive override resolution.
Override and fingerprint validation
tests/routing-compatibility-model-matching.test.ts
Tests cover tagged models, case-folded keys, unrelated models, prototype-shaped IDs, and fingerprint computation without inherited function values.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to f4213

The change aligns report lookup behavior with runtime overrides and adds focused regression coverage; no actionable merge-blocking risk remains beyond normal checks and review.

Possibly related PRs

Suggested reviewers: wibias, ingwannu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: aligning report lookup of per-model overrides with runtime behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working review-ready

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant