test: cover judge.mjs - #16
Conversation
judge.mjs is the L2 precision gate and runs on every review (`precision-filter` defaults to "true"), but had no tests. These deliberately do not test the prompt or the judge's taste -- that is model quality, not a contract. They test every place this script decides to DROP a finding, because each one is a silent way to lose a real defect. 27 tests: - threshold parsing: "0.8oops", "0x1", "1e-1" and friends are rejected with exit 2 rather than silently truncated the way parseFloat would; [0,1] bounds enforced; boundaries accepted. - schema drift in the judge's own response, which is where plain JS truthiness is actively dangerous: the STRING "false" must still drop, and a confidence of `true` or `[1]` must NOT coerce to 1 and sail past every threshold. Out-of-range confidence drops rather than clamping. - fail-open: a finding the judge never classified is kept even at threshold 1, so a judge blind spot cannot become a silent drop. - malformed groups: an out-of-range representative_id falls back to a valid member, and a group with no resolvable representative is skipped loudly instead of taking real findings down with it. - transport: HTTP error and unparseable envelope exit 1 without writing a filtered result; a fenced ```json reply is unwrapped. - request shape: model, temperature 0, bearer auth. The LLM is a local mock and HOME is redirected to an empty dir so the ~/.opencodereview/config.json fallback cannot pick up a real local config. Async spawn (not spawnSync) because the mock runs in the test process. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b028bb280b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ...process.env, | ||
| HOME: home, | ||
| USERPROFILE: home, | ||
| OCR_LLM_URL: "", | ||
| OCR_LLM_TOKEN: "", | ||
| JUDGE_MODEL: "", |
There was a problem hiding this comment.
When the test runner's environment defines OCR_LLM_AUTH_HEADER (for example, a developer using the supported x-api-key configuration), spawnJudge inherits it while overriding only the URL, token, and model. The child therefore sends Bearer test-token under that custom header, and the request-shape test incorrectly fails because it asserts headers.authorization; this is reproducible with OCR_LLM_AUTH_HEADER=x-api-key node --test scripts/judge.test.mjs. Clear or explicitly set this variable in the helper so the suite remains hermetic.
Useful? React with 👍 / 👎.
1 participant
scripts/judge.mjsis the L2 precision gate and runs on every review (precision-filterdefaults to"true"), but had no tests. This adds 27.Scope
These deliberately do not test the prompt or the judge's taste — that is model quality, not a contract, and pinning it would just make the tests brittle. What they do test is every place this script decides to drop a finding, because each one is a silent way to lose a real defect.
What is pinned
Threshold parsing -
--thresholdgates every keep/drop, so a silently coerced value moves the gate without telling anyone."0.8oops","0x1","1e-1"and friends exit 2 instead of becoming0.8/0the wayparseFloatwould.[0,1]bounds are enforced; the boundaries themselves are accepted.Schema drift in the judge's response - this is where plain JS truthiness is actively dangerous:
keep: "false"confidence: trueNumber(true)= 1 -> clears any thresholdconfidence: [1]Number([1])= 1 -> clears any thresholdconfidence: 2confidence: "0.95"Fail-open - a finding the judge never classified into any group is kept even at threshold 1, so a judge blind spot cannot become a silent drop.
Malformed groups - an out-of-range
representative_idfalls back to a validmember_id(the coverage pass has already marked those members handled, so the group vanishing would take real findings with it). A group with no resolvable representative is skipped loudly.Transport - HTTP error and unparseable envelope exit 1 without writing a filtered result; a fenced
```jsonreply is unwrapped.Request shape - model,
temperature: 0(a precision gate has to be deterministic), bearer auth.Test hygiene
The LLM is a local mock, and
HOMEis redirected to an empty directory so the~/.opencodereview/config.jsonfallback cannot pick up a real local harness config and make the tests lie. Asyncspawnrather thanspawnSync, because the mock runs in the test process and the event loop has to stay live to answer the child.🤖 Generated with Claude Code