Skip to content

fix(eval): judge 预检与解析对同一 env 结论不一致,空白 API key 通过预检、带空白的 provider 路由错端点 #306

Description

@wxrbyte

问题与复现

packages/shared/src/evaluation/judge-client.ts 里两个相邻函数对同一份配置用了不同的归一化:

  • resolveJudgeConfigStatus(:154-156)对 provider / model 做 .trim(),apiKey 则原样取用(既不 trim、也不判空白);
  • 它随后在 :166 行把未修剪的 env / explicit 交给 resolveJudgeConfig,而后者只做一次 .toLowerCase()(:178-180)。

于是预检给出的结论与真正落地的配置不是一回事。用 bun run 直接打印(修复前):

API key = "   "(shell / .env 里的空值行)
  preflight : requested=true missing=[] config=true
  resolved  : provider="anthropic" model="claude-sonnet-4" apiKey="   "
    → 预检放行,客户端拿着空 key 发请求,直到运行期才 JUDGE_API_ERROR

provider = " anthropic"(两侧带空白)
  preflight : requested=true missing=[] config=true
  resolved  : provider="openai-compatible" model="claude-sonnet-4" apiKey="k"
    → 预检说「anthropic 已配置」,实际按 openai-compatible 选端点与请求形状

API key = "sk-test\n"(从密钥文件读入、带尾随换行)
  preflight : requested=true missing=[] config=true
  resolved  : ... apiKey="sk-test\n"

第二行有可观测后果:scripts/eval/run.ts:815 会打印 Judge: openai-compatible/claude-sonnet-4,且 baseUrl 取 FINAGENT_JUDGE_BASE_URL ?? OPENAI_BASE_URL(:198)—— 把 Anthropic 的模型名发到 OpenAI 端点。第三行则会把换行一起带进鉴权头。

resolveJudgeConfigStatus 存在的意义正是 issue #113 要求的「显式失败,而不是静默降级为仅确定性评估」,但它对空白取值与两侧带空白的取值失效。注意:这里的 config === undefined 并不可达(修剪后非空必然意味着原值非空),所以预检永远"成功",问题只体现在取值本身。

预期与修复方向

provider / model / apiKey 三个字段在两处使用同一套归一化:先 trim(),再判断是否为空、再决定路由。baseUrl 在两个函数里都没有 trim,不存在不一致,本次不动。

复现证据

  • OS:Windows 11;Bun 1.4.2;上游 main 3a17eca6
  • 新增 packages/shared/src/evaluation/judge-client.test.ts(7 条用例):修复前 3 pass / 4 fail,修复后 7 pass / 0 fail
  • bun test packages/shared/src/evaluation --isolate:修复后 195 pass / 1 fail(唯一失败 langfuse backend > does not throw when Langfuse is down — agent path keeps a diagnostic 在 main 原始字节下同样失败,属既存问题)
  • bun run typecheck:core / i18n / shared / ui / electron 五工作区 exit 0
  • 补充事实:judge-client 此前完全没有测试文件,本次是首个

范围

只改 judge-client.ts 里三个字段的归一化。不改 resolveJudgeConfigStatus 的返回结构与 missing 文案、不改 requested 的语义、不改 resolveJudgeConfig 的返回形状与 header 解析、不改 baseUrl。

Activity

  1. wxrbyte commented on Oct 7, 2026

    @wxrbyte
    ContributorAuthor

    /claim

  2. github-actions commented on Oct 7, 2026

    @github-actions

    @wxrbyte 已认领该 Issue,并已自动设为 Assignee。提交 PR 时请在描述中写 Closes #306。

  3. added
    claimedClaimed by a contributor and currently in progress
    on Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

claimedClaimed by a contributor and currently in progress

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions