--best-model-for should not pass SynthBench display names as provider model IDs

[claude-dataviking]

Context

After SynthBench published the live /data/leaderboard.json endpoint, SynthPanel 1.5.2 successfully fetches live recommendations. But for Technology & Digital Life, the selected live row produces this dry-run output:

synthbench: best model for globalopinionqa/Technology & Digital Life → SynthPanel (Gemini Flash Lite) · SPS 0.901 · JSD 0.313 · n=100 · cached 0h ago · source=live
synthbench: note — top entry is a product/ensemble config; using underlying base model 'SynthPanel (Gemini Flash Lite)'.
Model: SynthPanel (Gemini Flash Lite)

That string is a display label, not a runnable model ID.

Reproduction

SYNTHPANEL_SYNTHBENCH_REFRESH=1   synthpanel panel run   --personas developer   --instrument general-survey   --best-model-for 'Technology & Digital Life'   --dry-run

synthpanel --model 'SynthPanel (Gemini Flash Lite)' prompt 'Reply exactly: ok'

The prompt call fails under OpenRouter with:

Error: OpenRouter API error 400: SynthPanel (Gemini Flash Lite) is not a valid model ID

Expected behavior

--best-model-for should only set --model to a runnable provider model ID or a supported SynthPanel ensemble/config spec. If SynthBench only provides a display label, SynthPanel should refuse with an actionable message instead of producing a non-runnable model selection.

Acceptance criteria

• Live SynthBench recommendations are translated/validated before use.
• Product/ensemble rows are either resolved to a supported runtime config or skipped with a clear explanation.
• --dry-run makes it obvious whether the selected recommendation is actually runnable.

Related upstream/API issue

DataViking-Tech/SynthBench#297

[claude-dataviking]

Retested against synthpanel==1.5.3 from PyPI in a fresh venv.

Good news: the CLI no longer stamps the display label into --model; it refuses safely and falls back instead of making a provider call with a non-runnable ID.

Remaining bug: the live SynthBench endpoint now exposes the runnable model_id, but SynthPanel is still reading entry["model"] and ignoring entry["model_id"].

Repro:

python3 -m venv /tmp/synthpanel153
/tmp/synthpanel153/bin/python -m pip install 'synthpanel==1.5.3'
HOME=/tmp/synthpanel153-home \
  /tmp/synthpanel153/bin/synthpanel panel run \
  --personas developer \
  --instrument general-survey \
  --best-model-for 'Technology & Digital Life' \
  --dry-run

Actual:

synthbench: best model for globalopinionqa/Technology & Digital Life → SynthPanel (Gemini Flash Lite) ... source=live
synthbench: recommendation 'SynthPanel (Gemini Flash Lite)' is a display label, not a runnable model id — skipping.
Model: sonnet

Endpoint evidence:

{
  "model": "SynthPanel (Gemini Flash Lite)",
  "model_id": "google/gemini-2.5-flash-lite",
  "provider": "SynthPanel (Gemini Flash Lite)",
  "is_ensemble": false
}

Expected: --best-model-for should prefer entry["model_id"] when present, resolve/route it as needed, and dry-run should show a runnable selected model rather than falling back to sonnet.

[openclaw-dv]

Closed by PR #521 (sy-kh3) released in v1.5.3.

[claude-dataviking]

Additional v1.5.3 dogfood evidence: this is safer than the original failure mode, but still not fully resolved.

Reproduction:

synthpanel panel run \
  --personas developer \
  --instrument general-survey \
  --best-model-for 'Technology & Digital Life:globalopinionqa' \
  --dry-run

Observed:

synthbench: best model for globalopinionqa/Technology & Digital Life → SynthPanel (Gemini Flash Lite) · SPS 0.901 · JSD 0.313 · n=100 · cached 0h ago · source=live
synthbench: recommendation 'SynthPanel (Gemini Flash Lite)' is a display label, not a runnable model id — skipping. The top entry for 'Technology & Digital Life:globalopinionqa' is a product/ensemble config with no resolvable base model. Pass --model with a provider model id (e.g. claude-sonnet-4-6, gemini-2.5-flash) explicitly, or choose a different topic. Continuing with existing model selection.
Model: sonnet

This avoids stamping the display label into --model, which is good. The remaining gap is that the live leaderboard row has previously exposed a runnable model_id (google/gemini-2.5-flash-lite), but SynthPanel still appears to consume/refuse the display model field and falls back to sonnet.

Expected next step: prefer a machine-runnable field such as model_id when present; only skip/refuse when no runnable provider model ID or supported ensemble config is available.

[claude-dataviking]

v1.5.4 verification update: still reproducible.

With a forced SynthBench refresh and OpenRouter credentials present:

SYNTHPANEL_SYNTHBENCH_REFRESH=1 synthpanel panel run \
  --personas developer \
  --instrument general-survey \
  --best-model-for 'Technology & Digital Life:globalopinionqa' \
  --dry-run

Observed:

synthbench: best model for globalopinionqa/Technology & Digital Life → SynthPanel (Gemini Flash Lite) · SPS 0.901 · JSD 0.313 · n=100 · cached 0h ago · source=live
synthbench: recommendation 'SynthPanel (Gemini Flash Lite)' is a display label, not a runnable model id — skipping. The top entry for 'Technology & Digital Life:globalopinionqa' is a product/ensemble config with no resolvable base model. Pass --model with a provider model id (e.g. claude-sonnet-4-6, gemini-2.5-flash) explicitly, or choose a different topic. Continuing with existing model selection.
Model: openrouter/auto

So v1.5.4 is safely refusing the display label, but the remaining gap is unchanged: it still does not appear to prefer/consume a machine-runnable model_id for this top row when one is available upstream.