Repository navigation
feat: analyze frames and storyboards with Kimi K3 - #219
Conversation
GLM 5.3 Flash misread a chess position in a production session (a black queen as an empty square, a rook or a king) and could not read most small on-screen text in storyboard tiles. Through the production analysts, Kimi K3 scored 99% on single frames and 86% on storyboard sheets, against 86% and 67% for GLM, with no failed calls. AGENT_VISUAL_MODEL selects the visual analyst model on Fireworks, set to kimi-k3; an empty value keeps GLM. Kimi K3 uses native low reasoning with the Priority tier and is priced at its Priority rates. It fails over to DeepSeek V4.1 Flash with its own run-level switch, so a visual model stall never moves other roles off GLM and a GLM outage leaves visual analysis on Kimi. The run deadline and cost check stay shared. Regenerating worker types also picks up the Polar product ID already committed in wrangler.jsonc.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Review of b9f99b5Verdict: CHANGES NEEDED. One P2 recovery/isolation defect remains; pricing and ordinary routing look consistent. P2 — Persist and restore the visual fallback separatelyplatform/src/agents/model.ts:48–54 initializes visual fallback to false in a WeakMap and forwards diagnostics to the shared callback. However, agent-runtime-do.ts:1763–1770 still persists every fallback as Reproduction with mocked provider responses and the runtime's exact persistence/restore predicate: Kimi returns 503 → visual DeepSeek succeeds → original shared fallback remains false → reconstruct the state from the emitted visual fallback event → shared fallback becomes true, a recreated visual model starts Kimi again, and recreated Agent Core starts DeepSeek. The memory updater also reconstructs state this way, so a visual outage moves that unrelated GLM role even without a restart. Impact: resumed runs lose the visual outage switch, retry the unhealthy primary, and incorrectly move otherwise healthy text roles to the backup. Persist/restore distinct visual and general fallback domains while retaining shared deadline, cost guard and diagnostics. Add recovery and memory-updater isolation coverage; the new same-object isolation tests do not exercise this boundary. P3 — Update the remaining GLM visual-model documentationreference/engineering/AGENT_PATHS_AND_LIMITS.md:283 still says storyboard/frame analysis starts with GLM, contradicting the Kimi configuration and updated opening. Nonblocking, but misleading during operations/debugging. Independent verification
No paid inference or production test was run for this review. Accuracy/latency/average-cost benchmark numbers remain author-reported, not independently reproduced. No code was implemented, pushed, merged or deployed. |
The separate visual fallback switch lived only in memory, while the runtime records every fallback as a model.fallback event and rebuilds one run-wide flag from it on resume and in the memory updater. A Kimi-only fallback was therefore restored as a run-wide one, moving healthy text roles to DeepSeek and retrying the stalled visual model. Kimi K3 now uses the run's single switch, like GLM: a stall of either moves the rest of the run to DeepSeek, and a resumed run restores that consistently. This removes the separate state and its tests and corrects the remaining GLM visual-model documentation.
Re-review of 23cd69aVerdict: APPROVE. No material findings remain under the revised, explicitly documented shared-fallback policy. The previous P2 is resolved by removing the separate WeakMap switch rather than persisting independent domains. Kimi and GLM now intentionally share one run-wide switch: a qualifying failure moves subsequent calls to DeepSeek, and recovered state agrees. My prior mocked-provider Kimi-503/recovery probe now selects DeepSeek consistently for resumed visual, core and memory roles. The previous P3 stale GLM visual-model paragraph is corrected too. Independent verification at this head:
This approves code correctness against the new shared-switch contract, not the former isolation behavior. No paid inference, live outage or production visual-accuracy retest was performed. Benchmark accuracy/latency/cost figures remain author-reported. No merge or deployment performed. |
Problem / Motivation
Session
8eb3885casked which piece stood on h2 in two chess frames. The visual analyst (GLM 5.3 Flash) reported h2 as empty, and on reanalysis as a black king next to the white king, an illegal position. The frames show a black queen on h2 at 1:48 and the white king on h2 at 2:04. Replays through the production analyst reproduced the misreading in every run, and neither prompt wording, upscaling nor one-frame requests changed it. Other vision models on the same frames and prompts read it correctly.What changed
Frame and storyboard analysis now use Kimi K3 on Fireworks, selected by a new
AGENT_VISUAL_MODELsetting. An empty value keeps GLM, so rolling back is a configuration change.Benchmark
All five deployed candidates ran through the unchanged production analysts with production provider options, three runs per task. Single frames covered 8 tasks: chess, on-screen text, name captions, a frame with no caption, subtitles and objects. Storyboards covered 23 tasks over 11 sheets of 6 videos: news, chess, a product launch, an Excel tutorial, cooking and a slide talk, including two-sheet requests.
GLM failed every run on reading small slide text in storyboard tiles ("AI OS", a product name, a "Reaction" label) and invented a caption name. Kimi K3 still misses some cases: no model detected a flipped chess board, and Kimi missed one cooking step that DeepSeek found. Kimi K3 Fast matches its accuracy but costs more and has no Priority tier.
Tests
Platform build, generated skill and documentation checks passed. Platform tests: 2,118 passed, 59 skipped. Workers integration: 417 passed. New tests cover visual model routing, the empty-value rollback, unknown model rejection, Kimi pricing, and Kimi failover through the shared switch, including a run restored from its fallback event. A live frame analysis through production
createAgentModelreached Kimi K3 with these settings and read the session's frames correctly in 4.7 seconds.Compatibility
No migration.
AGENT_VISUAL_MODELis a new plain variable inwrangler.jsonc. Regenerating Worker types also records the Polar product ID already committed inwrangler.jsonc. Frame analysis costs about $0.01 per call instead of $0.0004, a few cents per run that analyzes images, inside the existing per-run model budget.