Skip to content

test: characterize plausible citation chip target mechanisms - #226

Draft
atlas-coding-agent wants to merge 2 commits into
devhims:mainfrom
atlas-coding-agent:fix/citation-target-consistency
Draft

atlas-coding-agent wants to merge 2 commits into
devhims:mainfrom
atlas-coding-agent:fix/citation-target-consistency

Conversation

@atlas-coding-agent

@atlas-coding-agent atlas-coding-agent commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Readers click a citation chip and land up to a minute before the moment the answer describes. The answer text sometimes names a third time. This draft adds tests that characterize plausible mechanisms for that. It does not establish the live root cause, and it does not fix it. It changes no product code.

Reported by the reporter on Oct 24 2026, 06:32 to 06:41 IST:

Session / turn Expected Actual
chess 426fcd5f-11a8-4413-88bb-321f4112cd88, T1 26f5171e-dff6-4311-bd83-fef6d3105498 h3 at 2:25, Qb7 at 2:30 chip 1:54, an unrelated queen sacrifice
T2 8a21338e-6f16-438e-9e44-935c4c3118c0 Qh2+ quote at 1:44 prose says 1:54, chip and href t=58s
T3 cb2e38b0-bc93-4e39-aa2c-423a006d5096 closing advice at 2:42 to 2:47 quote back at 1:44, advice still 1:54
watercolor 6438cb12-c5c8-439e-b111-d604f02187a7, T3 c52a03be-d51d-4899-832a-fcddb48e6dcb 6:35 to 6:38 prose link 6:33, chips 5:38

Plausible mechanisms (not confirmed for these sessions)

The tests show two code paths that move a chip off the claimed moment. The chip label, href and stored citation still agree on every path tested.

The reported wrong times (0:58, 1:54, 5:38) are consistent with window starts. I did not inspect the stored citation IDs of these sessions, so this is timing arithmetic, not evidence of the cause. Nothing checks the times the model writes in prose. The model also never sees start times for analysis references. That makes the prose mismatch a possible model grounding issue, also unconfirmed.

What changed

Tests only, on synthetic captions at the reported times. They cover:

  • the same target across prose, tables and hrefs
  • window-start labels
  • nested merges, including an old stored marker mixed with a new alias
  • reload serialization
  • raw reference leakage

Tests

CI Verify passed on this PR. Local runs are blocked because the sandbox registry returns 403 for zod@4.4.3 and zod-to-json-schema@3.25.2. Base is upstream 3d11c12.

Decision pending

Preserving an exact caption when a broader cited window covers it needs owner approval. That approval has not been given. Model-input time metadata and range-format chips are not proposed here.

@vercel

vercel Bot commented Oct 24, 2026

Copy link
Copy Markdown

@kiro-agent is attempting to deploy a commit to the devhims' projects Team on Vercel.

A member of the Team first needs to authorize it.

@atlas-coding-agent atlas-coding-agent changed the title test: pin citation chip targets for window and nested excerpts test: characterize plausible citation chip target mechanisms Oct 24, 2026

devhims commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Reviewed head daf7d92.

Verdict: CHANGES NEEDED / validation blocked for overall readiness. The tests-only change is acceptable in its narrow scope; I found no blocking code defect. Overall approval is withheld because the Vercel authorization status is failed and I could not verify whether it is a required merge check. Confirm the required-check policy before treating this PR as merge-ready; this review does not authorize deployment.

Re-review: the additional commit changes explanatory comments and a test name only. The description and platform/test/citation-target-consistency.test.ts:8–11 now correctly characterize plausible mechanisms instead of claiming a proven historical root cause. That wording concern is resolved.

Coverage reviewed:

  • platform/test/citation-target-consistency.test.ts:57–115 covers exact-caption labels, whole-window fallback, nested grouping, mixed old stored markers/new aliases, serialization and raw-reference suppression.
  • web/lib/agent-citations.test.ts:27–43 checks matching timestamp hrefs in prose and table cells.
  • No runtime product code changes, new permissions, or security-sensitive behavior were introduced.

Verification:

  • Exact-head Verify workflow passed: https://github.com/devhims/video2ctx/actions/runs/40067930018
  • Independently reran the updated platform citation suite: 6/6 passed. The unchanged web citation suite previously passed 4/4 independently. Focused execution used connector-fetched source files with cached dependencies, not a clean full-repository installation.
  • Rechecked the exact head before posting. No duplicate review was present.

Limits and user impact: these are synthetic characterization tests, not a citation fix. They do not prove the causal path of chess session 426fcd5f-11a8-4413-88bb-321f4112cd88 (runs 26f5171e-dff6-4311-bd83-fef6d3105498, 8a21338e-6f16-438e-9e44-935c4c3118c0, cb2e38b0-bc93-4e39-aa2c-423a006d5096) or watercolor session 6438cb12-c5c8-439e-b111-d604f02187a7 (run c52a03be-d51d-4899-832a-fcddb48e6dcb). Stored live citation IDs were not inspected here. JSON serialization is not a browser/database reload test. No fresh live dashboard tests were run. Existing window-start and nested-merge behavior remains unchanged, so do not close the live issue as fixed by this PR. Citation-policy implementation remains subject to the owner's approval.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants