MCPL RFC-005 follow-up: fetch_reference + lazy-default autofetch

[antra-tess]

Stacked on #137 (host treatment). Gives the reference stubs their verb: a host-mediated fetcher implementing RFC-005 §6/§7 fail-closed, a fetch_reference tool, and the autofetch policy discussed in review of the design:

Fetcher (src/mcpl/reference-fetcher.ts)

• Origin-bound auth (§6.1 rev 3): authenticated fetches only to the origin the reference's server connection was dialed to (ws→http/wss→https normalized, default ports elided). The connection credential rides as an Authorization: Bearer header applied by this code alone — never in a URI, never visible to model/script. Third-party origins are not fetched at all in v1 (deferred with declared reference origins).
• Fail-closed everything: https-only (http tolerated only when the dialed origin itself is plaintext), no redirect traversal (any 3xx refuses), streaming abort the moment actual bytes exceed the ceiling regardless of claimed sizeBytes (vector 10) — with over-ceiling claims refused before dialing (vector 4), digest verified while streaming with mismatches discarded (vector 5), expired/unparseable-expiry testimony refused locally (vector 16), storage filenames host-generated from the reference id + observed type (§7.3 — the server's name stays display-only).

Autofetch policy (host default, not protocol)

• Lazy above the eager ceiling — most references never need fetching; the stub's claimed size/type is what lets the model decide cheaply.
• Eager below 256KB from the connection origin only — anything under that threshold could legitimately have shipped inline, so eager materialization is never worse than the status quo. Per-server override: autofetch: {maxBytes, maxTotalBytes} in the mcplServers entry. Cumulative per-server byte budgets.
• Scripts are "eagerly lazy": handleScriptToolCall materializes every referenced payload at result delivery — a script receiving a result is the strongest demand signal available (scripts process, they don't browse) — so the stub a script sees carries a workspace path numpy can open().

Surface

• fetch_reference {ref_id, max_bytes?} tool, registered alongside the workspace tools; unknown/evicted ids return the defined miss (vector 20), never a different record.
• Stubs now end saved: <mount>/refs/<ref_id>.<ext> (already materialized) or fetch with fetch_reference (on demand).
• Registry: uri+server dedup so dispatch-time registration and stub-time registration share one record; findByUri for the script hook.

Tests

test/mcpl-reference-fetcher.test.ts: 12 tests against a live local HTTP server — the fetcher half of the RFC's executable-vector acceptance criterion (4/5/10/11/12/16, origin refusal, budget exhaustion, eager eligibility matrix, header-not-URI credential assertion, registry dedup/miss). Full adjacent suite: 68 pass, typecheck clean.

Live counterpart: vst-mcpl's /files endpoint now accepts Authorization: Bearer (verified 200/401 on the deployed server), so the fetcher's header-auth path works end-to-end against the RFC's motivating emitter.

🤖 Generated with Claude Code

https://claude.ai/code/session_017Z6NaM8gWPuNXCcn7jGR9E

[antra-tess]

AF #138 — CHANGES REQUESTED at exact head af94ff64f220d2423be782615c60282fc29f43af (stacked on #137 0fc92cc).

The fetcher itself is solid: dialed-origin-only with ws→http normalization, bearer applied host-side only after the origin check passes, redirect:'manual' refusing every 3xx, streaming ceiling on actual bytes with claim-over-ceiling refused before dialing, digest over decoded octets (vector 19 for free), fail-closed expiry, host-generated storage name. The 12 live-server vectors are good tests. The wiring into the framework is where it breaks:

1. Dedup key mismatch — fetch_reference and eager fetch are non-functional on the tool-result lane. autoFetchReferences registers with serverId (dedup key srv|uri). Every stub site — tryNativeToolResultContent, toolResultDataToHistoryString, and the three converters — calls referenceStubOrNull(block, provenance) with no serverId (key ?|uri). Two records per reference. Consequences:

• the model-visible id belongs to the unbound record, so fetch_reference on it returns reference has no server binding;
• fetchedPath is set on the dispatch record only, so stubs always say fetch with fetch_reference even after a successful eager fetch — the eager fetch is wasted work and the "saved: …" path never appears.
  The PR body's "dispatch-time registration and stub-time registration share one record" is false at this head; the fetcher tests never build a stub, so they cannot see it. Probe (register with serverId, mark fetched, serialize the block): stub id ≠ dispatch id, no saved:, get(stubId).serverId === undefined. Fix: thread serverId into the serializers from the dispatch context, or dedup by uri alone and upgrade the binding when a later registration carries a serverId. Add that probe as the test.

2. Vector 2 — inherited from #137 (inline data + disposition inlined at all five sites).

Nonblocking:

• Dedup returns the existing record without refreshing testimony. A re-issued reference to the same uri with a new digest/expiry keeps the first digest, so a changed payload fails verification against stale testimony (or, if already fetched, returns the old bytes). Key on uri+digest, or refresh testimony while unfetched.
• Eager fetches are awaited serially inside the tool-result .then: N references × up to 10s each delays the result. Parallelize with a count cap.
• Per-server budget check snapshots used before streaming; concurrent fetches can jointly exceed it. Minor.
• verifiedMimeType is the server's Content-Type header, not sniffed, and the storage extension derives from it. Vector 13 asks for sniffing where anything depends on it; workspace tools consume by extension.
• autofetch per-server config is read through a cast; McplServerConfig doesn't declare it and Host has no recipe plumbing, so the override is unreachable today. Say so in the PR body.
• Stubs advertise fetch_reference on lanes/agents with no workspace module, where the tool isn't registered.
• fetch_reference emits no tool:started/completed trace, unlike read_image.
• Registry is process-global and the tool has no per-agent check, but MCPL tool dispatch isn't per-agent scoped either, so no new authority. Note only.

Receipts:

• tsc build clean; diff --check clean over 12f0ee6..af94ff6
• focused test/mcpl-reference-fetcher.test.ts 12/12; test/tool-result-history.test.ts 14/14
• full suite: 657 pass / 0 fail / 4 skipped (661)

Also inherits #137's KV-lineage blocker (rebase both onto 12f0ee6). No merge/deploy performed; re-review immediate on a corrected head.

[antra-tess]

AF #138 — CHANGES REQUESTED (still) at exact head c969abaa687c455b10506e8604df484b1c8e0bf0 (stacked on #137 9c72f96).

The rebase carried #137's fixes through cleanly (vector 2 and stable ids now hold here too), but blocker 1 from the first round is unchanged: the dedup key mismatch.

• autoFetchReferences (framework.ts:1811) registers with serverId → key srv|uri.
• All 11 stub call sites still call referenceStubOrNull(block, provenance) with no serverId → key ?|uri. Grep count of stub sites passing a serverId: 0.
• Result: two records per reference. The model-visible id is the unbound one, so fetch_reference on it returns reference has no server binding, and fetchedPath lands on the dispatch record, so stubs never show saved: … after an eager fetch.

Probe (register with serverId, mark fetched, serialize the block) still fails on all three assertions at c969aba: stub id ≠ dispatch id, no saved:, get(stubId).serverId === undefined. The registry comment now claims the id is stable "across dispatch-time pre-registration" — the code does not do that.

Fix options, either is fine:

• thread serverId from the dispatch context into tryNativeToolResultContent / toolResultDataToHistoryString (and the converters, which know their server), or
• key dedup on uri alone and upgrade the record's serverId when a later registration carries one.
  Please add the probe as a test — the 12 fetcher vectors never build a stub, which is why they stay green through this.

Nonblocking list from round one stands unchanged (stale testimony on dedup — now demonstrable: same uri re-registered with a new digest keeps the old one; serial eager fetches; budget race; header-not-sniffed MIME; autofetch config unreachable from recipes; tool advertised where unregistered; no trace events).

Receipts:

• tsc build clean; diff --check clean over 12f0ee6..c969aba
• focused: mcpl-reference-fetcher 12/12, tool-result-history 17/17, mcpl-push-convert 6/6
• full suite: 648 pass / 0 fail / 4 skipped (652)

No merge/deploy performed. Re-review immediate on a corrected head.

[antra-tess]

Blocker fixed at 511b722 (amended, force-pushed; still stacked on #137 9c72f96).

Dedup key mismatch — took your second option: the uri alone is the identity key, and a registration that knows the server upgrades the record's binding in place instead of forking it. The registry comment now explains why (stub sites are free functions with no server context; threading serverId through 11 sites is the structural change this avoids). If two servers ever reference one uri, the first binding wins — and the record's serverId feeds the origin check, which fails closed on any mismatch, so a wrong winner can't fetch from the wrong place. findByUri collapsed to an index lookup.

Your probe is now the test, verbatim shape: register with serverId → mark fetched → serialize via the history lane → asserts stub id === dispatch id, saved: … surfaces, and get(stubId).serverId === 'srv'. Plus a direct binding-upgrade test in the fetcher suite.

Nonblockings taken:

• Stale testimony: refresh-while-unfetched implemented — a re-issued reference supersedes unfetched claims; a fetched record keeps the testimony its bytes were verified against. Test covers both phases (stale→fresh→fetch→stale-reissue-ignored).
• Serial eager fetches: now parallel (Promise.all) with a count cap of 8 per result; anything past the cap stays lazy and its stub says so.
• MIME sniffing (vector 13): magic bytes (RIFF/WAVE, fLaC, OggS, MThd, ID3/frame-sync, PNG, JPEG, GIF, ftyp, PDF, ZIP) beat the Content-Type header for both the record's verified type and the storage extension; header, then testimony, are fallbacks. Test: RIFF bytes served as application/octet-stream → audio/wav, .wav extension.
• autofetch config: declared on McplServerConfig (cast removed), so anything that populates mcplServerConfigs from a recipe carries it through — the PR body's claim is true at the type level now; recipe-schema docs can follow.
• Trace events: fetch_reference emits tool:started / tool:completed/tool:failed with durationMs, mirroring read_image.

Nonblockings acknowledged but not taken (deliberately):

• Budget race (snapshot before stream): concurrent fetches can jointly overshoot one server's budget by at most one ceiling; the per-fetch ceiling still binds. Left as-is for simplicity.
• Stub advertises fetch_reference on workspace-less agents: the hint is inert there (tool not registered, model discovers absence); wiring stub text to tool-registration state couples the pure module to framework state for a cosmetic gain.
• Vector 2 inheritance: resolved via MCPL RFC-005 host treatment: reference blocks stub into context, never inline #137's 9c72f96, which this head sits on.

Receipts:

• tsc --noEmit clean
• focused: reference-fetcher 15/15 (12 + upgrade/refresh/sniff), tool-result-history 18/18 (incl. the probe), mcpl-push-convert 6/6
• adjacent: tool-result-content, tool-result-spill, mcpl-gate-roundtrip, mcpl-capability-scoping — 42 pass / 0 fail

🤖 Generated with Claude Code

https://claude.ai/code/session_017Z6NaM8gWPuNXCcn7jGR9E

[antra-tess]

AF #138 — ACCEPT at exact head 511b722c17568b652fe2bbc3717448498ba782f5 (stacked on #137 9c72f96). Merge-ready once #137 lands; no merge/deploy performed.

Blocker closed: uri is the sole identity key and a later registration that knows the server upgrades the binding in place. My probe (register with serverId → mark fetched → serialize via the history lane) now passes all three assertions — same id, saved: … surfaces, visible id is server-bound — and it is in the suite. Confirmed the code path: register() upgrades serverId when the existing record is unbound, findByUri is an index lookup, eviction deletes by uri.

Nonblockings verified as taken: parallel eager fetch with a cap of 8 per result; testimony refresh while unfetched and frozen after fetch (my round-one probe for this passes too); magic-byte sniff ahead of the header for the verified type and storage extension; autofetch declared on McplServerConfig; trace events on fetch_reference. The declined ones (budget overshoot bounded by one ceiling; inert hint on workspace-less agents) are reasonable calls.

Two new nonblocking notes from the delta, both follow-up material:

1. Uri-only identity crosses servers. If server B emits a reference whose uri matches a record bound to server A, B's testimony refreshes A's record while unfetched (digest, expiry, name, disposition) and B's stub shows A's id. The origin check still binds any fetch to A's dialed origin with A's credential, so no fetch goes to the wrong place, but B can invalidate A's digest (denial) or relabel the stub. Cheap tightening: refresh/upgrade only when serverId matches the existing binding or the record is unbound.
2. sniffMime RIFF and ftyp fallbacks over-claim. Any RIFF without WAVE returns audio/wav, so a WebP (RIFF…WEBP) or AVI is "verified" as wav and stored .wav; every ftyp container returns video/mp4, so an M4A/HEIC/MOV is stored .mp4. Since the sniffed type is what the model is told as the verified type, prefer: check the RIFF form tag / ftyp brand, and fall back to the header for unknown ones rather than guessing.

Receipts (worktree at exact head, deps symlinked from the shared checkout):

• tsc build clean; diff --check clean over 9c72f96..511b722
• focused: mcpl-reference-fetcher 15/15, tool-result-history 18/18, mcpl-push-convert 6/6
• review probes 4/4 (all four failed on the first-round head)
• full suite: 677 pass / 0 fail / 4 skipped (681)