feat(facts): @derive/facts — the read contract, the rename, and two parser bugs it surfaced - #587
Merged
Conversation
an1va
force-pushed
the
feat/slot-read-contract
branch
from
July 31, 2026 00:12
684e5ef to
1fb5a08
Compare
Previewhttps://derive-pr-587.derive-to.workers.dev Deployed from It shares production's database — sign in with your real account, and treat anything you change here as changed for real. It has no routes, no cron, no queue consumer and no OG renderer, so it cannot serve derive.to, run scheduled work, or write images onto real artifacts. Unlike production it serves artifact HTML on its own origin (that is what makes frame-side changes visible here). Storage is still sandboxed away, but untrusted HTML and the sign-in form share a hostname — treat this URL as you would any link: don't type a password into it because a page asked you to. |
Phase 3, prototyped on top of #580 rather than shipped. Nothing here is published as a standard, and it should not be until somebody outside the team emits a slot — publishing a spec for a format nothing emits is how CSVW and W3C Web Annotation died, and both were better specified than the things that beat them. Writing the parser is what showed the parser is the least valuable part: a few hundred lines, no incumbent, nothing an implementer would get wrong. SPEC.md is the actual contribution — the read contract, where every item is a decision an independent host must make and the wrong choice fails silently. It is written from THIS host's live defects: absence is not a zero, cross-record rows join to the current version, aggregates disclose so gate before counting, never report what was filtered, a caller-specific response is not shared-cacheable, and a sandboxed page needs CORS to read its own data (and still cannot if the record is gated). Extraction is a MOVE, not a copy: core re-exports the package, so every consumer is unchanged and there is one implementation. A second copy is the failure this codebase has now shipped three times. lint:slots-portable keeps the claim true — zero imports, zero runtime deps — and fails loudly if it matches no source at all. Verified against a known-bad import before wiring in, and the package was bundled standalone (8.8KB, esbuild, bare Node, no repo) and run to prove it is copyable.
…ditions
The prototype was thin: a file move, a document, and a linter, verifying
nothing new. This is the part that had to be run.
A real page, in a real browser, fetching its own JSONL export and drawing
seven versions of its own history with no backend and no build step. That
is the claim the whole thesis rests on and it had never once been executed.
examples/self-charting.html is the page; examples/self-charting.png is the
browser shot; zero console errors.
Running it produced the finding a document could not. Self-reading needs
THREE conditions and only the third was known:
1. the world link grants read — else 404, an opaque origin sends no
credentials
2. public history is enabled — else the series is EXACTLY ONE POINT
3. the response carries CORS — else fetch throws before any response
Condition 2 is the one nobody predicts, and it is correct behaviour: a
history export must not become a way around the gate guarding older
versions, so an anonymous caller on a record without public history gets
the current value only. Which means a public record's own page charts a
single point by default and looks broken. Measured on a six-version
record: anonymous 1 line, authorized 6, public-history-on 6.
SPEC 6.1 now carries the table and the measurement.
…h a measurement
SPEC said 'a host wanting SQL does not need the host's help: point DuckDB
at the URL.' That was an assertion. It is now a result, and it is the
entire justification for refusing to build server-side aggregation, so it
had to be run rather than believed.
DuckDB pointed straight at the live export, no extension install, no
download step, no host involvement:
- the shape needs no massaging: n is BIGINT, at infers as TIMESTAMP, and
the payload as STRUCT(run, pass, fail), so data.pass just works
- SHAPE DRIFT is handled, which is the realistic case. Across a series
where v8 added a flaky array and v9 dropped fail, the union schema came
back STRUCT(run, pass, fail, flaky VARCHAR[]), absent fields read NULL
rather than raising, and count() reported 8/9 with fail, 1/9 with flaky
- nothing fabricates zeroes: an engine that never read section 3.2
honours its rule anyway, because absence in JSON is absence
- THREE records union into ONE table with read_json_auto([...],
filename=true): group by artifact, window over versions, rank by latest.
Exactly the cross-record aggregation a host is tempted to build.
Two caveats now stated instead of discovered: the export should advertise
Accept-Ranges (without it an engine reads the whole file, harmless at KB
and not at MB), and read_json_auto infers from a sample, so a field that
first appears late in a long series can be missed — the reader's fix, but
it looks like missing data.
examples/ carries all four runnable scripts and the browser shot.
…e caveat The last run used Node bindings, which is the easy environment. The claim worth testing is SQL IN THE DOCUMENT, and the sandbox is where every constraint lives — so it was run there. From inside an opaque origin, every step of the hostile path worked: cross-origin ESM import of duckdb-wasm from a CDN; a Web Worker spawned from a BLOB URL (the step most likely to be refused, and the one worth checking before promising anyone SQL in a page); registerFileURL against the record's own export with read_json_auto over it, ten versions, no fetching by hand; and a window function computing per-version deltas entirely in the browser. So the real requirement is only the three conditions in SPEC 6.1 plus a reachable CDN. The host contributes a static file and a CORS header and never sees the query. The engine also confirmed the Accept-Ranges caveat in its own words — the console carried exactly one warning, 'falling back to full HTTP read'. DuckDB tried range requests, the export does not advertise support, and it degraded to a whole-file read rather than failing. Good failure mode, and the argument for fixing it before a series gets large. examples/duckdb-in-page.html plus the browser shot.
The read layer owes three reads, and the spec covered two: structured (slots) and cross-record. The third is reading ANY PART of a big document fast — outline, named section, line window — so a reader spends tokens proportional to the part it needs, not to the document containing it. Section 3.4 names it and carries the three rules: partial reads obey the visibility section unchanged; derived views are recomputable and may be cached content-addressed WITH a generation that bumps on any deriving-code change (unlike slots, which are canonical rows); and measure before caching — this host gates its own cache PR on a per-read timing probe rather than an intuition.
an1va
marked this pull request as ready for review
July 31, 2026 02:30
an1va
force-pushed
the
feat/slot-read-contract
branch
from
July 31, 2026 02:30
735f863 to
5ce27e3
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL's five high-severity alerts arrived the day the parser moved to its own package, on code that had passed silently inside core for two PRs — the move re-attributed every line as new, which is the only reason they surfaced. All five were real. The one that stings: the close-tag matcher accepted only the literal </script>, but a browser ends the element at </script >, </script foo>, any junk before the >. So this parser read PAST a close tag the browser honored and the two disagreed about where a body ends — the exact drift SPEC.md declares a normative hazard, present in the reference implementation of that spec. Fixed to the browser's grammar, with a parity test verified to fail against the old regex. The other four were polynomial backtracking in the fence regex and the numeric-cell detectors — a ReDoS in a parser meant to run on untrusted documents on any host, and in an advisory helper that must never be the slow path of a publish. The fence mega-regex is now a line scanner, linear by construction and a more literal match for the documented grammar; cell detection is bounded captures with an anchored numeric check. The suite now includes CodeQL's own attack strings with time bounds. A moved file getting fresh eyes found what two green PRs did not: default scanners only look at what changed, so code that never moves is code that never gets re-read.
…tring The clause the bug taught: a browser ends a script element at </script followed by whitespace, slash, or >, so an implementation matching only the literal </script> silently disagrees with the browser about where a body ends. Written into the spec because the reference implementation itself got it wrong.
CodeQL's re-evaluation cut the alerts 5 to 2, and the two survivors were in the FIX: the rewritten numeric-cell regex kept \s* runs around the optional %, which is the same polynomial shape in a smaller pattern. The capture is trimmed first and the pattern now contains no whitespace, so there is nothing left to backtrack over. The round's second lesson, recorded next to the regex: the fix for an ambiguous regex is not a subtler regex.
… forever
Artifacts carry facts. The name earns its keep beyond the pun: it names an
ASSERTION, which is what all five uses are (status, findings, metrics,
config, series), and it puts provenance in the noun — a fact and the
version that asserted it are inseparable, which is the actual difference
between this and a database.
The old name was worse than unlovely, it was inverted. 'slot' means a
named placeholder that content is injected INTO — Web Components, Vue,
Svelte. Ours is the content. It also collided inside this very repo: the
UI kit puts data-slot on every card, sheet and dialog.
Renamed: the package (@derive/facts), every exported identifier, the
attribute (data-fact), the media type (application/derive-facts), the
markdown fence word, both CI guards, the publishing skill (regenerated,
not hand-edited), the spec, and the examples.
NOT renamed, deliberately: the version_data table and its slot column.
They are internal, invisible to any caller, and renaming them means a
migration against a table that shipped to production hours ago. The cost
is a column name that lags its concept; the alternative is migration risk
for zero user-visible gain.
Back-compat is permanent and tested. A version is IMMUTABLE, so documents
already published carry the old spelling in bytes nothing may rewrite —
they must keep parsing a decade from now. Five tests pin it: new spelling,
old spelling, a mixed document, both fence words, and an ordinary code
fence left alone.
Two process notes worth keeping. BSD sed has no \b, so the first rename
pass silently changed NOTHING while printing a checkmark per file, and it
typechecked green because it was uniformly unrenamed — the checkmark was
the lie, the grep was the truth. And a blanket prose rename mangled nine
unrelated files where slot meant something else ('chunk slots', 'costs the
agent a slot to understand'); those were reverted and the pass rescoped.
Caught by accident and worth stating plainly: the prose pass changed a
RESPONSE KEY (slots: -> facts: in read(data:"*")) and every test still
passed, because the same sweep renamed the assertions. Green tests over a
changed API is the exact trap this project keeps writing down, so rather
than reverting half of it or leaving it inconsistent, the surface is now
renamed on purpose and completely:
read(data:"x") -> { fact: "x", data: {...} }
read(data:"*") -> { facts: [{ fact, bytes }] }
find(data:"x") -> { fact: "x", results: [...] }
find(data:"*") -> { slots -> facts: [{ fact, artifacts, latest_at }] }
publish receipt -> { data: [{ fact, bytes }] }
The PARAMETER is unchanged on both tools — it reads fine, it is
what a caller types, and moving it would break cached client schemas for
no gain.
Defensible because the surface is hours old with no external consumers,
and a convention that calls them facts everywhere except its own JSON is
the kind of half-rename people quote back at you.
…xposed Asked for: accept the old spelling AND self-heal. The only self-heal a byte-faithful host may perform is an INVITATION. Rewriting an author's source to suit our vocabulary would be faster and is forbidden — a version is the author's bytes, and a host that edits them has broken the promise the whole provenance story rests on. So parsing a legacy block now adds one advisory per publish (not per block) naming the current spelling, promising the old one keeps working, and saying outright that nothing rewrites what was published. Everything is still stored: the old spelling is not a degraded mode. Writing it exposed a real hole: missingFactAdvisory's 'already tried to carry data' guard had been narrowed to the new spelling by the rename, so a page whose OLD-spelling block merely failed to parse would be nagged to add a fact it already had — the most irritating possible advice. It now matches both. Test fixtures moved to the current spelling so they exercise what new authors write, with the legacy cases kept explicit and deliberate.
…slot data in the card Main's perf round replaced infoFor's four reads with a single unfurlInfo, which is a real win on the most-trafficked anonymous surface. Two conflicts, both resolved by taking BOTH sides rather than either: ports.ts main's new unfurlInfo + currentVersions, our vocabulary embeds.ts main's one-query path, our factSummary(facts) Worth stating: their rewrite KEPT dataSummary, so the card still leads with the artifact's own numbers — the seam held from their side without either of us coordinating, which is what the embeds test exists to guarantee. unfurlInfo's payload field is renamed slots -> facts through the port and both dialects so the new code does not reintroduce the old vocabulary the moment it lands.
Verified alongside: #587 makes NO data model change. deploy/d1-schema.sql is byte-identical to main and these two comment lines were the only diff in either schema file, so there is no migration and nothing to roll back.
an1va
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Renames data slots to facts (artifacts carry facts), extracts the parser into a standalone dependency-free package, and writes the read contract down. Along the way it fixes five CodeQL alerts and two back-compat holes that only surfaced because the code moved.
No data model change.
deploy/d1-schema.sqlis byte-identical to main; the only schema-file diff is comments. No migration, nothing to roll back.version_data.slotstays as production has it — deliberately, since renaming a column that shipped yesterday buys nothing a caller can see.Why rename
slotmeans a named placeholder that content is injected INTO — Web Components, Vue, Svelte. Ours is the content: the exact inverse. It also collided inside this repo, where the UI kit putsdata-sloton every card, sheet and dialog. Facts names an assertion, which is what all five uses are (status, findings, metrics, config, series), and puts provenance in the noun: a fact and the version that asserted it are inseparable.The old spelling parses forever. A version is immutable, so documents already published carry
data-slotin bytes nothing may rewrite. Both spellings extract identically, pinned by five tests. A legacy block earns one advisory per publish inviting the author to switch — the only self-heal a byte-faithful host may perform, since rewriting someone's source would break the promise the provenance story rests on.What is actually new (not naming)
</script>, but a browser also ends the element at</script >or</script foo>— so the reference implementation of the spec disagreed with browsers about where a block ends, the exact hazard its own SPEC declares normative.fact/facts(thedata:parameter is unchanged on both tools). Breaking in principle; nothing in the repo or outside it reads the old shape.The contract, and the proof
packages/facts/SPEC.mdis the contribution. Having written the parser, the parser is the least valuable part — a few hundred lines with no incumbent. The difficulty is the contract, where every clause is a decision that fails silently if made wrong, and each is drawn from a live defect here: absence is never a fabricated zero, cross-record rows join to the current version, aggregates disclose so gate before counting, never report what was filtered, caller-specific responses are not shared-cacheable, and a sandboxed page needs CORS to read its own data.packages/facts/examples/holds five runnable things and two browser screenshots, so none of it rests on assertion:registerFileURL, window function, all from an opaque origin. It also confirmed theAccept-Rangescaveat in its own words.lint:facts-portablekeeps the package dependency-free and fails loudly if it matches no source at all;lint:facts-visibilityis the renamed multi-artifact read guard. Both were verified against a known-bad input before being trusted.Verified on the deployed preview
One document carrying both spellings, published through the preview:
data-fact→ extracted,data-slot→ extracted. Test artifact deleted (204, then 404 from prod and preview). CORS header live on the fact route. Note for reviewers: this preview binds production's D1/R2, so it is not a sandbox.Full gate green — typecheck, every package's tests, all lint including both new guards.
🤖 Generated with Claude Code