Repository navigation
Schema decision: is known_nulls an exact count or a ceiling? #121
Description
schema.known_nulls maps a column to a null count, and #14's template treats it as exact: n != known.get(col, 0) fails. Two dynamic snapshots now carry counts that a refresh will change:
business_cycle_data.csv.yml:{YR1960: 9}— stable by construction, growth is undefined in the first year for every economy.- the FRED composite:
{UNRATE: 349, UMCSENT: 616, CPILFESL: 457, M0892AUSM156SNBR: 1132}— leading nulls before each series starts, on a monthly panel whose length grows with every refresh, so at least some of these counts move as soon as the panel is extended.
Frozen extracts want exact counts (a changed null is a changed file). Tracking snapshots want a shape constraint, not a number.
Options.
- Exact everywhere; the refresh stamps the new counts alongside
retrievedanddate_range.end, so the manifest diff shows them. Simple, but it turns a data invariant into a field the pipeline rewrites. - Ceiling everywhere:
n <= known. Never fails on a refresh that fills in history, but it silently accepts a column going empty below the ceiling. - Exact by default, with an explicit ceiling form for tracking snapshots, e.g.
{UNRATE: {max: 400}}or a{col: "<=400"}string. Two forms to document and validate. - A positional rule instead of a count: nulls allowed only before a series' first observation or in the newest k rows, never inside (what business_cycle's
validate()already implements by hand), with the count kept as documentation.
Decide, then update manifest-schema.yml's known_nulls comment and the template's check in step with the manifest-driven validate().
Sub-issue of #14.
Activity
mmcky commentedon Sep 6, 2026
Decision, 2026-09-07: the integer stays exact; dynamic snapshots declare null placement, not a count
Option 4 from the body, chosen after measuring what the manifests declare and what the two dynamic builders already assert.
What is in the tree today. 12 of 41 manifests declare non-empty
known_nulls, all as plain integers; ten are static files where exact is right. Two manifests (unemployment_rate_annual.csv.yml,private_credit_to_gdp.csv.yml) use a third form,known_nulls_total: N, described in their own comments as a ceiling — and that key is declared nowhere: not inmanifest-schema.yml, not in any script. So three shapes are in the wild and one is documented.Which counts actually move on a refresh. The FRED composite's nulls split by position (first vintage, 1919-01 to 2026-07):
Only M0892's count grows, by one per month, because the series ended in 1942-06 and every new row adds a trailing null; the other three are stable unless the source backfills. The inner nulls are the 2025-10 shutdown hole and UMCSENT's sparse pre-1978 years.
business_cycle_data's{YR1960: 9}is stable by construction; the two World Bank siblings have leading nulls for GBR and FRA and one trailing null for an unpublished newest year. Neither dynamic builder asserts a count — both assert placement (before first observation, in known inner holes, after last observation), and the manifest comments say the counts are "recorded for the reader, not asserted".The rules.
known_nullsis exact:n != knownfails. Required for any nulled column in averbatimorconstructedfile.dynamic-snapshotdeclares placement per column with a small closed vocabulary —leading,trailing, andinner: [...]listing the known holes (dates for a time index, column names for a wide file). The validator checks that every null sits where declared and nowhere else. A dynamic file may still write an integer for a column whose count is stable (YR1960: 9) to get the stronger check.class, which every manifest already carries — no new per-file judgement field.known_nulls_totalis retired: a file-level total cannot say where the nulls are. Its two users move toleadingon the affected columns andtrailingon the newest year.{max: N}ceiling form. A ceiling silently accepts a column emptying out below the bound, which is the failure the check exists to catch.Edits this implies. Placement blocks in the four dynamic manifests, the
known_nullscomment and anullsplacement entry inmanifest-schema.yml, and the template's check replaced by the shared one from #119. Closes with the PR that records the vocabulary inmanifest-schema.yml.mmcky commentedon Sep 6, 2026
Correction to the decision above, found while writing it into the tree:
known_nulls_totalis used by four manifests, not two. Besides the two World Bank siblings (where it was a ceiling and is now replaced by anulls:placement block), it appears nine times inside thesheets:entries ofassignat.xlsx.ymlanddette.xlsx.yml. Those are positional reads withheader: null, so the range has no column names to count against, and a file-level exact total is the only form a count can take. There it is not a ceiling and not drift — it is an exact count for a static file, which is exactly what the integer rule wants.So the rule as recorded in
manifest-schema.yml:known_nulls_total: Nis legal only inside aheader: nullsheet entry, as an exact total; it is retired for named columns and never a ceiling. The rest of the decision stands unchanged.Record the four schema and naming decisions: rename, pattern capture …
Manifest-driven validate(): the schema block is the spec, shared by t…