Repository navigation
Schema decision: the canonical dtype vocabulary for schema.columns #122
Description
Every schema.columns entry declares a dtype, and the invariant tests in PLAN Phase 5 are meant to check it. Across the 41 manifests the vocabulary today is:
| value | occurrences |
|---|---|
float64 |
126 |
float32 |
57 |
int64 |
49 |
str |
43 |
string |
21 |
object |
9 |
datetime64 |
2 |
datetime |
2 |
Three names for text (str, string, object) and two for dates, so no validator can compare a declared dtype with what pandas reads without a mapping. Business_cycle's manifest says str where manifest-schema.yml's example says string.
To decide.
- The canonical set: pandas dtype strings (
float64,int64,string,datetime64[ns],bool,category) is the obvious candidate sincevalidate()will compare against a pandas frame; whetherfloat32stays as a distinct declaration or collapses tofloat. - What the check means for a CSV: a CSV has no dtypes, so the declaration is a statement about what
pd.read_csvwith the manifest's read options yields. Whether the manifest records those read options (index_col,parse_dates) or the check uses pandas defaults. - Text columns:
stringversusobject, given the lectures read with default pandas settings that yieldobject. - Migration: a one-time normalisation pass over the manifests once the set is fixed, and a
manifest-schema.ymlconformance check that rejects anything outside it.
Sub-issue of #14.
Activity
PLAN Phase 5: point the PR-validation bullet at the #14 tracker's sub…
mmcky commentedon Sep 6, 2026
Decision, 2026-09-07: pandas-3 names, compared by family;
stris the text dtypeRe-measured rather than reusing the table in the body. The lectures pin
anaconda=2026.07, which is pandas 3;lecture-datascience.mystis still onanaconda=2024.06(pandas 2), and Pyodide inlecture-wasmis a pandas-2 reader too.Eight spellings in use.
float64126,float3257,int6449,str43,string21,object9,datetime642,datetime2. All 57float32are in the four Stata.dtamanifests where float32 is the real storage type — not drift.What breaks when a validator compares declared against read. Every CSV read with pandas defaults, each of the 212 named columns compared with
str(dtype):string/objectvsstr)str/stringvsobject)The 43 manifests already on
strare right under the lectures' pin;stringandobjectare the drift. The 19 not-found columns are allcountries.csv, which is semicolon-delimited and whose manifest already recordsdelimiter: ";"— so the repo already has a read-options precedent (delimiter, andread_ason the xlsx sheets) and the validator only has to honour it. The four date columns can never match withoutparse_dates. The one "other" is a real declaration error:forbes-billionaires.csv'sgovernmentis declaredstrbut is a boolean column with 2606 nulls in 2935 rows, which pandas reads asobject.The rules.
str(dtype)of a default read:float64,int64,str,bool,datetime64. Storage types that non-CSV formats genuinely carry (float32,int32,int8from.dta) stay legal.string,objectanddatetimeare retired.str/string/objectare one text family, anddatetime64matches any unit (pandas 3 givesdatetime64[us], pandas 2[ns]). The check therefore stays green on pandas-2 readers without a second vocabulary.delimiterandread_askeys and derivesparse_datesfrom any column declareddatetime64, so a date declaration is both the contract and the instruction for reading it. No new field.stringand 9object→str; 2datetime→datetime64; forbesgovernment→boolwith its nulls declared per Schema decision: is known_nulls an exact count or a ceiling? #121.manifest-schema.ymlcarries the closed list and the conformance check rejects anything outside it.Closes with the sweep PR, which can precede #119.
Record the four schema and naming decisions: rename, pattern capture …
Manifest-driven validate(): the schema block is the spec, shared by t…