Three data.gov collections whose feeds explain each other

[ralyodio]

nichedb had no view of aviation, water or consumer finance, and the point of
adding them together is that none of the nine sources is worth much alone.
Every one is keyless, every one was queried live before it was written, and
each collection is built so that its feeds answer a question none of them
answers by itself.

AVIATION -- why flights are late

The FAA publishes, every couple of minutes, the traffic management initiatives
in force: ground stops, ground delay programs, airspace flow programs, airport
closures. It is a snapshot, not a log. It says a ground stop exists; it never
says one ended, and nothing anywhere records how long it ran. So the adapter
keeps the open programs in its cursor with the time each was first seen, writes
one row per program that updates while it lasts, and writes it once more when it
leaves the snapshot with endedAt and a duration. "BOS ground stop, 3h 40m,
thunderstorms" is a fact about a day of flying that exists only if something was
watching the whole time.

Beside it, the two halves of what a pilot is told: every SIGMET and AIRMET in
force, and the decoded observation at the airport underneath. The FAA says ORD
is stopped for thunderstorms; the METAR says it is IFR with a 30-knot gust and
the SIGMET is drawn over the field. One airport, one hour, three sources.

WATER -- too much and too little

River gauges at or above their action stage, from the 12,000 NOAA forecasts,
observed and forecast as separate rows because "is it flooding" and "will it"
are different questions and a forecast that turns out wrong should stay next to
the observation that contradicted it. Coastal water levels measured against the
height at which each station floods, so a reading arrives as "minor coastal
flooding, 0.13 ft over stage" rather than as a number with no scale. And the US
Drought Monitor, weekly, which is the slow half: a gauge says what is happening
this hour, the drought map says what the last six months did to the ground that
river runs through.

CONSUMER FINANCE -- what people say their bank did, and who the bank is

17.6 million CFPB complaints, around seven thousand a day, published within a
day or two and naming the company. The company is a bare uppercase string with
no identifier of any kind, so the collection carries the two FDIC feeds that
turn it into an institution: the register of all 4,235 insured banks, and the
584,000 structure changes over it. Both normalise the name the same way this one
does, and a test fails if they ever stop.

SEVEN THINGS THE LIVE DATA DECIDED

• The CFPB search accepts an offset, echoes it, and ignores it. frm=0,
  frm=500 and frm=1000 return byte-identical pages. The first version paged
  by offset and stored the same 500 complaints ten times over -- 4,500 duplicate
  ids in one run -- while reporting 5,000 new rows. It walks days instead, with
  size up to the 10,000 result window, and splits a day that will not fit by
  state, because the date filters are day-granular and every other facet is
  dominated by one value.
• A METAR bounding box silently caps at 400 stations. A box over the continental
  United States returns exactly 400; its eastern half returns 247 and its
  western half 240. Nothing in the response says it was truncated. The adapter
  quarters a box whose answer comes back at the cap and recurses: 1,101 stations
  where the single box had reported 400, and 68 airports below VFR where it had
  found 30.
• FDIC transaction numbers are not rows. One merger writes a row per institution
  and per office it touches: 1,702 changes in one run carried 1,169 distinct
  transaction numbers. Keyed on TRANSNUM a third of the register overwrites the
  rest, and the loss looks like a quiet quarter. ID is the row.
• The FDIC's twenty-odd boolean event flags are zero on the great majority of
  transactions, mergers included. CHANGECODE_DESC is the field it actually
  fills in. And the institution endpoint has no CLASS field at all -- it is
  BKCLASS -- so asking for the wrong one files every bank in the country as
  unclassified and raises nothing.
• CFPB narratives are published months after the complaint. Complaints received
  since 10 August carry one narrative between them; the same query from 1 May
  returns 42,516. A narratives source that tailed the newest rows would be
  permanently empty, so it sweeps a 180-day window one day per run.
• NWPS accepts state, wfo and rfc parameters, ignores them, and answers
  with all 12,000 gauges -- a failure that looks exactly like success. Regions
  here are bounding boxes. And its "no reading" sentinel is -999, which is
  finite; Number(null) is 0, which is also finite. A test written for the
  first case caught the second: an absent stage was being published as a river
  at exactly zero feet.
• The Drought Monitor's classes are cumulative. d0 is the area at D0 or worse,
  so adding d0 through d4 reported 154% of Delaware in drought. Both readings
  are stored, and cumulativeNote says which is which.

Two smaller ones. The FAA sends Airport Closures twice in one snapshot, one
block for airports shut outright and one for airports shut to transient general
aviation, so a reader that indexed blocks by name would keep the second and lose
the first. And a SIGMET has no area field: the region is a bare line of
two-letter codes above the FROM line in the bulletin text, and the only
alternative was labelling every hazard in the country KKCI -- the office that
issues them all, and a place no weather is ever over.

Also fixed: sources.slug and feeds.slug are unique across the whole
database, not per collection, and nothing in the seed log says so. There is now
a test.

Verified live against every upstream: 9 FAA programs in force, 13 SIGMETs,
1,101 METAR stations, 63 river gauges in flood across the eight regions, 31 tide
stations, 312 drought rows, 15,177 complaints over three days with no duplicate
ids, 4,235 banks and 1,702 structure changes. 555 tests pass, biome clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs