NTSB accidents: read the bulk archive, because there is no working API

[ralyodio]

The aviation collection shipped without the half that makes it worth keeping.
faa-nas-status says the airspace is delayed today and aviation-metar says
what the sky is doing over the field, and neither says what happens when it
goes wrong. This adds the 31,000 accidents and incidents the NTSB has
investigated, and the reason it belongs beside the other two rather than in a
collection of its own: the events table carries a metar column holding the
raw observation at the moment of the accident, in the identical format
aviation-metar publishes hourly. The same field, from the same service, thirty
years apart. 252 of 400 accidents in one sampled slice carry one.

WHY A 96 MB DOWNLOAD

The NTSB's query service answers a malformed request with a 400 that names the
problem and a well-formed one with 500 An unknown exception occured [sic]. It
is not an interface. What the NTSB does publish reliably is avall.zip: 96 MB
holding a 558 MB Access database, rebuilt monthly, newest event eight days old.
The monthly delta archives beside it stop in December 2022, so the full file is
the file. mdbtools and unzip go into the Dockerfile; mdb-json writes one
JSON object per line, which is why the narrative table reads as 522,053 lines
and 28,436 rows -- the CSV export does not escape the newlines inside a
narrative and cannot be split on them.

RE-READING IT IS THE POINT, NOT THE COST

An NTSB investigation is published long before it is finished: a preliminary
report within days, a probable cause that can take two years, all under one
ev_id. So the adapter re-reads events it already holds and lets the upsert do
the work. An unchanged accident hashes to the row already stored and costs no
write; one the NTSB has since ruled on gains its probable cause in place rather
than arriving as a second row contradicting the first. In the newest slice 6 of
300 are ruled; 8,000 rows deeper it is 307 of 400. Those 294 rows are what a
re-read is for.

A run emits a bounded slice, newest first, and remembers where it stopped,
because upsertItems sends every row in one statement and 31,000 accidents
carrying their narratives is 140 MB of text in a single query. The extracts are
cached against the file's publication date, read off the NTSB's own listing
page, so only the first slice pays for the download and a new archive restarts
the walk at the newest accident.

FOUR THINGS THE DATA DECIDED

• The dates are 08/22/26, and new Date reads that as 1926. Every accident in
  the file would publish a century early and sort ahead of everything else in
  the collection. The database starts in 1982, so 82 and above is last century.
• A regex alternation of (\d{2}|\d{4}) matches the 20 of 03/04/2015 and
  yields 2020 -- a wrong date that parses, sorts, and raises nothing. Four
  digits are tried first. A test written for the century boundary caught it.
• The latitude column holds packed DMS (003000N) beside a dec_latitude of
  30. Read as a number the packed pair puts a Texas accident three thousand
  degrees north, so only the decimal columns are used.
• damage is SUBS, and printed raw it reads "Aircraft subs." SUBS is the
  threshold that makes an event an accident rather than an incident, so it is
  the most important word in the row and it is now a word. The operator columns
  carry a bare N on many rows, which printed straight reads "operated by N".

Also: the three collection tests derived their expected kinds from a hardcoded
list of adapter names, so adding an adapter to a collection made the list wrong
rather than making the test pass. They now derive from the registry. That is
what failed when these feeds were added, which is the test working.

Verified against the real 2026-09-01 archive: 31,124 events, 31,670 aircraft
rows, 28,436 narratives; a 400-row slice at offset 8,000 gave 307 ruled, 60
fatal and 252 with a METAR, with probable-cause text intact. 577 tests pass,
biome clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs