From efa8ee21d52d80db73ca6831ad5f2b2c4b0ada47 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Wed, 9 Sep 2026 17:58:11 +0000 Subject: [PATCH] Three data.gov collections whose feeds explain each other nichedb had no view of aviation, water or consumer finance, and the point of adding them together is that none of the nine sources is worth much alone. Every one is keyless, every one was queried live before it was written, and each collection is built so that its feeds answer a question none of them answers by itself. AVIATION -- why flights are late The FAA publishes, every couple of minutes, the traffic management initiatives in force: ground stops, ground delay programs, airspace flow programs, airport closures. It is a snapshot, not a log. It says a ground stop exists; it never says one ended, and nothing anywhere records how long it ran. So the adapter keeps the open programs in its cursor with the time each was first seen, writes one row per program that updates while it lasts, and writes it once more when it leaves the snapshot with `endedAt` and a duration. "BOS ground stop, 3h 40m, thunderstorms" is a fact about a day of flying that exists only if something was watching the whole time. Beside it, the two halves of what a pilot is told: every SIGMET and AIRMET in force, and the decoded observation at the airport underneath. The FAA says ORD is stopped for thunderstorms; the METAR says it is IFR with a 30-knot gust and the SIGMET is drawn over the field. One airport, one hour, three sources. WATER -- too much and too little River gauges at or above their action stage, from the 12,000 NOAA forecasts, observed and forecast as separate rows because "is it flooding" and "will it" are different questions and a forecast that turns out wrong should stay next to the observation that contradicted it. Coastal water levels measured against the height at which each station floods, so a reading arrives as "minor coastal flooding, 0.13 ft over stage" rather than as a number with no scale. And the US Drought Monitor, weekly, which is the slow half: a gauge says what is happening this hour, the drought map says what the last six months did to the ground that river runs through. CONSUMER FINANCE -- what people say their bank did, and who the bank is 17.6 million CFPB complaints, around seven thousand a day, published within a day or two and naming the company. The company is a bare uppercase string with no identifier of any kind, so the collection carries the two FDIC feeds that turn it into an institution: the register of all 4,235 insured banks, and the 584,000 structure changes over it. Both normalise the name the same way this one does, and a test fails if they ever stop. SEVEN THINGS THE LIVE DATA DECIDED - The CFPB search accepts an offset, echoes it, and ignores it. `frm=0`, `frm=500` and `frm=1000` return byte-identical pages. The first version paged by offset and stored the same 500 complaints ten times over -- 4,500 duplicate ids in one run -- while reporting 5,000 new rows. It walks days instead, with `size` up to the 10,000 result window, and splits a day that will not fit by state, because the date filters are day-granular and every other facet is dominated by one value. - A METAR bounding box silently caps at 400 stations. A box over the continental United States returns exactly 400; its eastern half returns 247 and its western half 240. Nothing in the response says it was truncated. The adapter quarters a box whose answer comes back at the cap and recurses: 1,101 stations where the single box had reported 400, and 68 airports below VFR where it had found 30. - FDIC transaction numbers are not rows. One merger writes a row per institution and per office it touches: 1,702 changes in one run carried 1,169 distinct transaction numbers. Keyed on TRANSNUM a third of the register overwrites the rest, and the loss looks like a quiet quarter. `ID` is the row. - The FDIC's twenty-odd boolean event flags are zero on the great majority of transactions, mergers included. `CHANGECODE_DESC` is the field it actually fills in. And the institution endpoint has no `CLASS` field at all -- it is `BKCLASS` -- so asking for the wrong one files every bank in the country as unclassified and raises nothing. - CFPB narratives are published months after the complaint. Complaints received since 10 August carry one narrative between them; the same query from 1 May returns 42,516. A narratives source that tailed the newest rows would be permanently empty, so it sweeps a 180-day window one day per run. - NWPS accepts `state`, `wfo` and `rfc` parameters, ignores them, and answers with all 12,000 gauges -- a failure that looks exactly like success. Regions here are bounding boxes. And its "no reading" sentinel is -999, which is finite; `Number(null)` is 0, which is also finite. A test written for the first case caught the second: an absent stage was being published as a river at exactly zero feet. - The Drought Monitor's classes are cumulative. `d0` is the area at D0 or worse, so adding d0 through d4 reported 154% of Delaware in drought. Both readings are stored, and `cumulativeNote` says which is which. Two smaller ones. The FAA sends `Airport Closures` twice in one snapshot, one block for airports shut outright and one for airports shut to transient general aviation, so a reader that indexed blocks by name would keep the second and lose the first. And a SIGMET has no `area` field: the region is a bare line of two-letter codes above the `FROM` line in the bulletin text, and the only alternative was labelling every hazard in the country `KKCI` -- the office that issues them all, and a place no weather is ever over. Also fixed: `sources.slug` and `feeds.slug` are unique across the whole database, not per collection, and nothing in the seed log says so. There is now a test. Verified live against every upstream: 9 FAA programs in force, 13 SIGMETs, 1,101 METAR stations, 63 river gauges in flood across the eight regions, 31 tide stations, 312 drought rows, 15,177 complaints over three days with no duplicate ids, 4,235 banks and 1,702 structure changes. 555 tests pass, biome clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs --- README.md | 20 +- packages/adapters/src/aviationweather.js | 552 +++++++++++++++++++++++ packages/adapters/src/cfpb.js | 446 ++++++++++++++++++ packages/adapters/src/coops.js | 309 +++++++++++++ packages/adapters/src/droughtmonitor.js | 295 ++++++++++++ packages/adapters/src/faanas.js | 339 ++++++++++++++ packages/adapters/src/fdic.js | 422 +++++++++++++++++ packages/adapters/src/index.js | 16 + packages/adapters/src/nwps.js | 254 +++++++++++ packages/core/src/seed.js | 131 ++++++ test/adapters.test.js | 24 + test/aviation.test.js | 250 ++++++++++ test/consumer-finance.test.js | 268 +++++++++++ test/water.test.js | 276 ++++++++++++ 14 files changed, 3599 insertions(+), 3 deletions(-) create mode 100644 packages/adapters/src/aviationweather.js create mode 100644 packages/adapters/src/cfpb.js create mode 100644 packages/adapters/src/coops.js create mode 100644 packages/adapters/src/droughtmonitor.js create mode 100644 packages/adapters/src/faanas.js create mode 100644 packages/adapters/src/fdic.js create mode 100644 packages/adapters/src/nwps.js create mode 100644 test/aviation.test.js create mode 100644 test/consumer-finance.test.js create mode 100644 test/water.test.js diff --git a/README.md b/README.md index a1ca1f5..7cfc5ad 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ The first deployment is [nichedb.dev](https://nichedb.dev). Run your own on anyt | **item** | One row a source produced. Title, URL, when (with `time_known` and `precision`), tags, and the adapter's payload in `data`. | | **feed** | A saved query over a collection. Has a page, RSS and JSON Feed renderings, an API endpoint, and followers who are told when it changes by push, email or signed webhook. | -Adapters are one file each in `packages/adapters/src`. Twenty-nine ship today across thirteen collections: +Adapters are one file each in `packages/adapters/src`. Seventy-five ship today across twenty-seven collections: | Collection | Adapters | Key needed | | --- | --- | --- | @@ -25,11 +25,25 @@ Adapters are one file each in `packages/adapters/src`. Twenty-nine ship today ac | tabletop | `scryfall-sets`, `scryfall-cards` | no | | space | `launch-library` | no | | chess | `lichess-broadcasts` | no | -| alerts | `usgs-earthquakes`, `nws-alerts`, `gdacs` | no | +| alerts | `usgs-earthquakes`, `gdacs` | no | +| weather | `nws-alerts`, `nhc-cyclones`, `swpc-space-weather`, `eonet-events` | no | | outages | `statuspage` (any Statuspage host) | no | | extensions | `firefox-addons`, `vscode-extensions`, `mcp-registry` | no | | health | `openfda-recalls`, `clinical-trials` | no | | research | `arxiv`, `crossref` | no | +| automotive | `fueleconomy-catalog`, `nhtsa-recalls`, `nhtsa-complaints`, `nhtsa-safety-ratings` | no | +| markets | `iso-mic-exchanges`, `alpaca-corporate-actions`, `alpaca-news`, `nasdaq-halts`, `ecb-fx-rates` | Alpaca only | +| crime | `socrata-crime`, `uk-police-crime`, `fbi-crime-estimates` | FBI only (free api.data.gov key) | +| public-money | `usaspending-awards`, `ocds-tenders`, `ted-notices` | no | +| housing | `uk-land-registry`, `freddie-mac-rates`, `building-permits` | no | +| jobs | `bls-series`, `eurostat`, `warn-layoffs` | no | +| ai-incidents | `rogue-ai-incidents`, `rogue-ai-research`, `aiid-reports` | no | +| news | `newsfeed`, `gdelt`, `rssamplifier`, `brisk`, `news-channels` | no | +| domains | `ntld-totals`, `ntld-tlds`, `ntld-launches`, `ntld-changes` | no | +| podcasts | `podcasts` | no | +| aviation | `faa-nas-status`, `aviation-hazards`, `aviation-metar` | no | +| water | `nwps-river-gauges`, `coops-water-levels`, `drought-monitor` | no | +| consumer-finance | `cfpb-complaints`, `fdic-institutions`, `fdic-structure-changes` | no | ## Enrichment @@ -59,7 +73,7 @@ bun run build:client bun run dev # web + worker in one process on :3000 ``` -Migrations apply themselves on boot. The three collections, their default sources and a dozen feeds are seeded on first boot; sources whose adapter needs a credential the deployment lacks are created paused. The first account to sign in is an admin. +Migrations apply themselves on boot. Every collection, its default sources and its feeds are seeded on first boot; sources whose adapter needs a credential the deployment lacks are created paused. The first account to sign in is an admin. `bun run ingest [slug ...]` runs sources from a terminal without Redis. `bun test` runs the suite against an in-process Postgres (PGlite), so it needs no server. diff --git a/packages/adapters/src/aviationweather.js b/packages/adapters/src/aviationweather.js new file mode 100644 index 0000000..22c4ca7 --- /dev/null +++ b/packages/adapters/src/aviationweather.js @@ -0,0 +1,552 @@ +import { defineAdapter, slugify } from '@nichedb/core/adapter'; + +/** + * The two halves of what a pilot is told about the weather, from the NOAA + * Aviation Weather Center: the hazards in the air, and the conditions on the + * ground at every reporting airport. + * + * Both are keyless, both answer in JSON, and together they are what makes the + * FAA's delay feed readable. `faa-nas-status` says ORD has a ground stop for + * thunderstorms; `aviation-metar` has the observation at ORD in the same + * minute and `aviation-hazards` has the convective SIGMET drawn over it. Three + * feeds, one airport, one hour -- which is the join no single upstream offers. + */ + +const HAZARD_URL = 'https://aviationweather.gov/api/data/airsigmet'; +const METAR_URL = 'https://aviationweather.gov/api/data/metar'; + +/** Seconds since the epoch, or nothing. */ +function epoch(v) { + const n = Number(v); + return Number.isFinite(n) && n > 0 ? new Date(n * 1000).toISOString() : null; +} + +const clean = (v) => { + const s = String(v ?? '').trim(); + return s && s !== 'null' ? s : null; +}; + +/** Feet, or nothing. A zero altitude on a SIGMET means "not stated", not sea level. */ +function feet(v) { + const n = Number(v); + return Number.isFinite(n) && n > 0 ? n : null; +} + +/** + * A number that may legitimately be zero. + * + * Temperature, dewpoint and wind direction all have a real zero, so the usual + * `Number(v) || null` would erase a calm north wind and a freezing morning. + */ +export function num(v) { + if (v === null || v === undefined || v === '') return null; + const n = Number(v); + return Number.isFinite(n) ? n : null; +} + +/* ---------------------------------------------------------------- hazards */ + +const HAZARD_NAMES = { + CONVECTIVE: 'thunderstorms', + TURB: 'turbulence', + ICE: 'icing', + IFR: 'instrument conditions', + MTW: 'mountain wave', + ASH: 'volcanic ash', + TS: 'thunderstorms', +}; + +/** + * The states and regions a SIGMET covers, from its own text. + * + * There is no `area` field. The bulletin carries the region as a bare line of + * two-letter codes -- `MI LH` for Michigan and Lake Huron -- immediately above + * the `FROM` line that traces the polygon, and that is the only place it + * appears. Read from the text rather than invented, and null when the bulletin + * is not laid out this way, because the alternative was labelling every hazard + * in the country `KKCI`: the Aviation Weather Center that issues them all, and + * a place no weather is ever over. + */ +export function regionsOf(raw) { + const lines = String(raw ?? '') + .split(/\r?\n/) + .map((l) => l.trim()); + const from = lines.findIndex((l) => /^FROM\b/.test(l)); + if (from < 1) return null; + const above = lines[from - 1]; + return /^([A-Z]{2}\s+)*[A-Z]{2}$/.test(above) ? above : null; +} + +/** Which way the hazard is going, in words, or nothing when it is not moving or not said. */ +export function movement(row) { + const dir = num(row?.movementDir); + const spd = num(row?.movementSpd); + if (dir === null || spd === null || spd <= 0) return null; + const points = [ + 'N', + 'NNE', + 'NE', + 'ENE', + 'E', + 'ESE', + 'SE', + 'SSE', + 'S', + 'SSW', + 'SW', + 'WSW', + 'W', + 'WNW', + 'NW', + 'NNW', + ]; + return `${points[Math.round((dir % 360) / 22.5) % 16]} at ${spd} kt`; +} + +/** The box the hazard's polygon fits in, so a row can be asked whether it covers an airport. */ +export function bboxOf(coords) { + const points = (Array.isArray(coords) ? coords : []) + .map((c) => [Number(c?.lat), Number(c?.lon)]) + .filter(([a, b]) => Number.isFinite(a) && Number.isFinite(b)); + if (!points.length) return null; + const lats = points.map((p) => p[0]); + const lons = points.map((p) => p[1]); + return { + minLat: Math.min(...lats), + minLon: Math.min(...lons), + maxLat: Math.max(...lats), + maxLon: Math.max(...lons), + }; +} + +export function hazardItem(row) { + const id = clean(row.airSigmetId) ?? clean(row.alphaChar); + const from = epoch(row.validTimeFrom); + const to = epoch(row.validTimeTo); + const type = clean(row.airSigmetType) ?? 'AIRMET'; + const hazard = clean(row.hazard) ?? 'unspecified'; + if (!from) return null; + + // A SIGMET is reissued under the same series letter every few hours, so the + // series alone is not an identity; the issue time makes each issuance its own + // row, which is what a reissue is. + const key = [ + clean(row.icaoId) ?? 'unknown', + type, + clean(row.seriesId) ?? clean(row.alphaChar) ?? id ?? 'x', + from, + ].join('-'); + + const label = HAZARD_NAMES[hazard.toUpperCase()] ?? hazard.toLowerCase(); + const low = feet(row.altitudeLow1 ?? row.altitudeLow2); + const high = feet(row.altitudeHi1 ?? row.altitudeHi2); + const band = + low && high + ? `between ${low.toLocaleString()} and ${high.toLocaleString()} ft` + : high + ? `below ${high.toLocaleString()} ft` + : null; + const severity = clean(row.severity); + const area = regionsOf(row.rawAirSigmet); + const office = clean(row.icaoId); + + return { + externalId: `avwx-${slugify(key)}`, + kind: 'aviation-hazard', + title: `${type} ${clean(row.seriesId) ?? ''}: ${label}${area ? ` over ${area}` : ''}`.replace( + /\s+/g, + ' ', + ), + summary: [ + `${type} for ${label}`, + area ? ` over ${area}` : '', + band ? ` ${band}` : '', + severity ? `, severity ${severity}` : '', + movement(row) ? `, moving ${movement(row)}` : '', + `. Valid from ${from}${to ? ` to ${to}` : ''}`, + office ? `, issued by ${office}` : '', + '.', + ].join(''), + url: 'https://aviationweather.gov/gfa/#sigmet', + publishedAt: from, + timeKnown: true, + precision: 'minute', + tags: [ + 'aviation', + 'weather', + 'hazard', + type.toLowerCase(), + slugify(label), + severity ? `severity:${slugify(severity)}` : null, + ...(area ? area.split(/\s+/).map((r) => r.toLowerCase()) : []), + ].filter(Boolean), + data: { + hazardType: type, + hazard, + hazardLabel: label, + severity, + series: clean(row.seriesId) ?? clean(row.alphaChar), + issuingOffice: office, + area, + areaBasis: area + ? 'The region codes on the bulletin itself. `issuingOffice` is the centre that wrote it, not a place the weather is over.' + : null, + movementDirectionDeg: num(row.movementDir), + movementSpeedKt: num(row.movementSpd), + bbox: bboxOf(row.coords), + validFrom: from, + validTo: to, + altitudeLowFt: low, + altitudeHighFt: high, + // The polygon the hazard is drawn over, kept so a row can be asked + // whether it covers an airport rather than only which office issued it. + coords: Array.isArray(row.coords) ? row.coords : null, + rawText: clean(row.rawAirSigmet), + source: 'NOAA Aviation Weather Center', + dataset: HAZARD_URL, + }, + }; +} + +export const aviationHazards = defineAdapter({ + name: 'aviation-hazards', + title: 'Aviation hazards (SIGMET/AIRMET)', + collection: 'aviation', + description: + 'Every SIGMET, AIRMET and centre weather advisory in force over the United States: thunderstorms, turbulence, icing, mountain wave and volcanic ash, each with the polygon it is drawn over and the altitude band it applies to. Keyless, from the NOAA Aviation Weather Center.', + docs: 'https://aviationweather.gov/data/api/', + kinds: ['aviation-hazard'], + cadenceMinutes: 15, + configFields: [ + { + key: 'hazard', + label: 'Only this hazard', + type: 'select', + options: ['', 'conv', 'turb', 'ice', 'ifr', 'mtw'], + help: 'Empty means every hazard in force.', + }, + ], + defaults: {}, + defaultSources: [ + { slug: 'aviation-hazards-us', name: 'Aviation hazards in force (US)' }, + { + slug: 'aviation-hazards-convective', + name: 'Convective SIGMETs', + config: { hazard: 'conv' }, + cadenceMinutes: 10, + }, + ], + async pull({ config, cursor, http, log }) { + const params = new URLSearchParams({ format: 'json' }); + if (config.hazard) params.set('hazard', String(config.hazard)); + const rows = await http.json(`${HAZARD_URL}?${params}`, { timeoutMs: 45_000 }); + if (!Array.isArray(rows)) throw new Error('the aviation weather API did not return a list'); + + const items = rows.map(hazardItem).filter(Boolean); + const newest = + items + .map((i) => i.publishedAt) + .filter(Boolean) + .sort() + .at(-1) ?? cursor.since; + log(`${items.length} hazard(s) in force${newest ? `, newest issued ${newest}` : ''}`); + return { items, cursor: { since: newest ?? null }, note: `${items.length} in force` }; + }, +}); + +/* ------------------------------------------------------------------ METAR */ + +/** + * The busiest US airports, which is what a station list should default to. + * + * Every one of these was in the FAA's own passenger-boardings ranking and + * answers the METAR endpoint. A deployment that wants the whole country sets a + * bounding box instead, and gets roughly two and a half thousand stations an + * hour. + */ +export const HUB_STATIONS = [ + 'KATL', + 'KDFW', + 'KDEN', + 'KORD', + 'KLAX', + 'KCLT', + 'KLAS', + 'KPHX', + 'KMCO', + 'KSEA', + 'KMIA', + 'KIAH', + 'KJFK', + 'KEWR', + 'KFLL', + 'KMSP', + 'KSFO', + 'KDTW', + 'KBOS', + 'KSLC', + 'KPHL', + 'KBWI', + 'KTPA', + 'KSAN', + 'KLGA', + 'KMDW', + 'KBNA', + 'KIAD', + 'KDCA', + 'KAUS', + 'KRDU', + 'KHNL', + 'KSTL', + 'KPDX', + 'KMCI', + 'KSMF', + 'KRSW', + 'KSJC', + 'KSNA', + 'KMSY', + 'KCLE', + 'KPIT', + 'KIND', + 'KCMH', + 'KSAT', + 'KJAX', + 'KOAK', + 'KMKE', + 'KABQ', + 'KBUR', + 'KANC', + 'KOMA', + 'KBUF', + 'KONT', + 'KBDL', + 'KRIC', + 'KTUS', + 'KOKC', + 'KELP', + 'KBOI', +]; + +/** Flight category, ordered worst first, for the "only when it matters" filter. */ +const BELOW_VFR = new Set(['MVFR', 'IFR', 'LIFR']); + +/** + * The most stations one bounding box will answer with. + * + * Measured, not documented. A box covering the continental United States + * returns exactly 400 stations; its eastern half returns 247 and its western + * half 240. 487 stations do not fit in a 400-station answer, and nothing in the + * response says so -- no error, no truncation flag, no count. A source built on + * one national box would therefore have quietly dropped a fifth of the + * country's airports on every run, and looked entirely healthy doing it. + */ +const BOX_CAP = 400; + +/** A bounding box as the API writes it: minLat,minLon,maxLat,maxLon. */ +export function parseBox(raw) { + const parts = String(raw ?? '') + .split(',') + .map((n) => Number(n.trim())); + return parts.length === 4 && parts.every(Number.isFinite) ? parts : null; +} + +/** The four quadrants of a box, for when its answer came back at the cap. */ +export function quarters([minLat, minLon, maxLat, maxLon]) { + const midLat = (minLat + maxLat) / 2; + const midLon = (minLon + maxLon) / 2; + return [ + [minLat, minLon, midLat, midLon], + [minLat, midLon, midLat, maxLon], + [midLat, minLon, maxLat, midLon], + [midLat, midLon, maxLat, maxLon], + ]; +} + +/** The watermark, less the grace a late-reporting station needs to survive it. */ +export function graceBefore(since, hours = 2) { + const t = new Date(since).getTime(); + if (!Number.isFinite(t)) return ''; + return new Date(t - hours * 3_600_000).toISOString(); +} + +export function metarItem(row) { + const station = clean(row.icaoId); + const at = clean(row.reportTime) ?? epoch(row.obsTime); + if (!station || !at) return null; + + const iso = new Date(at).toISOString(); + const cat = clean(row.fltCat); + const temp = num(row.temp); + const wind = num(row.wspd); + const gust = num(row.wgst); + const visib = clean(row.visib); + const name = clean(row.name) ?? station; + + return { + externalId: `metar-${station}-${iso}`, + kind: 'observation', + title: `${station} ${cat ?? 'observation'}: ${name}`, + summary: [ + `${name} at ${iso.replace('T', ' ').slice(0, 16)}Z:`, + cat ? `${cat} conditions,` : null, + visib ? `visibility ${visib} sm,` : null, + wind !== null + ? `wind ${num(row.wdir) === null ? 'variable' : `${num(row.wdir)}°`} at ${wind} kt${ + gust ? ` gusting ${gust}` : '' + },` + : null, + temp !== null ? `temperature ${temp}°C.` : null, + ] + .filter(Boolean) + .join(' ') + .replace(/,$/, '.'), + url: `https://aviationweather.gov/data/metar/?ids=${encodeURIComponent(station)}`, + publishedAt: iso, + timeKnown: true, + precision: 'minute', + tags: [ + 'aviation', + 'weather', + 'observation', + station.toLowerCase(), + cat ? cat.toLowerCase() : null, + cat && BELOW_VFR.has(cat) ? 'below-vfr' : null, + gust ? 'gusting' : null, + ].filter(Boolean), + data: { + station, + stationName: name, + flightCategory: cat, + observedAt: iso, + temperatureC: temp, + dewpointC: num(row.dewp), + windDirectionDeg: num(row.wdir), + windSpeedKt: wind, + windGustKt: gust, + visibilitySm: visib, + altimeterHpa: num(row.altim), + seaLevelPressureHpa: num(row.slp), + cloudLayers: Array.isArray(row.clouds) ? row.clouds : null, + place: { + country: 'US', + lat: num(row.lat), + lon: num(row.lon), + elevationM: num(row.elev), + }, + raw: clean(row.rawOb), + source: 'NOAA Aviation Weather Center', + dataset: METAR_URL, + }, + }; +} + +export const aviationMetar = defineAdapter({ + name: 'aviation-metar', + title: 'Airport weather observations (METAR)', + collection: 'aviation', + description: + 'The hourly observation at an airport, decoded: flight category, wind, visibility, cloud layers and the raw METAR. Sixty US hubs out of the box, or any station list, or a bounding box for every reporting airport inside it. Keyless.', + docs: 'https://aviationweather.gov/data/api/', + kinds: ['observation'], + cadenceMinutes: 60, + configFields: [ + { key: 'stations', label: 'Stations', type: 'list', help: 'ICAO ids, e.g. KJFK.' }, + { + key: 'bbox', + label: 'Bounding box', + help: 'minLat,minLon,maxLat,maxLon — every reporting station inside it. Overrides the station list.', + }, + { + key: 'belowVfrOnly', + label: 'Only below VFR', + type: 'select', + options: ['', 'yes'], + help: 'Keep only stations reporting MVFR, IFR or LIFR — the ones where the weather is in the way.', + }, + ], + defaults: {}, + defaultSources: [ + { slug: 'metar-us-hubs', name: 'Airport weather: 60 US hubs' }, + { + slug: 'metar-below-vfr', + name: 'Airports below VFR (continental US)', + config: { bbox: '24,-125,50,-66', belowVfrOnly: 'yes' }, + cadenceMinutes: 30, + }, + ], + async pull({ config, cursor, http, log, deadline }) { + const ask = async (params) => { + const rows = await http.json(`${METAR_URL}?${params}`, { timeoutMs: 60_000 }); + if (!Array.isArray(rows)) throw new Error('the METAR API did not return a list'); + return rows; + }; + + /** One box, split as many times as it takes for the answer not to be at the cap. */ + const readBox = async (box, depth = 0) => { + const rows = await ask( + new URLSearchParams({ format: 'json', bbox: box.map((n) => n.toFixed(4)).join(',') }), + ); + if (rows.length < BOX_CAP || depth >= 3 || Date.now() > deadline) { + if (rows.length >= BOX_CAP) { + log( + `box ${box.join(',')} still at the ${BOX_CAP}-station cap after splitting; some stations were not read`, + ); + } + return rows; + } + const out = []; + for (const q of quarters(box)) out.push(...(await readBox(q, depth + 1))); + return out; + }; + + const box = parseBox(config.bbox); + let rows; + if (config.bbox && !box) { + throw new Error('aviation-metar bbox must be minLat,minLon,maxLat,maxLon'); + } else if (box) { + const found = await readBox(box); + // Splitting can return the same station from two boxes that share an edge. + const byStation = new Map(); + for (const r of found) byStation.set(`${r.icaoId}-${r.reportTime ?? r.obsTime}`, r); + rows = [...byStation.values()]; + } else { + const stations = (config.stations ?? []) + .map((s) => String(s).trim().toUpperCase()) + .filter(Boolean); + rows = await ask( + new URLSearchParams({ + format: 'json', + ids: (stations.length ? stations : HUB_STATIONS).join(','), + }), + ); + } + + const belowVfrOnly = String(config.belowVfrOnly ?? '') === 'yes'; + const items = rows + .map(metarItem) + .filter(Boolean) + .filter((i) => !belowVfrOnly || BELOW_VFR.has(String(i.data.flightCategory))) + /* Most of a 2,500-station answer is the same observations the last run + * already stored, so the run is cut back to what is new. The watermark is + * the newest report time seen, held back two hours: it is a single number + * across stations that report at different minutes past the hour, and + * without the grace a station whose clock or upload runs late would fall + * behind the watermark on every run and never be stored at all. */ + .filter((i) => !cursor.since || i.publishedAt > graceBefore(cursor.since)); + + const newest = + rows + .map((r) => clean(r.reportTime) ?? epoch(r.obsTime)) + .filter(Boolean) + .map((d) => new Date(d).toISOString()) + .sort() + .at(-1) ?? cursor.since; + + log( + `${items.length} new observation(s) from ${rows.length} station(s)${ + belowVfrOnly ? ' below VFR' : '' + }`, + ); + return { items, cursor: { since: newest ?? null }, note: `${items.length} observations` }; + }, +}); diff --git a/packages/adapters/src/cfpb.js b/packages/adapters/src/cfpb.js new file mode 100644 index 0000000..90cbd4a --- /dev/null +++ b/packages/adapters/src/cfpb.js @@ -0,0 +1,446 @@ +import { defineAdapter, slugify } from '@nichedb/core/adapter'; + +/** + * Every complaint Americans send the Consumer Financial Protection Bureau + * about a bank, a lender, a credit bureau or a debt collector. + * + * 17.6 million of them, around seven thousand a day, published within a day or + * two of being filed and naming the company each one is about. There is no + * other public record of what financial firms are actually doing to their + * customers at this resolution, and it is keyless. + * + * WHAT MAKES IT WORTH A COLLECTION RATHER THAN A SOURCE + * + * A complaint on its own is one person's account. What makes it evidence is the + * company it names, and the company is a bare uppercase string -- `TRANSUNION + * INTERMEDIATE HOLDINGS, INC.` -- with no identifier of any kind attached. Two + * other feeds in this collection turn that string into an institution: + * `fdic-institutions` has every insured bank with its charter, its regulator + * and its assets, and `fdic-structure-changes` has what happened to it since. + * Both normalise the name the same way this one does -- `data.companyKey` here + * and `data.nameKey` there are the same slug, and there is a test that fails if + * they ever stop being -- so the same string can be read as "a $60bn bank + * supervised by the OCC whose Westport branch closed in August", which is the + * question a complaint count is actually asked in aid of. + * + * THE FIELD THAT IS USUALLY EMPTY + * + * `complaint_what_happened` is the consumer's own narrative. Publishing it is + * opt-in, the Bureau scrubs it first, and the scrubbing takes months: of the + * complaints received in the last month, one carries a narrative; of those + * received since May, 42,516 do. So a narrative arrives attached to a complaint + * that is already old, long after any feed following the newest rows has moved + * past it. That is why the narratives source sweeps a trailing window one day + * per run instead of tailing the front, and why the two are separate sources + * rather than one query with a flag. + */ + +const SEARCH = 'https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/'; + +/** + * How many rows one request can return. + * + * The search is Elasticsearch with its default result window, so 10,000 is the + * ceiling on a single answer and there is no way past it: `frm` is accepted, + * echoed and IGNORED -- `frm=0`, `frm=500` and `frm=1000` return byte-identical + * pages -- so anything built on offset paging silently stores the same first + * page over and over and believes it read everything. This adapter therefore + * never pages. It narrows the window until the answer fits. + */ +const MAX_SIZE = 10_000; + +/** The states, for splitting a day that will not fit in one answer. */ +const STATES = [ + 'AL', + 'AK', + 'AZ', + 'AR', + 'CA', + 'CO', + 'CT', + 'DE', + 'DC', + 'FL', + 'GA', + 'HI', + 'ID', + 'IL', + 'IN', + 'IA', + 'KS', + 'KY', + 'LA', + 'ME', + 'MD', + 'MA', + 'MI', + 'MN', + 'MS', + 'MO', + 'MT', + 'NE', + 'NV', + 'NH', + 'NJ', + 'NM', + 'NY', + 'NC', + 'ND', + 'OH', + 'OK', + 'OR', + 'PA', + 'RI', + 'SC', + 'SD', + 'TN', + 'TX', + 'UT', + 'VT', + 'VA', + 'WA', + 'WV', + 'WI', + 'WY', + 'PR', + 'VI', + 'GU', + 'AS', + 'MP', + 'AE', + 'AP', + 'AA', + 'FM', + 'MH', + 'PW', +]; + +/** The UTC day, as the date filters want it. */ +export const utcDay = (d) => new Date(d).toISOString().slice(0, 10); + +/** The day after this one. */ +export function nextDay(day) { + const d = new Date(`${day}T00:00:00Z`); + d.setUTCDate(d.getUTCDate() + 1); + return utcDay(d); +} + +/** Every day from `from` to `to` inclusive, capped. */ +export function daysBetween(from, to, cap) { + const out = []; + let day = from; + while (day <= to && out.length < cap) { + out.push(day); + day = nextDay(day); + } + return out; +} + +const clean = (v) => { + const s = String(v ?? '').trim(); + return s && s.toLowerCase() !== 'null' ? s : null; +}; + +/** The day part of an ISO timestamp, which is the granularity the filter takes. */ +export const day = (iso) => String(iso ?? '').slice(0, 10); + +/** + * How the company answered, in the Bureau's own vocabulary. + * + * `In progress` means the company has not answered yet, and it is the most + * common value on recent complaints for the obvious reason. It is kept as a + * tag rather than folded into "no response", because a complaint filed + * yesterday and one ignored for a year are not the same fact. + */ +/** + * A short name for the product, because the Bureau's own are long enough to be + * cut in half by a tag length and nearly all of them are the same one. + * + * Worth knowing before reading any count from this dataset: 9,843 of the + * 10,000 complaints filed on 7-8 September 2026 were about credit reporting, + * and one was about a mortgage. The database is overwhelmingly a record of + * disputes with the three credit bureaus, and any comparison across products + * that does not say so is misleading. + */ +export const PRODUCT_FAMILIES = [ + [/credit report|consumer report/i, 'credit-reporting'], + [/debt collection/i, 'debt-collection'], + [/mortgage/i, 'mortgage'], + [/credit card|prepaid card/i, 'credit-card'], + [/student loan/i, 'student-loan'], + [/vehicle loan|lease/i, 'auto-loan'], + [/payday|title loan|personal loan|advance loan/i, 'payday-loan'], + [/checking|savings|bank account/i, 'bank-account'], + [/money transfer|virtual currency|money service/i, 'money-transfer'], + [/credit management|debt settlement|debt or credit/i, 'debt-relief'], +]; + +export function productFamily(product) { + const s = String(product ?? ''); + for (const [re, name] of PRODUCT_FAMILIES) if (re.test(s)) return name; + return s ? 'other' : null; +} + +const RESPONSE_TAGS = { + 'closed with explanation': 'explained', + 'closed with non-monetary relief': 'relief-non-monetary', + 'closed with monetary relief': 'relief-monetary', + 'closed without relief': 'no-relief', + 'closed with relief': 'relief', + closed: 'closed', + 'in progress': 'in-progress', + untimely: 'untimely', +}; + +export function toItem(hit) { + const s = hit?._source ?? hit; + const id = clean(s?.complaint_id); + const received = clean(s?.date_received); + const company = clean(s?.company); + const product = clean(s?.product); + if (!id || !received || !company) return null; + + const issue = clean(s.issue); + const subIssue = clean(s.sub_issue); + const subProduct = clean(s.sub_product); + const state = clean(s.state); + const narrative = s.has_narrative ? clean(s.complaint_what_happened) : null; + const response = clean(s.company_response); + + return { + externalId: `cfpb-${id}`, + kind: 'complaint', + title: `${company}: ${issue ?? product ?? 'complaint'}${state ? ` (${state})` : ''}`, + summary: [ + `A consumer${state ? ` in ${state}` : ''} complained to the CFPB about ${company}`, + product + ? ` over ${product.toLowerCase()}${subProduct ? ` (${subProduct.toLowerCase()})` : ''}` + : '', + issue ? `. Issue: ${issue}${subIssue ? ` — ${subIssue}` : ''}` : '', + `. Received ${day(received)}`, + response ? `, company response: ${response.toLowerCase()}` : '', + '.', + narrative ? ` ${narrative.slice(0, 900)}` : '', + ].join(''), + url: `https://www.consumerfinance.gov/data-research/consumer-complaints/search/detail/${id}`, + publishedAt: received, + timeKnown: true, + precision: 'minute', + tags: [ + 'consumer-finance', + 'complaint', + 'us', + slugify(company).slice(0, 60), + productFamily(product), + issue ? slugify(issue).slice(0, 60) : null, + state ? state.toLowerCase() : null, + narrative ? 'has-narrative' : null, + response ? (RESPONSE_TAGS[response.toLowerCase()] ?? slugify(response).slice(0, 30)) : null, + clean(s.timely) === 'No' ? 'untimely-response' : null, + clean(s.submitted_via) ? `via:${slugify(s.submitted_via)}` : null, + ].filter(Boolean), + data: { + complaintId: id, + company, + // The company name exactly as the Bureau writes it, so it can be joined + // to the FDIC feeds in this collection without guessing at the casing. + companyKey: slugify(company), + product, + productFamily: productFamily(product), + subProduct, + issue, + subIssue, + narrative, + hasNarrative: Boolean(s.has_narrative), + narrativeNote: + 'Publishing the consumer’s own account is opt-in and the CFPB scrubs it before release, so most complaints have none. Absent means not published, not that nothing was said.', + receivedAt: received, + sentToCompanyAt: clean(s.date_sent_to_company), + companyResponse: response, + companyPublicResponse: clean(s.company_public_response), + timelyResponse: clean(s.timely), + submittedVia: clean(s.submitted_via), + consumerDisputed: clean(s.consumer_disputed), + place: { country: 'US', state, zip: clean(s.zip_code) }, + source: 'CFPB Consumer Complaint Database', + dataset: SEARCH, + }, + }; +} + +export const cfpbComplaints = defineAdapter({ + name: 'cfpb-complaints', + title: 'CFPB consumer complaints', + collection: 'consumer-finance', + description: + 'Complaints Americans file with the Consumer Financial Protection Bureau about banks, lenders, credit bureaus and debt collectors — around seven thousand a day, each naming the company, the product and the issue. A third of them eventually carry the consumer’s own account of what happened, and those are swept up separately because the Bureau publishes them months after the complaint. Keyless.', + docs: 'https://cfpb.github.io/api/ccdb/', + kinds: ['complaint'], + cadenceMinutes: 60, + configFields: [ + { key: 'product', label: 'Only this product', help: 'e.g. Mortgage, Debt collection.' }, + { key: 'company', label: 'Only this company', help: 'The name exactly as the CFPB writes it.' }, + { key: 'state', label: 'Only this state', help: 'Two-letter code.' }, + { + key: 'narrativesOnly', + label: 'Only with a narrative', + type: 'select', + options: ['', 'yes'], + help: 'Keep only the complaints where the consumer’s own account was published.', + }, + { + key: 'sweepDays', + label: 'Sweep window (days)', + type: 'number', + help: 'Re-read one older day per run over a window this wide, instead of following the newest. Set this for narratives, which are published long after the complaint.', + }, + { key: 'maxDays', label: 'Days per run', type: 'number', help: 'Default 5.' }, + { + key: 'backfillDays', + label: 'Days to read on a first run', + type: 'number', + help: 'Default 2.', + }, + ], + defaults: {}, + defaultSources: [ + { slug: 'cfpb-complaints-all', name: 'CFPB complaints: everything' }, + { + /* + * A narrative is not published with its complaint. The Bureau scrubs it + * first, and the wait is months: complaints received since 10 August + * carry exactly one narrative between them, while the same query from 1 + * May returns 42,516. A feed that followed the newest complaints would + * therefore be permanently empty, so this one sweeps the window where + * narratives actually appear, one day per run. + */ + slug: 'cfpb-complaints-narratives', + name: 'CFPB complaints in the consumer’s own words', + config: { narrativesOnly: 'yes', sweepDays: 180 }, + }, + { + slug: 'cfpb-complaints-debt-collection', + name: 'CFPB complaints about debt collectors', + config: { product: 'Debt collection' }, + cadenceMinutes: 60 * 3, + }, + { + /* + * Everything that is not a credit-report dispute, which is 1.6% of the + * database and the half most people mean when they ask what consumers + * complain about. There is no "not" filter, so it is one source per + * product; these are the four with enough volume to be worth following. + */ + slug: 'cfpb-complaints-bank-accounts', + name: 'CFPB complaints about bank accounts', + config: { product: 'Checking or savings account' }, + cadenceMinutes: 60 * 3, + }, + { + slug: 'cfpb-complaints-credit-cards', + name: 'CFPB complaints about credit cards', + config: { product: 'Credit card' }, + cadenceMinutes: 60 * 3, + }, + ], + async pull({ config, cursor, http, log, deadline }) { + const today = utcDay(Date.now()); + const sweepDays = Math.max(0, Math.min(Number(config.sweepDays) || 0, 3650)); + + /** One day, in as few requests as the result window allows. */ + const readDay = async (dayString) => { + const query = (extra = {}) => { + const params = new URLSearchParams({ + size: String(MAX_SIZE), + no_aggs: 'true', + sort: 'created_date_desc', + date_received_min: dayString, + date_received_max: dayString, + ...extra, + }); + if (config.product) params.set('product', String(config.product)); + if (config.company) params.set('company', String(config.company)); + if (config.state) params.set('state', String(config.state)); + if (String(config.narrativesOnly ?? '') === 'yes') params.set('has_narrative', 'true'); + return http.json(`${SEARCH}?${params}`, { timeoutMs: 180_000 }); + }; + + const body = await query(); + const hits = body?.hits?.hits; + if (!Array.isArray(hits)) throw new Error('the CFPB search did not return hits'); + const total = Number(body?.hits?.total?.value ?? hits.length); + if (total <= MAX_SIZE || config.state) return { hits, total, split: false }; + + /* A day the window cannot hold is re-read state by state. Nothing else + * splits it: the date filters are day-granular, so there is no narrower + * window, and every other facet is dominated by one value. A row with no + * state is picked up by the unsplit read, which is why its hits are kept + * rather than thrown away. */ + log( + `${dayString} has ${total} complaints, over the ${MAX_SIZE} result window; splitting by state`, + ); + const byState = [...hits]; + for (const state of STATES) { + if (Date.now() > deadline) break; + const part = await query({ state }); + const partHits = part?.hits?.hits; + if (Array.isArray(partHits)) byState.push(...partHits); + } + return { hits: byState, total, split: true }; + }; + + const items = []; + const seen = new Set(); + const keep = (hit, floor) => { + const item = toItem(hit); + if (!item || seen.has(item.externalId)) return null; + seen.add(item.externalId); + if (floor && item.publishedAt <= floor) return null; + items.push(item); + return item; + }; + + if (sweepDays) { + /* Sweep mode: one older day per run, walking forward and wrapping at the + * end of the window. Nothing is filtered by a watermark, because the + * point is to pick up rows that changed after they were first read. */ + const start = utcDay(Date.now() - sweepDays * 24 * 3_600_000); + let day = cursor.sweepDay ?? start; + if (day < start || day > today) day = start; + const { hits, total } = await readDay(day); + for (const hit of hits) keep(hit, null); + const nextSweep = nextDay(day) > today ? start : nextDay(day); + log(`swept ${day}: ${items.length} complaint(s) of ${total}, next ${nextSweep}`); + return { + items, + cursor: { ...cursor, sweepDay: nextSweep }, + note: `${items.length} from ${day}`, + }; + } + + const backfill = Math.max(1, Math.min(Number(config.backfillDays) || 2, 365)); + const maxDays = Math.max(1, Math.min(Number(config.maxDays) || 5, 60)); + const since = cursor.since ?? new Date(Date.now() - backfill * 24 * 3_600_000).toISOString(); + const days = daysBetween(day(since), today, maxDays); + + let newest = since; + let read = 0; + for (const d of days) { + if (Date.now() > deadline) { + log(`out of time after ${read} day(s)`); + break; + } + const { hits, total } = await readDay(d); + read += 1; + for (const hit of hits) { + const item = toItem(hit); + if (item && item.publishedAt > newest) newest = item.publishedAt; + keep(hit, cursor.since ?? null); + } + if (d === today) log(`${d}: ${total} complaint(s) so far today`); + } + + log(`${items.length} new complaint(s) across ${read} day(s), newest ${newest}`); + return { items, cursor: { ...cursor, since: newest }, note: `${items.length} complaints` }; + }, +}); diff --git a/packages/adapters/src/coops.js b/packages/adapters/src/coops.js new file mode 100644 index 0000000..0ae5b02 --- /dev/null +++ b/packages/adapters/src/coops.js @@ -0,0 +1,309 @@ +import { defineAdapter } from '@nichedb/core/adapter'; + +/** + * The coast, measured: how high the water actually is at a NOAA tide station, + * against the height at which that station floods. + * + * The river half of this collection has a flood category attached to every + * reading because the National Weather Service assigns one. The coastal half + * does not: `datagetter` returns a water level in feet and nothing else, and + * whether 3.1 ft is a Tuesday or a flooded parking lot depends entirely on the + * station. The thresholds exist, in a different service, one call per station: + * `mdapi`'s `floodlevels` gives the minor, moderate and major heights for that + * gauge. + * + * So the adapter reads the thresholds once and keeps them in its cursor. They + * change when NOAA re-surveys a station, which is a thing that happens every + * few years and not between two polls, so they are refreshed weekly and + * otherwise cost nothing. This is what lets a reading be published as "Battery, + * New York: minor coastal flooding, 10.6 ft, 0.1 ft over minor flood stage" + * rather than as a number with no scale. + * + * WHAT GETS STORED + * + * Every reading at every configured station, not only the flooding ones. A + * coastal water level is a continuous series whose ordinary values are what + * make the extreme ones legible, there are around 300 stations rather than + * 12,000, and each reading is one small row. The rivers adapter makes the + * opposite choice for the opposite reason, and both notes say why. + */ + +const DATA_URL = 'https://api.tidesandcurrents.noaa.gov/api/prod/datagetter'; +const META_URL = 'https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations'; + +/** + * The default station list: one gauge for each stretch of US coast that floods. + * + * Chosen so that a storm anywhere on the Atlantic, Gulf, Pacific or Great Lakes + * shore shows up in at least one of them, and every id was confirmed to answer + * `datagetter` with a water level and `floodlevels` with a threshold. + */ +export const DEFAULT_STATIONS = [ + '8418150', // Portland, ME + '8443970', // Boston, MA + '8461490', // New London, CT + '8510560', // Montauk, NY + '8518750', // The Battery, NY + '8534720', // Atlantic City, NJ + '8557380', // Lewes, DE + '8574680', // Baltimore, MD + '8594900', // Washington, DC + '8638610', // Sewells Point, VA + '8658120', // Wilmington, NC + '8665530', // Charleston, SC + '8670870', // Fort Pulaski, GA + '8720218', // Mayport, FL + '8723214', // Virginia Key, FL + '8724580', // Key West, FL + '8726520', // St Petersburg, FL + '8729108', // Panama City, FL + '8735180', // Dauphin Island, AL + '8761724', // Grand Isle, LA + '8770570', // Sabine Pass, TX + '8771450', // Galveston Pier 21, TX + '8779770', // Port Isabel, TX + '9410170', // San Diego, CA + '9410660', // Los Angeles, CA + '9414290', // San Francisco, CA + '9435380', // South Beach, OR + '9447130', // Seattle, WA + '9455920', // Anchorage, AK + '9751639', // Christiansted, VI + '9759110', // Magueyes Island, PR + '1612340', // Honolulu, HI +]; + +const clean = (v) => { + const s = String(v ?? '').trim(); + return s && s !== 'null' ? s : null; +}; + +export function num(v) { + if (v === null || v === undefined || v === '') return null; + const n = Number(v); + return Number.isFinite(n) ? n : null; +} + +/** Worst first, because a reading over the major threshold is also over the minor one. */ +const LEVELS = [ + ['major', 'major coastal flooding'], + ['moderate', 'moderate coastal flooding'], + ['minor', 'minor coastal flooding'], + ['action', 'above its action stage'], +]; + +/** + * Which flood threshold a reading has passed, if any. + * + * NOAA publishes two sets of thresholds per station, its own (`nos_*`) and the + * National Weather Service's (`nws_*`), and they disagree by a few tenths of a + * foot. The NWS set is preferred because it is the one the flood warnings in + * this deployment's weather collection are written against, so a reading here + * and an alert there agree about whether it is flooding. + */ +export function exceedance(level, thresholds) { + if (level === null || !thresholds) return null; + for (const [key, label] of LEVELS) { + const t = + num(thresholds[`nws_${key}`]) ?? num(thresholds[key]) ?? num(thresholds[`nos_${key}`]); + if (t !== null && level >= t) { + return { category: key, label, threshold: t, over: Number((level - t).toFixed(2)) }; + } + } + return null; +} + +export function toItem({ station, name, state, lat, lon, reading, thresholds, datum, units }) { + const level = num(reading?.v); + const at = clean(reading?.t); + if (level === null || !at) return null; + + /* datagetter writes `2026-09-09 17:06` with no zone marker at all, and it + * answers in station local time or GMT depending on what was asked for. This + * adapter always asks for GMT, so the Z is added here: left off, the stored + * timestamp would be read as whatever zone the reader happened to be in, and + * a Pacific gauge would appear to report eight hours before it did. */ + const when = `${at.replace(' ', 'T')}:00Z`; + const flood = exceedance(level, thresholds); + const unit = units === 'metric' ? 'm' : 'ft'; + + return { + externalId: `coops-${station}-${when}`, + kind: 'water-level', + title: `${name}: ${flood ? flood.label : 'water level'} ${level} ${unit}`, + summary: [ + `${name}${state ? `, ${state}` : ''} measured ${level} ${unit} above ${datum}`, + ` at ${when.replace('T', ' ').replace('Z', '')}Z`, + flood + ? `, which is ${flood.over} ${unit} over its ${flood.category} flood stage of ${flood.threshold} ${unit}.` + : thresholds + ? '. Below every flood threshold for this station.' + : '. No flood thresholds are published for this station.', + ].join(''), + url: `https://tidesandcurrents.noaa.gov/stationhome.html?id=${station}`, + publishedAt: when, + timeKnown: true, + precision: 'minute', + tags: [ + 'water', + 'coastal', + 'us', + 'water-level', + `station:${station}`, + state ? state.toLowerCase() : null, + flood ? `flood:${flood.category}` : null, + flood ? 'flooding' : null, + ].filter(Boolean), + data: { + station, + stationName: name, + waterLevel: level, + unit, + datum, + observedAt: when, + floodCategory: flood?.category ?? null, + floodThreshold: flood?.threshold ?? null, + overThresholdBy: flood?.over ?? null, + thresholds: thresholds ?? null, + thresholdBasis: thresholds + ? 'National Weather Service flood stages for this station where published, NOAA’s own otherwise.' + : null, + place: { country: 'US', state: state ?? null, lat, lon }, + source: 'NOAA Tides and Currents (CO-OPS)', + dataset: DATA_URL, + }, + }; +} + +export const coopsWaterLevels = defineAdapter({ + name: 'coops-water-levels', + title: 'Coastal water levels', + collection: 'water', + description: + 'The observed water level at NOAA tide stations, measured against the height at which each station floods, so a reading arrives as “minor coastal flooding, 0.4 ft over stage” rather than as a bare number. Thirty-two gauges covering every US coast out of the box, or any station list. Keyless.', + docs: 'https://api.tidesandcurrents.noaa.gov/api/prod/', + kinds: ['water-level'], + cadenceMinutes: 30, + configFields: [ + { key: 'stations', label: 'Station ids', type: 'list', help: 'NOAA CO-OPS 7-digit ids.' }, + { + key: 'floodingOnly', + label: 'Only when flooding', + type: 'select', + options: ['', 'yes'], + help: 'Keep only readings at or above a published flood threshold.', + }, + { + key: 'datum', + label: 'Datum', + type: 'select', + options: ['', 'MLLW', 'MHHW', 'NAVD', 'STND'], + help: 'MLLW unless set. The flood thresholds are published against MLLW.', + }, + ], + defaults: {}, + defaultSources: [ + { slug: 'coastal-water-levels', name: 'Coastal water levels: every US coast' }, + { + slug: 'coastal-flooding', + name: 'Coastal flooding only', + config: { floodingOnly: 'yes' }, + cadenceMinutes: 15, + }, + ], + async pull({ config, cursor, http, log, deadline }) { + const stations = (config.stations ?? []).map((s) => String(s).trim()).filter(Boolean); + const list = stations.length ? stations : DEFAULT_STATIONS; + const datum = clean(config.datum) ?? 'MLLW'; + const floodingOnly = String(config.floodingOnly ?? '') === 'yes'; + + /* Thresholds are per station and effectively fixed, so they are read once + * and carried in the cursor. Re-reading them on every run would triple the + * request count for numbers that change when NOAA re-levels a benchmark. */ + const cached = cursor.thresholds ?? {}; + const fetchedAt = cursor.thresholdsAt ?? null; + const stale = !fetchedAt || Date.now() - new Date(fetchedAt).getTime() > 7 * 24 * 3_600_000; + const thresholds = { ...cached }; + + /* The reading itself arrives with a station id, a name and coordinates and + * no state at all, so `The Battery` would be published as a place in no + * country. The station register carries the state and it is one request for + * all 302 of them, so it is read with the thresholds and cached the same + * way rather than asked per station. */ + let stationMeta = cursor.stationMeta ?? {}; + if (stale || !Object.keys(stationMeta).length) { + const index = await http.jsonOrNull(`${META_URL}.json?type=waterlevels`, { + timeoutMs: 60_000, + }); + const rows = Array.isArray(index?.stations) ? index.stations : []; + if (rows.length) { + stationMeta = Object.fromEntries( + rows.map((r) => [ + String(r.id), + { name: clean(r.name), state: clean(r.state), lat: num(r.lat), lng: num(r.lng) }, + ]), + ); + } + } + + const items = []; + let flooding = 0; + for (const station of list) { + if (Date.now() > deadline) { + log(`out of time after ${items.length} station(s)`); + break; + } + if (stale || thresholds[station] === undefined) { + thresholds[station] = + (await http.jsonOrNull(`${META_URL}/${station}/floodlevels.json`, { + timeoutMs: 20_000, + })) ?? null; + } + + const params = new URLSearchParams({ + date: 'latest', + station, + product: 'water_level', + datum, + units: 'english', + time_zone: 'gmt', + format: 'json', + application: 'nichedb', + }); + const body = await http.jsonOrNull(`${DATA_URL}?${params}`, { timeoutMs: 20_000 }); + const reading = body?.data?.[0]; + // A station off line answers 200 with an `error` object rather than a + // status, so an empty `data` is the normal way this fails and is not + // worth a run failure. + if (!reading) continue; + + const meta = stationMeta[station] ?? {}; + const item = toItem({ + station, + name: clean(body?.metadata?.name) ?? meta.name ?? station, + state: meta.state ?? null, + lat: num(body?.metadata?.lat) ?? meta.lat ?? null, + lon: num(body?.metadata?.lon) ?? meta.lng ?? null, + reading, + thresholds: thresholds[station], + datum, + units: 'english', + }); + if (!item) continue; + if (item.data.floodCategory) flooding += 1; + if (floodingOnly && !item.data.floodCategory) continue; + items.push(item); + } + + log(`${items.length} reading(s), ${flooding} at or above a flood threshold`); + return { + items, + cursor: { + thresholds, + stationMeta: stationMeta, + thresholdsAt: stale ? new Date().toISOString() : fetchedAt, + }, + note: `${items.length} readings, ${flooding} flooding`, + }; + }, +}); diff --git a/packages/adapters/src/droughtmonitor.js b/packages/adapters/src/droughtmonitor.js new file mode 100644 index 0000000..e1b5146 --- /dev/null +++ b/packages/adapters/src/droughtmonitor.js @@ -0,0 +1,295 @@ +import { defineAdapter, slugify } from '@nichedb/core/adapter'; + +/** + * The US Drought Monitor: how much of a place is in drought, and how badly, + * redrawn every Thursday. + * + * This is the slow half of the water collection and the reason the collection + * is about water rather than about floods. A river gauge says what is happening + * this hour; the Drought Monitor says what the last six months did to the + * ground that river runs through, which is what decides whether the next storm + * runs off or soaks in. The two are read together or neither is worth much. + * + * WHAT THE NUMBERS MEAN + * + * The API answers with the share of an area in each drought class, and the + * classes are cumulative: `d0` is the share at D0 or worse, `d1` the share at + * D1 or worse, and so on to `d4`, exceptional drought. So `d0: 79, d4: 1.7` + * does not mean 80.7% of the state is in drought -- it means 79% of it is in + * some drought and 1.7% of that is in the worst class there is. Reported here + * as both the cumulative shares the API sends and the share in each class on + * its own, because every mistake anyone makes with this dataset is that one. + * + * WHY BY STATE + * + * The county service exists and would be 3,144 requests for one weekly map; + * the state service is 52 and carries the same story at the resolution a feed + * can be read at. A deployment that wants counties names them in the config and + * gets one row each. + * + * The area of interest is a FIPS number, not a postal code. `aoi=IA` is + * accepted, returns `[]` and looks exactly like a week with no drought in Iowa, + * so the state codes below are numeric and the config field says so. + */ + +const BASE = 'https://usdmdataservices.unl.edu/api'; + +/** State and territory FIPS, which is what the service means by an area of interest. */ +export const STATE_FIPS = { + '01': 'Alabama', + '02': 'Alaska', + '04': 'Arizona', + '05': 'Arkansas', + '06': 'California', + '08': 'Colorado', + '09': 'Connecticut', + 10: 'Delaware', + 11: 'District of Columbia', + 12: 'Florida', + 13: 'Georgia', + 15: 'Hawaii', + 16: 'Idaho', + 17: 'Illinois', + 18: 'Indiana', + 19: 'Iowa', + 20: 'Kansas', + 21: 'Kentucky', + 22: 'Louisiana', + 23: 'Maine', + 24: 'Maryland', + 25: 'Massachusetts', + 26: 'Michigan', + 27: 'Minnesota', + 28: 'Mississippi', + 29: 'Missouri', + 30: 'Montana', + 31: 'Nebraska', + 32: 'Nevada', + 33: 'New Hampshire', + 34: 'New Jersey', + 35: 'New Mexico', + 36: 'New York', + 37: 'North Carolina', + 38: 'North Dakota', + 39: 'Ohio', + 40: 'Oklahoma', + 41: 'Oregon', + 42: 'Pennsylvania', + 44: 'Rhode Island', + 45: 'South Carolina', + 46: 'South Dakota', + 47: 'Tennessee', + 48: 'Texas', + 49: 'Utah', + 50: 'Vermont', + 51: 'Virginia', + 53: 'Washington', + 54: 'West Virginia', + 55: 'Wisconsin', + 56: 'Wyoming', + 72: 'Puerto Rico', +}; + +/** The classes, worst last, with the words the Monitor itself uses. */ +export const CLASSES = [ + ['d0', 'abnormally dry'], + ['d1', 'moderate drought'], + ['d2', 'severe drought'], + ['d3', 'extreme drought'], + ['d4', 'exceptional drought'], +]; + +export function pct(v) { + const n = Number(v); + return Number.isFinite(n) ? Number(n.toFixed(2)) : null; +} + +/** + * The share in each class on its own, from the cumulative shares the API sends. + * + * D4 is already exclusive; every other class is itself minus the next one up. + * Clamped at zero because the source rounds each share independently and two + * rounded numbers can cross by a hundredth. + */ +export function byClass(row) { + const out = {}; + for (let i = 0; i < CLASSES.length; i += 1) { + const [key] = CLASSES[i]; + const here = pct(row[key]) ?? 0; + const worse = i === CLASSES.length - 1 ? 0 : (pct(row[CLASSES[i + 1][0]]) ?? 0); + out[key] = Number(Math.max(0, here - worse).toFixed(2)); + } + return out; +} + +/** The worst class with any area in it, which is the headline. */ +export function worstClass(row) { + for (let i = CLASSES.length - 1; i >= 0; i -= 1) { + const [key, label] = CLASSES[i]; + if ((pct(row[key]) ?? 0) > 0) return { key, label, share: pct(row[key]) }; + } + return null; +} + +export function toItem(row, { area, areaType, fips }) { + const date = String(row.mapDate ?? '').slice(0, 10); + if (!date || !/^\d{4}-\d{2}-\d{2}$/.test(date)) return null; + + const worst = worstClass(row); + const inDrought = pct(row.d1) ?? 0; + const any = pct(row.d0) ?? 0; + const exclusive = byClass(row); + + return { + externalId: `usdm-${areaType}-${fips}-${date}`, + kind: 'drought', + title: worst + ? `${area}: ${any}% abnormally dry or worse, ${inDrought}% in drought, ${worst.share}% ${worst.label}` + : `${area}: no drought`, + summary: worst + ? [ + `${any}% of ${area} was abnormally dry or worse in the US Drought Monitor map of ${date},`, + `${inDrought}% was in moderate drought or worse,`, + `and ${worst.share}% was in ${worst.label}, the worst class the map put it in.`, + ].join(' ') + : `No part of ${area} was abnormally dry in the US Drought Monitor map of ${date}.`, + url: `https://droughtmonitor.unl.edu/CurrentMap/StateDroughtMonitor.aspx?${ + areaType === 'state' ? String(row.stateAbbreviation ?? '').toUpperCase() : 'conus' + }`, + publishedAt: date, + timeKnown: false, + precision: 'day', + tags: [ + 'water', + 'drought', + 'us', + areaType, + String(row.stateAbbreviation ?? '').toLowerCase() || null, + worst ? `drought:${worst.key}` : 'drought:none', + (pct(row.d3) ?? 0) > 0 ? 'extreme-drought' : null, + ].filter(Boolean), + data: { + area, + areaType, + fips, + state: row.stateAbbreviation ?? null, + county: row.county ?? null, + mapDate: date, + validFrom: row.validStart ? String(row.validStart).slice(0, 10) : null, + validTo: row.validEnd ? String(row.validEnd).slice(0, 10) : null, + none: pct(row.none), + cumulative: Object.fromEntries(CLASSES.map(([k]) => [k, pct(row[k])])), + inClass: exclusive, + cumulativeNote: + 'The `cumulative` shares are the area at that class OR WORSE, which is how the Drought Monitor publishes them; `inClass` is the area in that class alone. d0 is abnormally dry, d4 exceptional drought.', + worstClass: worst?.key ?? null, + worstClassLabel: worst?.label ?? null, + source: 'US Drought Monitor (NDMC, USDA, NOAA)', + dataset: `${BASE}/StateStatistics/GetDroughtSeverityStatisticsByAreaPercent`, + }, + }; +} + +export const droughtMonitor = defineAdapter({ + name: 'drought-monitor', + title: 'US Drought Monitor', + collection: 'water', + description: + 'How much of each state is in drought and how badly, from the map the National Drought Mitigation Center, the USDA and NOAA redraw every Thursday. Cumulative shares as published and the share in each class on its own, because conflating the two is the standard mistake with this dataset. Keyless.', + docs: 'https://droughtmonitor.unl.edu/DmData/DataDownload/WebServiceInfo.aspx', + kinds: ['drought'], + cadenceMinutes: 60 * 12, + configFields: [ + { + key: 'areaType', + label: 'Area', + type: 'select', + options: ['state', 'national', 'county'], + help: 'National is one row a week for the country.', + }, + { + key: 'counties', + label: 'County FIPS', + type: 'list', + help: 'Five-digit county FIPS, for areaType=county. Numeric, not postal codes.', + }, + { key: 'weeks', label: 'Weeks to read', type: 'number', help: 'Default 6.' }, + ], + defaults: {}, + defaultSources: [ + { slug: 'drought-by-state', name: 'Drought by state' }, + { + slug: 'drought-national', + name: 'Drought across the country', + config: { areaType: 'national' }, + }, + ], + async pull({ config, cursor, http, log, deadline }) { + const areaType = String(config.areaType ?? 'state').toLowerCase(); + const weeks = Math.max(1, Math.min(Number(config.weeks) || 6, 52)); + const end = new Date(); + const start = new Date(end.getTime() - weeks * 7 * 24 * 3_600_000); + const fmt = (d) => `${d.getUTCMonth() + 1}/${d.getUTCDate()}/${d.getUTCFullYear()}`; + + const targets = + areaType === 'national' + ? [ + { + service: 'USStatistics', + aoi: 'us', + area: 'the continental United States', + fips: 'us', + }, + ] + : areaType === 'county' + ? (config.counties ?? []) + .map((c) => String(c).trim()) + .filter(Boolean) + .map((fips) => ({ + service: 'CountyStatistics', + aoi: fips, + area: `county ${fips}`, + fips, + })) + : Object.entries(STATE_FIPS).map(([fips, area]) => ({ + service: 'StateStatistics', + aoi: fips, + area, + fips, + })); + + if (!targets.length) throw new Error('drought-monitor has no area to read'); + + const items = []; + for (const t of targets) { + if (Date.now() > deadline) { + log(`out of time after ${items.length} row(s)`); + break; + } + const url = + `${BASE}/${t.service}/GetDroughtSeverityStatisticsByAreaPercent` + + `?aoi=${encodeURIComponent(t.aoi)}&startdate=${fmt(start)}&enddate=${fmt(end)}&statisticsType=1`; + const rows = await http.json(url, { + headers: { accept: 'application/json' }, + timeoutMs: 30_000, + }); + if (!Array.isArray(rows)) continue; + for (const row of rows) { + /* The national service answers with several areas of interest in one + * list -- CONUS, Total, and the territories -- so the row's own label + * is used where it has one rather than the name that was asked for. */ + const area = row.areaOfInterest ?? row.county ?? t.area; + const fips = row.fips ?? t.fips; + const item = toItem(row, { area, areaType, fips: `${fips}-${slugify(area)}` }); + if (item) items.push(item); + } + } + + log(`${items.length} drought row(s) across ${targets.length} area(s)`); + return { + items, + cursor: { ...cursor, lastRun: new Date().toISOString() }, + note: `${items.length} rows`, + }; + }, +}); diff --git a/packages/adapters/src/faanas.js b/packages/adapters/src/faanas.js new file mode 100644 index 0000000..5a37780 --- /dev/null +++ b/packages/adapters/src/faanas.js @@ -0,0 +1,339 @@ +import { defineAdapter, xmlItems } from '@nichedb/core/adapter'; + +/** + * The FAA's own view of why the national airspace is running late. + * + * `nasstatus.faa.gov` publishes, every couple of minutes, the traffic + * management initiatives in force right now: ground stops, ground delay + * programs, airspace flow programs, collaborative trajectory options programs, + * airport closures and the plain arrival/departure delays at airports running + * behind. It is the document the airlines read, it is keyless, and it is about + * two kilobytes. + * + * WHAT AN ITEM IS HERE + * + * The feed is a snapshot, not a log: it says what is in force, never what was. + * A ground stop that ran for two hours appears in maybe forty consecutive + * snapshots and then simply stops appearing, and nothing anywhere records that + * it ended. So this adapter keeps the open programs in its cursor with the time + * each was first seen, and: + * + * * an item's external id is keyed on that first-seen time, so one program is + * one row that updates as the delay grows, rather than a new row every two + * minutes; + * * when a program leaves the snapshot it is emitted once more, with + * `endedAt` and a duration, and then dropped from the cursor. + * + * That last emission is the whole point. "BOS ground stop, 3h 40m, thunder- + * storms" is a fact about a day of flying that the FAA publishes nowhere; it + * only exists if something was watching the snapshot the entire time. + * + * The reason string is the FAA's own wording, kept verbatim, because the reason + * is what makes this feed worth joining to the weather: `aviation-weather` has + * the METAR and the SIGMET for the same airport at the same minute, so "weather + * / thunderstorms" at ORD can be read next to the observation that says why. + */ + +/** Programs, in the shape the snapshot writes them. */ +const PROGRAMS = [ + { + tag: 'Program', + kind: 'ground-stop', + label: 'Ground stop', + airport: (f) => text(f.ARPT), + detail: (f) => ({ endTime: text(f.End_Time) || null }), + line: (f) => (text(f.End_Time) ? `until ${text(f.End_Time)}` : null), + }, + { + tag: 'Ground_Delay', + kind: 'ground-delay', + label: 'Ground delay program', + airport: (f) => text(f.ARPT), + detail: (f) => ({ averageDelay: text(f.Avg) || null, maximumDelay: text(f.Max) || null }), + line: (f) => (text(f.Avg) ? `averaging ${text(f.Avg)}, up to ${text(f.Max)}` : null), + }, + { + tag: 'Delay', + kind: 'airport-delay', + label: 'Departure and arrival delays', + airport: (f) => text(f.ARPT), + detail: (f) => ({ legs: legsOf(f) }), + line: (f) => + legsOf(f) + .map( + (l) => + `${l.type.toLowerCase()}s ${l.min}${l.max ? ` to ${l.max}` : ''}${l.trend ? `, ${l.trend}` : ''}`, + ) + .join('; ') || null, + }, + { + tag: 'Airport', + kind: 'airport-closure', + label: 'Airport closure', + airport: (f) => text(f.ARPT), + detail: (f) => ({ start: text(f.Start) || null, reopen: text(f.Reopen) || null }), + line: (f) => (text(f.Reopen) ? `reopens ${text(f.Reopen)}` : null), + }, + { + tag: 'Airspace_Flow', + kind: 'airspace-flow', + label: 'Airspace flow program', + airport: (f) => text(f.CTL_Element), + detail: (f) => ({ + averageDelay: text(f.Avg) || null, + start: text(f.AFP_StartTime) || text(f.FCA_Start_DateTime) || null, + end: text(f.AFP_EndTime) || text(f.FCA_End_DateTime) || null, + }), + line: (f) => (text(f.Avg) ? `averaging ${text(f.Avg)}` : null), + }, + { + tag: 'CTOP', + kind: 'trajectory-options', + label: 'Collaborative trajectory options program', + airport: (f) => text(f.Program_Name), + detail: (f) => ({ + averageDelay: text(f.Avg) || null, + start: text(f.CTOP_Start_Time) || null, + end: text(f.CTOP_End_Time) || null, + }), + line: (f) => (text(f.Avg) ? `averaging ${text(f.Avg)}` : null), + }, +]; + +const text = (field) => String((Array.isArray(field) ? field[0] : field)?.text ?? '').trim(); + +/** + * The arrival and departure legs of one delay entry. + * + * `Delay` carries one or two `Arrival_Departure` children distinguished only by + * a `Type` attribute, so an airport delayed in both directions is a single row + * with two very different numbers in it. + * + * The `Min`/`Max`/`Trend` inside a leg arrive as that leg's unparsed body -- + * the XML reader in core stops at the outermost tag it matched rather than + * descending -- so they are read out of the text here. + */ +export function legsOf(fields) { + const raw = fields.Arrival_Departure; + const list = raw === undefined ? [] : Array.isArray(raw) ? raw : [raw]; + const child = (body, tag) => + new RegExp(`<${tag}>([\\s\\S]*?)`).exec(String(body ?? ''))?.[1]?.trim() ?? ''; + return list + .map((leg) => ({ + type: String(leg?.attrs?.Type ?? '').trim() || 'Arrival', + min: child(leg?.text, 'Min'), + max: child(leg?.text, 'Max'), + trend: child(leg?.text, 'Trend'), + })) + .filter((l) => l.min || l.max); +} + +/** + * Read one snapshot into the programs it holds. + * + * Parsed per program tag over the whole document rather than by walking + * `Delay_type` blocks, because the FAA emits the same block name more than once + * in a single snapshot -- two separate `Airport Closures` sections is normal, + * one for airports shut outright and one for airports shut to transient general + * aviation -- and a reader that assumed one block per type would silently keep + * whichever came last. + */ +export function parseStatus(xml) { + const updated = /([^<]+)<\/Update_Time>/.exec(xml)?.[1]?.trim() ?? null; + const out = []; + for (const p of PROGRAMS) { + for (const fields of xmlItems(xml, p.tag)) { + // `Delay` also matches `Ground_Delay`'s inner text in a document where the + // two nest; requiring the identifying field keeps only the real ones. + const where = p.airport(fields); + if (!where) continue; + const reason = text(fields.Reason); + out.push({ + key: `${p.kind}:${where}`, + kind: p.kind, + label: p.label, + where, + reason: reason || null, + line: p.line(fields), + detail: p.detail(fields), + }); + } + } + return { updated, programs: dedupe(out) }; +} + +/** One row per program per place: the same airport twice in a snapshot is the snapshot repeating itself. */ +function dedupe(programs) { + const seen = new Map(); + for (const p of programs) if (!seen.has(p.key)) seen.set(p.key, p); + return [...seen.values()]; +} + +/** How long a program has been in force, in the words a person would use. */ +export function duration(fromISO, toISO) { + const ms = new Date(toISO).getTime() - new Date(fromISO).getTime(); + if (!Number.isFinite(ms) || ms < 0) return null; + const mins = Math.round(ms / 60_000); + if (mins < 60) return `${mins}m`; + const h = Math.floor(mins / 60); + const m = mins % 60; + return m ? `${h}h ${m}m` : `${h}h`; +} + +export function toItem(program, { firstSeen, now, ended = false }) { + const ran = ended ? duration(firstSeen, now) : null; + return { + externalId: `faa-${program.kind}-${program.where}-${firstSeen}`, + kind: program.kind, + title: `${program.label}: ${program.where}${program.reason ? ` — ${program.reason}` : ''}${ + ended && ran ? ` (ended after ${ran})` : '' + }`, + summary: [ + `${program.label} at ${program.where}`, + program.reason ? `because of ${program.reason}` : null, + program.line ? `(${program.line})` : null, + ended + ? `. In force from ${firstSeen} to ${now}${ran ? `, ${ran}` : ''}.` + : `. In force since ${firstSeen}, still listed at ${now}.`, + ] + .filter(Boolean) + .join(' ') + .replace(' .', '.'), + url: 'https://nasstatus.faa.gov/', + publishedAt: firstSeen, + timeKnown: true, + precision: 'minute', + tags: [ + 'aviation', + 'us', + 'faa', + program.kind, + program.where.toLowerCase(), + ended ? 'ended' : 'in-force', + ...reasonTags(program.reason), + ].filter(Boolean), + data: { + programType: program.kind, + programLabel: program.label, + airport: program.where, + reason: program.reason, + ...program.detail, + firstSeenAt: firstSeen, + lastSeenAt: now, + endedAt: ended ? now : null, + durationHuman: ran, + status: ended ? 'ended' : 'in-force', + statusNote: ended + ? 'The FAA snapshot stopped listing this program between the previous poll and this one, so it ended some time in that window rather than exactly at endedAt.' + : 'Still listed in the FAA snapshot at lastSeenAt.', + source: 'FAA National Airspace System status', + dataset: 'https://nasstatus.faa.gov/api/airport-status-information', + }, + }; +} + +/** The coarse cause, so "every weather ground stop this month" is one query. */ +export function reasonTags(reason) { + const s = String(reason ?? '').toLowerCase(); + const tags = []; + if (/weather|thunder|snow|fog|wind|ice|rain|low ceiling|visibility/.test(s)) tags.push('weather'); + if (/thunderstorm|convective/.test(s)) tags.push('thunderstorms'); + if (/volume|traffic management/.test(s)) tags.push('traffic-volume'); + if (/equipment|radar|outage/.test(s)) tags.push('equipment'); + if (/runway|taxiway|construction|maintenance/.test(s)) tags.push('runway'); + if (/staffing|controller/.test(s)) tags.push('staffing'); + if (/disabled aircraft|accident|incident|emergency/.test(s)) tags.push('incident'); + return tags; +} + +export const faaAirportStatus = defineAdapter({ + name: 'faa-nas-status', + title: 'FAA airspace status', + collection: 'aviation', + description: + 'Ground stops, ground delay programs, airspace flow programs, airport closures and the airports running behind, as the FAA lists them in force. Each program is one row that updates while it lasts and is written once more when it ends, with how long it ran — which is the part the FAA itself never publishes. Keyless.', + docs: 'https://nasstatus.faa.gov/', + kinds: [ + 'ground-stop', + 'ground-delay', + 'airport-delay', + 'airport-closure', + 'airspace-flow', + 'trajectory-options', + ], + cadenceMinutes: 5, + configFields: [ + { + key: 'airports', + label: 'Only these airports', + type: 'list', + help: 'Three-letter FAA codes. Empty means every airport in the snapshot.', + }, + ], + defaults: {}, + defaultSources: [{ slug: 'faa-nas-status', name: 'FAA airspace status: the whole country' }], + async pull({ config, cursor, http, log }) { + const xml = await http.text('https://nasstatus.faa.gov/api/airport-status-information', { + headers: { accept: 'application/xml, text/xml, */*' }, + timeoutMs: 30_000, + }); + const { updated, programs } = parseStatus(xml); + const now = new Date().toISOString(); + + const only = (config.airports ?? []).map((a) => String(a).trim().toUpperCase()).filter(Boolean); + const wanted = only.length ? programs.filter((p) => only.includes(p.where)) : programs; + + const open = { ...(cursor.open ?? {}) }; + const items = []; + + for (const p of wanted) { + const firstSeen = open[p.key] ?? now; + open[p.key] = firstSeen; + items.push(toItem(p, { firstSeen, now })); + } + + /* Whatever was open last time and is not in this snapshot has ended. It is + * written once with its duration and then forgotten -- keeping it would + * re-emit the same ended program on every run forever. */ + const live = new Set(wanted.map((p) => p.key)); + for (const [key, firstSeen] of Object.entries(cursor.open ?? {})) { + if (live.has(key)) continue; + delete open[key]; + // Split once: a CTOP's program name is free text and may contain a colon. + const at = key.indexOf(':'); + const kind = key.slice(0, at); + const where = key.slice(at + 1); + const spec = PROGRAMS.find((p) => p.kind === kind); + items.push( + toItem( + { + key, + kind, + label: spec?.label ?? kind, + where, + reason: cursor.reasons?.[key] ?? null, + line: null, + detail: {}, + }, + { firstSeen, now, ended: true }, + ), + ); + } + + // The reason a program gave while it was open, kept so the closing row can + // still say why it happened after the snapshot has stopped saying so. + const reasons = {}; + for (const p of wanted) if (p.reason) reasons[p.key] = p.reason; + + log( + `${wanted.length} program(s) in force${updated ? ` as of ${updated}` : ''}, ${ + items.length - wanted.length + } ended`, + ); + return { + items, + cursor: { open, reasons, updated }, + note: `${wanted.length} in force`, + }; + }, +}); diff --git a/packages/adapters/src/fdic.js b/packages/adapters/src/fdic.js new file mode 100644 index 0000000..5731a51 --- /dev/null +++ b/packages/adapters/src/fdic.js @@ -0,0 +1,422 @@ +import { defineAdapter, slugify } from '@nichedb/core/adapter'; + +/** + * Who the bank in the complaint actually is, and what has happened to it. + * + * The FDIC's BankFind API is the register of every US insured institution -- + * 4,235 active ones, their charter class, their regulator, their assets and + * their address -- plus two event streams over it: 584,000 structure changes + * (mergers, acquisitions, name changes, relocations, charter conversions) and + * every bank failure since 1934. Keyless, JSON, and updated daily. + * + * It is here because `cfpb-complaints` names companies and identifies none of + * them. A complaint against a bank that was absorbed eighteen months ago is a + * complaint against whoever absorbed it, and the structure-change feed is the + * only public record of that. Consolidation is also the story: roughly a + * hundred and fifty US banks disappear into other banks every year, which is + * why a directory of them is worth keeping rather than a snapshot. + * + * TWO ADAPTERS, ONE UPSTREAM + * + * The register and the events are different shapes and different cadences, so + * they are separate adapters over the same host rather than one adapter with a + * mode switch: an institution is a thing that exists, an acquisition is + * something that happened on a date, and a feed of the first sorted by date + * would be meaningless. + */ + +const BASE = 'https://api.fdic.gov/banks'; + +const clean = (v) => { + const s = String(v ?? '').trim(); + return s && s.toLowerCase() !== 'null' ? s : null; +}; + +/** + * A dollar figure from a field the API reports in thousands. + * + * Guarded before the cast, because `Number(null)` is 0 and a bank with no + * assets reported would otherwise be published as a bank with no assets. + */ +export function thousands(v) { + if (v === null || v === undefined || v === '') return null; + const n = Number(v); + return Number.isFinite(n) ? n * 1000 : null; +} + +export function money(n) { + if (!Number.isFinite(n)) return null; + if (n >= 1e12) return `$${(n / 1e12).toFixed(2)}tn`; + if (n >= 1e9) return `$${(n / 1e9).toFixed(2)}bn`; + if (n >= 1e6) return `$${(n / 1e6).toFixed(1)}m`; + return `$${Math.round(n).toLocaleString()}`; +} + +/** + * The FDIC writes dates as `7/17/2026` in some fields and `20260717` in others. + * Both, and an ISO one, reach here. + */ +export function fdicDate(raw) { + const s = String(raw ?? '').trim(); + if (!s || s === '0') return null; + const iso = /^(\d{4})-(\d{2})-(\d{2})/.exec(s); + if (iso) return `${iso[1]}-${iso[2]}-${iso[3]}`; + const packed = /^(\d{4})(\d{2})(\d{2})$/.exec(s); + if (packed) return `${packed[1]}-${packed[2]}-${packed[3]}`; + const us = /^(\d{1,2})\/(\d{1,2})\/(\d{4})$/.exec(s); + if (us) { + const p = (n) => String(n).padStart(2, '0'); + return `${us[3]}-${p(us[1])}-${p(us[2])}`; + } + return null; +} + +/* ----------------------------------------------------------- institutions */ + +export function institutionItem(row) { + const d = row?.data ?? row; + const cert = clean(d?.CERT); + const name = clean(d?.NAME); + if (!cert || !name) return null; + + const assets = thousands(d.ASSET); + const established = fdicDate(d.ESTYMD); + const active = String(d.ACTIVE ?? '') === '1'; + const city = clean(d.CITY); + const state = clean(d.STALP); + const regulator = clean(d.REGAGNT); + /* The charter class is `BKCLASS`, not `CLASS`: BankFind's institution + * endpoint has no `CLASS` field at all, so a reader that asked for one would + * quietly file every bank in the country as unclassified. */ + const bankClass = clean(d.BKCLASS); + const changed = fdicDate(d.RUNDATE) ?? fdicDate(d.PROCDATE) ?? established; + + return { + externalId: `fdic-cert-${cert}`, + kind: 'institution', + title: + `${name}${city ? ` — ${city}, ${state ?? ''}` : ''}${assets ? ` (${money(assets)})` : ''}`.trim(), + summary: [ + `${name} is an FDIC-insured ${bankClass ? `${bankClass} ` : ''}institution`, + city ? ` in ${city}${state ? `, ${state}` : ''}` : '', + assets ? `, holding ${money(assets)} in assets` : '', + established ? `, established ${established}` : '', + regulator ? `, regulated by ${regulator}` : '', + active ? '.' : '. It is no longer active.', + ].join(''), + url: `https://banks.data.fdic.gov/bankfind-suite/bankfind/details/${cert}`, + publishedAt: changed ?? established, + timeKnown: false, + precision: 'day', + tags: [ + 'consumer-finance', + 'bank', + 'us', + slugify(name).slice(0, 60), + state ? state.toLowerCase() : null, + bankClass ? `class:${slugify(bankClass)}` : null, + regulator ? `regulator:${slugify(regulator)}` : null, + active ? 'active' : 'inactive', + assets && assets >= 1e11 ? 'assets:100bn-plus' : null, + assets && assets >= 1e10 && assets < 1e11 ? 'assets:10bn-plus' : null, + ].filter(Boolean), + data: { + cert, + name, + nameKey: slugify(name), + active, + class: bankClass, + specialisation: clean(d.SPECGRPN), + regulator, + charterClass: clean(d.CHARTER), + assets, + deposits: thousands(d.DEP), + equity: thousands(d.EQ), + offices: Number(d.OFFICES) || null, + establishedOn: established, + insuredOn: fdicDate(d.EFFDATE), + website: clean(d.WEBADDR), + holdingCompany: clean(d.NAMEHCR), + place: { + country: 'US', + state, + stateName: clean(d.STNAME), + city, + address: clean(d.ADDRESS), + zip: clean(d.ZIP), + county: clean(d.COUNTY), + }, + source: 'FDIC BankFind', + dataset: `${BASE}/institutions`, + }, + }; +} + +export const fdicInstitutions = defineAdapter({ + name: 'fdic-institutions', + title: 'FDIC insured institutions', + collection: 'consumer-finance', + description: + 'Every FDIC-insured bank and thrift: charter class, regulator, assets, deposits, holding company and address, keyed on the certificate number the FDIC identifies it by. The register that turns a company name in a complaint into an institution. Keyless.', + docs: 'https://api.fdic.gov/banks/docs/', + kinds: ['institution'], + cadenceMinutes: 60 * 24, + configFields: [ + { + key: 'activeOnly', + label: 'Active only', + type: 'select', + options: ['yes', ''], + help: 'Empty includes institutions that have closed.', + }, + { key: 'state', label: 'Only this state', help: 'Two-letter code.' }, + { key: 'maxPages', label: 'Pages per run', type: 'number', help: '1,000 a page. Default 6.' }, + ], + defaults: { activeOnly: 'yes' }, + defaultSources: [{ slug: 'fdic-institutions', name: 'FDIC insured institutions' }], + async pull({ config, cursor, http, log, deadline }) { + const limit = 1000; + const maxPages = Math.max(1, Math.min(Number(config.maxPages) || 6, 20)); + const filters = []; + if (String(config.activeOnly ?? 'yes') === 'yes') filters.push('ACTIVE:1'); + if (config.state) filters.push(`STALP:${String(config.state).toUpperCase()}`); + + const items = []; + let total = 0; + for (let page = 0; page < maxPages; page += 1) { + if (Date.now() > deadline) break; + const params = new URLSearchParams({ + limit: String(limit), + offset: String(page * limit), + format: 'json', + sort_by: 'CERT', + sort_order: 'ASC', + }); + if (filters.length) params.set('filters', filters.join(' AND ')); + const body = await http.json(`${BASE}/institutions?${params}`, { timeoutMs: 60_000 }); + const rows = body?.data; + if (!Array.isArray(rows)) throw new Error('BankFind did not return a data array'); + total = Number(body?.meta?.total) || total; + for (const row of rows) { + const item = institutionItem(row); + if (item) items.push(item); + } + if (rows.length < limit) break; + } + + log(`${items.length} institution(s) of ${total || 'unknown'} in the register`); + return { + items, + cursor: { ...cursor, readAt: new Date().toISOString() }, + note: `${items.length} banks`, + }; + }, +}); + +/* ------------------------------------------------------ structure changes */ + +/** + * What the FDIC calls the event, coarsened into something a feed can be + * filtered on. + * + * The register does publish a "what happened" field -- `CHANGECODE` with a + * `CHANGECODE_DESC` beside it -- and it is the field to read. The boolean + * flags that also ride along on every row (`RELOCATE_FLAG`, `NEW_CHARTER_FLAG` + * and twenty more) are all zero on the great majority of transactions, + * including on mergers, so a reader built on them would report almost nothing + * happening while the register recorded five thousand events a year. + * + * The leading digit is the family: 2xx a merger or absorption, 3xx and 4xx a + * change of membership, charter or regulator, 5xx a change of name or address, + * 7xx something happening to a branch, 8xx participation in someone else's + * reorganisation. Most of the volume is 7xx, which is the interesting part: + * branch openings and closings are where a bank's retreat from a town is + * visible years before anything is written about it. + */ +export const CHANGE_FAMILIES = [ + [/^1/, 'establishment'], + [/^2/, 'merger'], + [/^3/, 'membership'], + [/^4/, 'charter'], + [/^5/, 'name-or-location'], + [/^6/, 'closing'], + [/^7/, 'branch'], + [/^8/, 'reorganisation'], +]; + +/** The handful of codes worth their own tag, because they are the ones people search for. */ +export const NOTABLE_CODES = { + 711: 'branch-opening', + 712: 'branch-purchased', + 713: 'branch-acquired-in-merger', + 721: 'branch-closing', + 722: 'branch-sold', + 223: 'merger-without-assistance', + 224: 'affiliated-merger', + 510: 'name-change', + 520: 'relocation', + 430: 'class-change', + 470: 'regulator-change', + 810: 'absorbed', + 820: 'corporate-reorganisation', +}; + +export function familyOf(code) { + const s = String(code ?? '').trim(); + for (const [re, name] of CHANGE_FAMILIES) if (re.test(s)) return name; + return 'other'; +} + +export function changeItem(row) { + const d = row?.data ?? row; + const trans = clean(d?.TRANSNUM); + const name = clean(d?.INSTNAME); + const when = fdicDate(d?.PROCDATE) ?? fdicDate(d?.EFFDATE); + if (!trans || !name || !when) return null; + + /* A transaction number is NOT a row: one merger writes a row for every + * institution and every office it touches, and 1,702 changes read in one run + * carried only 1,169 distinct transaction numbers. Keyed on TRANSNUM alone, a + * third of the register would overwrite the other two thirds and the loss + * would look exactly like a quiet quarter. `ID` is the row, and where the + * register omits it the office and institution numbers rebuild one. */ + const rowId = + clean(d?.ID) ?? + [trans, clean(d?.UNINUM), clean(d?.OFF_NUM), clean(d?.CHANGECODE)].filter(Boolean).join('_'); + + const code = clean(d.CHANGECODE); + const what = clean(d.CHANGECODE_DESC) ?? 'Change on the FDIC register'; + const family = familyOf(code); + const branch = clean(d.OFF_NAME); + const formerBranch = clean(d.FRM_OFF_NAME); + const city = clean(d.OFF_PCITY) ?? clean(d.PCITY); + const state = clean(d.OFF_PSTALP) ?? clean(d.PSTALP); + const where = [city, state].filter(Boolean).join(', '); + // A branch event is about the branch; an institution event is about the bank. + const subject = family === 'branch' && branch ? `${name} — ${branch}` : name; + + return { + externalId: `fdic-change-${rowId}`, + kind: 'structure-change', + title: `${subject}${where ? `, ${where}` : ''}: ${what}`, + summary: [ + `${what} recorded by the FDIC for ${name}`, + branch ? `, ${branch}` : '', + where ? ` in ${where}` : '', + `. Effective ${fdicDate(d.EFFDATE) ?? when}, processed ${when}.`, + formerBranch && formerBranch !== branch ? ` Previously ${formerBranch}.` : '', + ].join(''), + url: clean(d.CERT) + ? `https://banks.data.fdic.gov/bankfind-suite/bankfind/details/${clean(d.CERT)}` + : 'https://banks.data.fdic.gov/bankfind-suite/', + publishedAt: when, + timeKnown: false, + precision: 'day', + tags: [ + 'consumer-finance', + 'bank', + 'us', + 'structure-change', + slugify(name).slice(0, 60), + state ? state.toLowerCase() : null, + city ? slugify(city).slice(0, 40) : null, + family, + code && NOTABLE_CODES[code] ? NOTABLE_CODES[code] : null, + code ? `code:${code}` : null, + ].filter(Boolean), + data: { + rowId, + transactionNumber: trans, + cert: clean(d.CERT), + name, + nameKey: slugify(name), + changeCode: code, + change: what, + changeFamily: family, + changeBasis: + 'CHANGECODE and CHANGECODE_DESC are the FDIC’s own classification; changeFamily is a coarsening of the code’s leading digit so a family can be filtered on.', + branch, + formerBranch, + effectiveOn: fdicDate(d.EFFDATE), + processedOn: fdicDate(d.PROCDATE), + endedOn: fdicDate(d.ENDDATE), + class: clean(d.CLASS_TYPE_DESC), + formerClass: clean(d.FRM_CLASS_TYPE_DESC), + regulator: clean(d.REGAGENT), + formerRegulator: clean(d.FRM_REGAGENT), + place: { + country: 'US', + state, + city, + county: clean(d.OFF_CNTYNAME) ?? clean(d.CNTYNAME), + address: clean(d.OFF_PADDR) ?? clean(d.PADDR), + zip: clean(d.OFF_PZIP5) ?? clean(d.PZIP5), + lat: Number(d.OFF_LATITUDE) || Number(d.LATITUDE) || null, + lon: Number(d.OFF_LONGITUDE) || Number(d.LONGITUDE) || null, + }, + source: 'FDIC BankFind structure changes', + dataset: `${BASE}/history`, + }, + }; +} + +export const fdicStructureChanges = defineAdapter({ + name: 'fdic-structure-changes', + title: 'FDIC bank structure changes', + collection: 'consumer-finance', + description: + 'Mergers, acquisitions, failures, charter conversions, name changes and relocations on the FDIC register, as they are processed — the public record of American bank consolidation, and the only way to know which institution a complaint about a bank that no longer exists now belongs to. Keyless.', + docs: 'https://api.fdic.gov/banks/docs/', + kinds: ['structure-change'], + cadenceMinutes: 60 * 6, + configFields: [ + { key: 'state', label: 'Only this state', help: 'Two-letter code.' }, + { key: 'maxPages', label: 'Pages per run', type: 'number', help: '1,000 a page. Default 3.' }, + { key: 'days', label: 'Days to read on a first run', type: 'number', help: 'Default 90.' }, + ], + defaults: {}, + defaultSources: [ + { slug: 'fdic-structure-changes', name: 'FDIC bank mergers, failures and conversions' }, + ], + async pull({ config, cursor, http, log, deadline }) { + const limit = 1000; + const maxPages = Math.max(1, Math.min(Number(config.maxPages) || 3, 20)); + const days = Math.max(1, Math.min(Number(config.days) || 90, 3650)); + const since = + cursor.since ?? new Date(Date.now() - days * 24 * 3_600_000).toISOString().slice(0, 10); + + /* PROCDATE is when the FDIC recorded the change and is the only field that + * moves monotonically -- EFFDATE is when it took effect and can be + * backdated by months, so a feed keyed on it would miss transactions that + * arrive late. The filter is a range because BankFind has no ">" operator. */ + const filters = [`PROCDATE:[${since.replace(/-/g, '')} TO 99991231]`]; + if (config.state) filters.push(`PSTALP:${String(config.state).toUpperCase()}`); + + const items = []; + let newest = since; + for (let page = 0; page < maxPages; page += 1) { + if (Date.now() > deadline) break; + const params = new URLSearchParams({ + filters: filters.join(' AND '), + limit: String(limit), + offset: String(page * limit), + format: 'json', + sort_by: 'PROCDATE', + sort_order: 'DESC', + }); + const body = await http.json(`${BASE}/history?${params}`, { timeoutMs: 60_000 }); + const rows = body?.data; + if (!Array.isArray(rows)) throw new Error('BankFind history did not return a data array'); + for (const row of rows) { + const item = changeItem(row); + if (!item) continue; + if (item.publishedAt > newest) newest = item.publishedAt; + items.push(item); + } + if (rows.length < limit) break; + } + + log(`${items.length} structure change(s) since ${since}, newest ${newest}`); + return { items, cursor: { since: newest }, note: `${items.length} changes` }; + }, +}); diff --git a/packages/adapters/src/index.js b/packages/adapters/src/index.js index 81b36b3..250abcb 100644 --- a/packages/adapters/src/index.js +++ b/packages/adapters/src/index.js @@ -2,17 +2,23 @@ import { aiid } from './aiid.js'; import { alpacaCorporateActions, alpacaNews } from './alpaca.js'; import { firefoxAddons } from './amo.js'; import { arxiv } from './arxiv.js'; +import { aviationHazards, aviationMetar } from './aviationweather.js'; import { blsSeries } from './bls.js'; import { brisk } from './brisk.js'; +import { cfpbComplaints } from './cfpb.js'; import { clinicalTrials } from './clinicaltrials.js'; +import { coopsWaterLevels } from './coops.js'; import { courtlistener } from './courtlistener.js'; import { crates } from './crates.js'; import { crossref } from './crossref.js'; +import { droughtMonitor } from './droughtmonitor.js'; import { ecbFxRates } from './ecb.js'; import { edgar } from './edgar.js'; import { eonetEvents } from './eonet.js'; import { eurostat } from './eurostat.js'; +import { faaAirportStatus } from './faanas.js'; import { fbiCrimeEstimates } from './fbicrime.js'; +import { fdicInstitutions, fdicStructureChanges } from './fdic.js'; import { federalRegister } from './federalregister.js'; import { freddieMacRates } from './freddiemac.js'; import { fueleconomyCatalog } from './fueleconomy.js'; @@ -35,6 +41,7 @@ import { nhcCyclones } from './nhc.js'; import { nhtsaComplaints, nhtsaRatings, nhtsaRecalls } from './nhtsa.js'; import { npm } from './npm.js'; import { ntldChanges, ntldLaunches, ntldTlds, ntldTotals } from './ntlddata.js'; +import { nwpsRiverGauges } from './nwps.js'; import { nws } from './nws.js'; import { ocdsTenders } from './ocds.js'; import { openfda } from './openfda.js'; @@ -98,6 +105,15 @@ export const ADAPTERS = [ nhcCyclones, swpcSpaceWeather, eonetEvents, + faaAirportStatus, + aviationHazards, + aviationMetar, + nwpsRiverGauges, + coopsWaterLevels, + droughtMonitor, + cfpbComplaints, + fdicInstitutions, + fdicStructureChanges, statuspage, firefoxAddons, vscodeExtensions, diff --git a/packages/adapters/src/nwps.js b/packages/adapters/src/nwps.js new file mode 100644 index 0000000..df9d393 --- /dev/null +++ b/packages/adapters/src/nwps.js @@ -0,0 +1,254 @@ +import { defineAdapter } from '@nichedb/core/adapter'; + +/** + * Every river gauge NOAA forecasts, and the ones that are in flood right now. + * + * The National Water Prediction Service publishes 12,000-odd gauges with an + * observed stage, a forecast stage and, for each, the flood category that stage + * falls in: none, action, minor, moderate, major. The National Weather Service + * alert feed this deployment already reads says a county is under a flood + * warning; this says which river, how high it is, and whether it is still + * rising. They are different facts and only one of them has a number in it. + * + * WHY ONLY THE GAUGES IN FLOOD + * + * All 12,000 gauges are one 13 MB answer that takes the better part of a minute + * to arrive, and on an ordinary day about thirty of them are above their action + * stage. Storing the other 11,970 hourly would be a river-height archive, which + * is a different product and one the USGS already runs. So a gauge earns a row + * by being at or above its action stage, observed or forecast, and the rest are + * counted in the run note and dropped. + * + * THE SENTINEL + * + * The API writes a missing reading as -999, not as null, and it does it in the + * `primary` field that carries the stage in feet. A gauge out of service + * therefore reads as a river 999 feet below datum unless the sentinel is caught, + * and -999 is finite, so `Number.isFinite` does not catch it. Both are checked + * here and the reading becomes null, which is what "we do not know" should look + * like. + */ + +/** + * The bounding boxes a source polls. + * + * The API takes a bounding box and nothing else -- `state`, `wfo` and `rfc` + * query parameters are accepted, ignored, and answered with all 12,000 gauges, + * which is the kind of failure that looks like success -- so a region here is a + * box, and the boxes tile the country. + */ +export const REGIONS = { + northeast: { name: 'Northeast', bbox: [-80, 38.5, -66.5, 47.5] }, + southeast: { name: 'Southeast', bbox: [-89, 24, -75, 38.5] }, + 'ohio-valley': { name: 'Ohio Valley', bbox: [-89, 36, -78, 43] }, + midwest: { name: 'Upper Midwest', bbox: [-104, 40, -84, 49.5] }, + south: { name: 'South', bbox: [-107, 25.5, -89, 40] }, + west: { name: 'West', bbox: [-125, 31, -104, 49.5] }, + alaska: { name: 'Alaska', bbox: [-170, 52, -130, 72] }, + 'puerto-rico': { name: 'Puerto Rico and the Virgin Islands', bbox: [-68, 17, -64, 19] }, +}; + +export const REGION_KEYS = Object.keys(REGIONS); + +/** The categories that make a gauge worth a row, worst last. */ +export const FLOOD_CATEGORIES = ['action', 'minor', 'moderate', 'major']; + +const SEVERITY = { + action: 'at its action stage', + minor: 'in minor flood', + moderate: 'in moderate flood', + major: 'in major flood', +}; + +/** + * A stage reading, or null. + * + * Two ways this reads a river that is not there. -999 is the API's "no + * reading", and it is finite, so `Number.isFinite` waves it through as a river + * 999 feet below datum. And `Number(null)` is 0, so an absent field waves + * through as a gauge reading exactly zero. Both are guarded, because both + * publish a number where there is no measurement. + */ +export function stage(v) { + if (v === null || v === undefined || v === '') return null; + const n = Number(v); + if (!Number.isFinite(n) || n <= -999) return null; + return n; +} + +/** The category, lowercased, or null for the several ways it says "nothing to report". */ +export function category(raw) { + const s = String(raw ?? '') + .trim() + .toLowerCase(); + return FLOOD_CATEGORIES.includes(s) ? s : null; +} + +/** A timestamp in the form a sentence wants it. */ +const stamp = (iso) => `${iso.replace('T', ' ').slice(0, 16)}Z`; + +/** A coordinate, or nothing. The same -999 sentinel turns up in the location. */ +const coord = (v) => { + if (v === null || v === undefined || v === '') return null; + const n = Number(v); + return Number.isFinite(n) && Math.abs(n) <= 180 && n > -999 ? n : null; +}; + +function place(gauge) { + return { + country: 'US', + state: gauge.state?.abbreviation ?? null, + stateName: gauge.state?.name ?? null, + lat: coord(gauge.latitude), + lon: coord(gauge.longitude), + forecastOffice: gauge.wfo?.abbreviation ?? null, + riverForecastCentre: gauge.rfc?.abbreviation ?? null, + }; +} + +/** + * One reading at one gauge: what the river is doing, or is forecast to do. + * + * Observed and forecast are separate rows on purpose. They answer different + * questions -- "is it flooding" and "will it" -- and a forecast that turns out + * wrong should stay in the record next to the observation that contradicted it, + * not be overwritten by it. + */ +export function toItem(gauge, which) { + const reading = gauge.status?.[which]; + const cat = category(reading?.floodCategory); + const level = stage(reading?.primary); + const at = reading?.validTime; + if (!cat || level === null || !at || at.startsWith('0001-')) return null; + + const lid = String(gauge.lid ?? '').toUpperCase(); + if (!lid) return null; + const when = new Date(at).toISOString(); + const unit = String(reading.primaryUnit ?? 'ft').trim(); + const name = String(gauge.name ?? lid).trim(); + const where = place(gauge); + const observed = which === 'observed'; + + return { + externalId: `nwps-${observed ? 'obs' : 'fcst'}-${lid}-${when}`, + kind: observed ? 'river-gauge' : 'river-forecast', + title: `${name}${where.state ? `, ${where.state}` : ''}: ${ + observed ? SEVERITY[cat] : `forecast ${SEVERITY[cat]}` + } at ${level} ${unit}`, + summary: [ + `${name}${where.state ? ` in ${where.stateName ?? where.state}` : ''}`, + observed + ? `was ${SEVERITY[cat]} at ${level} ${unit} on ${stamp(when)}.` + : `is forecast to be ${SEVERITY[cat]} at ${level} ${unit} at ${stamp(when)}.`, + where.forecastOffice ? `Forecast office ${where.forecastOffice}.` : null, + ] + .filter(Boolean) + .join(' '), + url: `https://water.noaa.gov/gauges/${lid}`, + publishedAt: when, + timeKnown: true, + precision: 'minute', + tags: [ + 'water', + 'flood', + 'us', + observed ? 'observed' : 'forecast', + `flood:${cat}`, + where.state ? where.state.toLowerCase() : null, + where.riverForecastCentre ? where.riverForecastCentre.toLowerCase() : null, + cat === 'major' || cat === 'moderate' ? 'significant-flooding' : null, + ].filter(Boolean), + data: { + gaugeId: lid, + gaugeName: name, + reading: observed ? 'observed' : 'forecast', + floodCategory: cat, + stage: level, + stageUnit: unit, + secondary: stage(reading.secondary), + secondaryUnit: String(reading.secondaryUnit ?? '').trim() || null, + validTime: when, + place: where, + categoryBasis: 'NWS flood categories for this gauge', + source: 'NOAA National Water Prediction Service', + dataset: 'https://api.water.noaa.gov/nwps/v1/gauges', + }, + }; +} + +export const nwpsRiverGauges = defineAdapter({ + name: 'nwps-river-gauges', + title: 'River gauges in flood', + collection: 'water', + description: + 'Every NOAA-forecast river gauge at or above its action stage, observed and forecast, with the height in feet and the flood category the National Weather Service assigns it. The number behind a flood warning. Keyless.', + docs: 'https://api.water.noaa.gov/nwps/v1/docs/', + kinds: ['river-gauge', 'river-forecast'], + cadenceMinutes: 60, + configFields: [ + { + key: 'region', + label: 'Region', + type: 'select', + options: ['', ...REGION_KEYS], + help: 'One of the built-in regions, or give a bounding box below.', + }, + { key: 'bbox', label: 'Bounding box', help: 'minLon,minLat,maxLon,maxLat' }, + { + key: 'minimumCategory', + label: 'Minimum flood category', + type: 'select', + options: ['', ...FLOOD_CATEGORIES], + help: 'Empty means action stage and above.', + }, + ], + defaults: {}, + defaultSources: REGION_KEYS.map((key) => ({ + slug: `river-gauges-${key}`, + name: `River gauges in flood: ${REGIONS[key].name}`, + config: { region: key }, + })), + async pull({ config, cursor, http, log }) { + const bbox = boxFor(config); + if (!bbox) throw new Error('nwps-river-gauges needs a region or a bounding box'); + const [xmin, ymin, xmax, ymax] = bbox; + const url = + `https://api.water.noaa.gov/nwps/v1/gauges?srid=EPSG_4326` + + `&bbox.xmin=${xmin}&bbox.ymin=${ymin}&bbox.xmax=${xmax}&bbox.ymax=${ymax}`; + + const body = await http.json(url, { timeoutMs: 120_000 }); + const gauges = Array.isArray(body?.gauges) ? body.gauges : null; + if (!gauges) throw new Error('the NWPS API did not return a gauge list'); + + const floor = category(config.minimumCategory); + const atLeast = floor ? FLOOD_CATEGORIES.indexOf(floor) : 0; + const items = gauges + .flatMap((g) => [toItem(g, 'observed'), toItem(g, 'forecast')]) + .filter(Boolean) + .filter((i) => FLOOD_CATEGORIES.indexOf(i.data.floodCategory) >= atLeast); + + const newest = + items + .map((i) => i.publishedAt) + .filter(Boolean) + .sort() + .at(-1) ?? cursor.since; + log(`${items.length} gauge reading(s) at or above action stage, of ${gauges.length} gauges`); + return { + items, + cursor: { since: newest ?? null }, + note: `${items.length} of ${gauges.length} gauges`, + }; + }, +}); + +/** A source's own box, or its region's. */ +export function boxFor(config) { + const raw = String(config.bbox ?? '').trim(); + if (raw) { + const parts = raw.split(',').map((n) => Number(n.trim())); + if (parts.length === 4 && parts.every(Number.isFinite)) return parts; + return null; + } + return REGIONS[String(config.region ?? '').toLowerCase()]?.bbox ?? null; +} diff --git a/packages/core/src/seed.js b/packages/core/src/seed.js index e9a3bd6..d226c4e 100644 --- a/packages/core/src/seed.js +++ b/packages/core/src/seed.js @@ -153,6 +153,24 @@ export const COLLECTIONS = [ description: 'Podcast shows split the one way no podcast app will split them: by who serves the feed. On one side the shows on a commercial host, a network or a broadcaster — 94% of the medium. On the other the shows published from the maker’s own domain, which is where the independent 6% is, and which nothing else lists separately because nobody sells it.', }, + { + slug: 'aviation', + name: 'Aviation', + description: + 'Why flights are late, from the three feeds that between them answer it: the traffic management initiatives the FAA has in force right now — ground stops, ground delay programs, airport closures — each written once more when it ends with how long it ran, which the FAA itself never publishes; every SIGMET and AIRMET in the air over the country; and the decoded observation at the airport underneath. One airport, one hour, three sources. All keyless.', + }, + { + slug: 'water', + name: 'Water', + description: + 'Too much water and too little, measured rather than forecast: every NOAA river gauge at or above its action stage with the height in feet and the flood category behind the warning, the observed level at tide stations on every US coast against the height at which each one floods, and the US Drought Monitor’s weekly read on how much of each state is dry and how badly. The fast half and the slow half of the same system.', + }, + { + slug: 'consumer-finance', + name: 'Consumer finance', + description: + 'What Americans say their banks, lenders, credit bureaus and debt collectors are doing to them, and who those companies actually are. Around ten thousand complaints a day from the Consumer Financial Protection Bureau, a third of them carrying the consumer’s own account; every FDIC-insured institution with its charter, regulator and assets; and every merger, failure, conversion and branch opening or closing on the FDIC register — which is the only public record of which bank a complaint about a vanished bank now belongs to.', + }, ]; export const DEFAULT_FEEDS = [ @@ -842,6 +860,119 @@ export const DEFAULT_FEEDS = [ name: 'Self-hosted podcasts in French', query: { sources: ['podcasts-self-hosted'], tags: ['lang:fr'] }, }, + /* + * Aviation reads as one story per airport, so the feeds are cuts of the + * cause rather than of the source: the delays, the weather that explains + * them, and the airports where the weather is actually in the way. + */ + { + collection: 'aviation', + slug: 'ground-stops-and-delays', + name: 'Ground stops and delay programs', + query: { kinds: ['ground-stop', 'ground-delay', 'airspace-flow', 'trajectory-options'] }, + }, + { + collection: 'aviation', + slug: 'weather-delays', + name: 'Delays caused by weather', + query: { sources: ['faa-nas-status'], tags: ['weather'] }, + }, + { + collection: 'aviation', + slug: 'airport-closures', + name: 'Airport closures', + query: { kinds: ['airport-closure'] }, + }, + { + collection: 'aviation', + slug: 'aviation-hazards', + name: 'SIGMETs and AIRMETs in force', + query: { kinds: ['aviation-hazard'] }, + }, + { + collection: 'aviation', + slug: 'airports-below-vfr', + name: 'Airports below VFR', + query: { kinds: ['observation'], tags: ['below-vfr'] }, + }, + { + collection: 'water', + slug: 'rivers-in-flood', + name: 'Rivers in flood', + query: { kinds: ['river-gauge'], tags: ['flood:minor', 'flood:moderate', 'flood:major'] }, + }, + { + collection: 'water', + slug: 'major-flooding', + name: 'Major and moderate flooding', + query: { tags: ['significant-flooding'] }, + }, + { + collection: 'water', + slug: 'river-forecasts', + name: 'Rivers forecast to flood', + query: { kinds: ['river-forecast'] }, + }, + { + collection: 'water', + slug: 'coastal-flooding', + name: 'Coastal flooding', + query: { kinds: ['water-level'], tags: ['flooding'] }, + }, + { + collection: 'water', + slug: 'drought', + name: 'Drought by state', + query: { kinds: ['drought'] }, + }, + { + collection: 'water', + slug: 'extreme-drought', + name: 'Extreme and exceptional drought', + query: { tags: ['extreme-drought'] }, + }, + { + collection: 'consumer-finance', + slug: 'consumer-complaints', + name: 'Consumer complaints', + query: { kinds: ['complaint'] }, + }, + { + collection: 'consumer-finance', + slug: 'complaints-in-their-own-words', + name: 'Complaints in the consumer’s own words', + query: { kinds: ['complaint'], tags: ['has-narrative'] }, + }, + { + collection: 'consumer-finance', + slug: 'mortgage-complaints', + name: 'Mortgage complaints', + query: { kinds: ['complaint'], tags: ['mortgage'] }, + }, + { + collection: 'consumer-finance', + slug: 'debt-collection-complaints', + name: 'Debt collection complaints', + query: { kinds: ['complaint'], tags: ['debt-collection'] }, + }, + { + collection: 'consumer-finance', + slug: 'bank-mergers-and-failures', + name: 'Bank mergers and failures', + query: { kinds: ['structure-change'], tags: ['merger', 'establishment'] }, + }, + { + collection: 'consumer-finance', + slug: 'branch-closings', + name: 'Branches opening and closing', + query: { kinds: ['structure-change'], tags: ['branch-closing', 'branch-opening'] }, + }, + { + collection: 'consumer-finance', + slug: 'insured-banks', + name: 'FDIC-insured banks', + query: { kinds: ['institution'] }, + }, ]; /** diff --git a/test/adapters.test.js b/test/adapters.test.js index 637cd4d..8926f95 100644 --- a/test/adapters.test.js +++ b/test/adapters.test.js @@ -14,6 +14,12 @@ import { parseSteamDate, toItem as steamItem } from '../packages/adapters/src/st import { toItem as usgsItem } from '../packages/adapters/src/usgs.js'; import { looseDate, normaliseItem, xmlItems } from '../packages/core/src/adapter.js'; +// The seed module reaches the database package, which reads the environment at +// import. It needs the variable to exist, not to connect: nothing here queries. +process.env.DATABASE_URL ??= 'postgres://test:test@localhost:5432/test'; +process.env.SITE_URL ??= 'https://nichedb.test'; +const { DEFAULT_FEEDS } = await import('../packages/core/src/seed.js'); + const fixture = (name) => readFile(new URL(`../packages/adapters/test/fixtures/${name}`, import.meta.url), 'utf8'); @@ -51,6 +57,9 @@ describe('registry', () => { 'news', 'domains', 'podcasts', + 'aviation', + 'water', + 'consumer-finance', ]).toContain(a.collection); } expect(adapterByName('steam').title).toContain('Steam'); @@ -66,6 +75,21 @@ describe('registry', () => { } } }); + test('no two seeded sources or feeds want the same slug', () => { + /* `sources.slug` and `feeds.slug` are each unique across the whole + * database, not per collection, so two adapters that happen to pick the + * same name do not both get a row: the second insert is swallowed as an + * existing one and that source never runs. Nothing in the seed log says + * so, which is why this is asserted here rather than discovered later. */ + const clashes = (slugs) => + [...slugs.reduce((m, s) => m.set(s, (m.get(s) ?? 0) + 1), new Map())] + .filter(([, n]) => n > 1) + .map(([s]) => s); + expect(clashes(ADAPTERS.flatMap((a) => (a.defaultSources ?? []).map((s) => s.slug)))).toEqual( + [], + ); + expect(clashes(DEFAULT_FEEDS.map((f) => f.slug))).toEqual([]); + }); }); describe('core helpers', () => { diff --git a/test/aviation.test.js b/test/aviation.test.js new file mode 100644 index 0000000..0a1c2f1 --- /dev/null +++ b/test/aviation.test.js @@ -0,0 +1,250 @@ +import { describe, expect, test } from 'bun:test'; +import { + bboxOf, + hazardItem, + metarItem, + movement, + parseBox, + quarters, + regionsOf, +} from '../packages/adapters/src/aviationweather.js'; +import { + duration, + toItem as faaItem, + legsOf, + parseStatus, + reasonTags, +} from '../packages/adapters/src/faanas.js'; +import { adapterByName } from '../packages/adapters/src/index.js'; +import { normaliseItem, xmlItems } from '../packages/core/src/adapter.js'; + +process.env.DATABASE_URL ??= 'postgres://test:test@localhost:5432/test'; +process.env.SITE_URL ??= 'https://nichedb.test'; +const { COLLECTIONS, DEFAULT_FEEDS } = await import('../packages/core/src/seed.js'); + +/* + * The FAA snapshot, in the shape nasstatus.faa.gov actually returns it, taken + * on 9 September 2026. Two things in here are the reason it is a fixture rather + * than a hand-simplified example: `Airport Closures` appears TWICE as a + * Delay_type, and a `Delay` carries its arrival and departure legs as nested + * children distinguished only by an attribute. + */ +const SNAPSHOT = `Wed Sep 9 17:23:43 2026 GMThttp://www.fly.faa.gov/AirportStatus.dtdGround Delay ProgramsBOSrunway construction58 minutes2 hours and 31 minutesMIAthunderstorms35 minutes1 hour and 59 minutesGround Stop ProgramsEWRweather / thunderstorms7 PM EDTArrival/Departure DelaysSFOlow ceilings46 minutes1 hourIncreasing15 minutes30 minutesDecreasingAirport ClosuresSNA!SNA 09/016 SNA AD AP CLSD DLY 0630-1315Sep 09 at 06:30 UTC.Sep 12 at 13:15 UTC.Airport ClosuresLAX!LAX 05/277 LAX AD AP CLSD TO NON SKED TRANSIENT GA ACFTMay 27 at 18:26 UTC.May 28 at 16:00 UTC.`; + +describe('the aviation collection', () => { + test('exists, and every aviation adapter is registered in it', () => { + expect(COLLECTIONS.map((c) => c.slug)).toContain('aviation'); + for (const name of ['faa-nas-status', 'aviation-hazards', 'aviation-metar']) { + const a = adapterByName(name); + expect(a).not.toBeNull(); + expect(a.collection).toBe('aviation'); + } + }); + + test('every aviation feed queries kinds the collection actually emits', () => { + const kinds = new Set( + ['faa-nas-status', 'aviation-hazards', 'aviation-metar'].flatMap( + (n) => adapterByName(n).kinds, + ), + ); + const feeds = DEFAULT_FEEDS.filter((f) => f.collection === 'aviation'); + expect(feeds.length).toBeGreaterThan(0); + for (const feed of feeds) { + for (const kind of feed.query.kinds ?? []) expect(kinds.has(kind)).toBe(true); + } + }); +}); + +describe('the FAA snapshot', () => { + test('reads every program type out of one snapshot', () => { + const { updated, programs } = parseStatus(SNAPSHOT); + expect(updated).toBe('Wed Sep 9 17:23:43 2026 GMT'); + const kinds = programs.map((p) => `${p.kind}:${p.where}`); + expect(kinds).toContain('ground-delay:BOS'); + expect(kinds).toContain('ground-delay:MIA'); + expect(kinds).toContain('ground-stop:EWR'); + expect(kinds).toContain('airport-delay:SFO'); + expect(kinds).toContain('airport-closure:SNA'); + }); + + test('keeps both Airport Closures blocks, because the FAA sends two', () => { + /* The snapshot really does repeat the Delay_type name. A reader that + * indexed blocks by their name would keep whichever came last and lose the + * other airport entirely, with nothing to show that it had. */ + const closures = parseStatus(SNAPSHOT) + .programs.filter((p) => p.kind === 'airport-closure') + .map((p) => p.where); + expect(closures).toEqual(expect.arrayContaining(['SNA', 'LAX'])); + }); + + test('a delay carries both directions, with the minutes belonging to each', () => { + const delay = xmlItems(SNAPSHOT, 'Delay').find((f) => f.ARPT?.text === 'SFO'); + const legs = legsOf(delay); + expect(legs).toHaveLength(2); + expect(legs.find((l) => l.type === 'Arrival')).toMatchObject({ + min: '46 minutes', + max: '1 hour', + trend: 'Increasing', + }); + expect(legs.find((l) => l.type === 'Departure').min).toBe('15 minutes'); + }); + + test('does not mistake Ground_Delay for Delay, or Airport_Closure_List for Airport', () => { + const { programs } = parseStatus(SNAPSHOT); + expect(programs.filter((p) => p.kind === 'airport-delay')).toHaveLength(1); + expect(programs.filter((p) => p.kind === 'airport-closure')).toHaveLength(2); + }); + + test('a program is one row that keeps its identity while it lasts', () => { + const [program] = parseStatus(SNAPSHOT).programs; + const first = normaliseItem( + faaItem(program, { firstSeen: '2026-09-09T17:00:00.000Z', now: '2026-09-09T17:00:00.000Z' }), + ); + const later = normaliseItem( + faaItem(program, { firstSeen: '2026-09-09T17:00:00.000Z', now: '2026-09-09T19:00:00.000Z' }), + ); + expect(later.externalId).toBe(first.externalId); + expect(later.contentHash).not.toBe(first.contentHash); + expect(later.tags).toContain('in-force'); + }); + + test('a program that leaves the snapshot is written once more, with how long it ran', () => { + const [program] = parseStatus(SNAPSHOT).programs; + const ended = normaliseItem( + faaItem(program, { + firstSeen: '2026-09-09T17:00:00.000Z', + now: '2026-09-09T20:40:00.000Z', + ended: true, + }), + ); + expect(ended.title).toContain('ended after 3h 40m'); + expect(ended.data.endedAt).toBe('2026-09-09T20:40:00.000Z'); + expect(ended.data.status).toBe('ended'); + expect(ended.tags).toContain('ended'); + }); + + test('durations read the way a person would say them', () => { + expect(duration('2026-09-09T17:00:00Z', '2026-09-09T17:25:00Z')).toBe('25m'); + expect(duration('2026-09-09T17:00:00Z', '2026-09-09T19:00:00Z')).toBe('2h'); + expect(duration('2026-09-09T17:00:00Z', '2026-09-09T19:05:00Z')).toBe('2h 5m'); + expect(duration('2026-09-09T19:00:00Z', '2026-09-09T17:00:00Z')).toBeNull(); + }); + + test('the cause is tagged from the FAA’s own wording', () => { + expect(reasonTags('weather / thunderstorms')).toEqual( + expect.arrayContaining(['weather', 'thunderstorms']), + ); + expect(reasonTags('runway construction')).toContain('runway'); + expect(reasonTags('disabled aircraft on the runway')).toContain('incident'); + expect(reasonTags(null)).toEqual([]); + }); +}); + +describe('aviation weather', () => { + const sigmet = { + icaoId: 'KKCI', + airSigmetType: 'SIGMET', + seriesId: '93C', + hazard: 'CONVECTIVE', + severity: 5, + validTimeFrom: 1788972900, + validTimeTo: 1788980100, + altitudeHi1: 29000, + movementDir: 240, + movementSpd: 35, + coords: [ + { lat: 46.3, lon: -83.2 }, + { lat: 44.6, lon: -83.0 }, + { lat: 44.9, lon: -85.4 }, + ], + rawAirSigmet: + 'WSUS32 KKCI 091655\nSIGC\nCONVECTIVE SIGMET 93C\nVALID UNTIL 1855Z\nMI LH\nFROM 70SE SSM-60NNE ASP\nDMSHG AREA EMBD TS MOV FROM 24035KT.', + }; + + test('the region comes off the bulletin, not from the office that wrote it', () => { + expect(regionsOf(sigmet.rawAirSigmet)).toBe('MI LH'); + const item = normaliseItem(hazardItem(sigmet)); + expect(item.title).toContain('over MI LH'); + expect(item.data.issuingOffice).toBe('KKCI'); + expect(item.tags).toContain('mi'); + // KKCI is the Aviation Weather Center. It is not a place weather is over. + expect(item.tags).not.toContain('kkci'); + }); + + test('a bulletin with no region line says so rather than inventing one', () => { + expect(regionsOf('SIGMET\nFROM 70SE SSM-60NNE ASP')).toBeNull(); + expect(regionsOf(null)).toBeNull(); + const item = hazardItem({ ...sigmet, rawAirSigmet: 'SIGMET\nno from line here' }); + expect(item.data.area).toBeNull(); + expect(item.title).not.toContain('over'); + }); + + test('a reissue under the same series is its own row', () => { + const a = hazardItem(sigmet); + const b = hazardItem({ ...sigmet, validTimeFrom: sigmet.validTimeFrom + 7200 }); + expect(a.externalId).not.toBe(b.externalId); + }); + + test('the polygon becomes a box, and movement becomes words', () => { + expect(bboxOf(sigmet.coords)).toEqual({ + minLat: 44.6, + minLon: -85.4, + maxLat: 46.3, + maxLon: -83.0, + }); + expect(bboxOf(null)).toBeNull(); + expect(movement(sigmet)).toBe('WSW at 35 kt'); + expect(movement({ movementDir: 0, movementSpd: 0 })).toBeNull(); + }); + + test('an observation keeps the zeroes that mean something', () => { + const item = normaliseItem( + metarItem({ + icaoId: 'KORD', + name: 'Chicago/O’Hare Intl, IL, US', + reportTime: '2026-09-09T17:00:00.000Z', + fltCat: 'IFR', + temp: 0, + dewp: 0, + wdir: 0, + wspd: 5, + visib: '1/2', + lat: 41.9, + lon: -87.9, + rawOb: 'METAR KORD 091651Z 36005KT 1/2SM', + }), + ); + // Zero degrees is a temperature and north is a direction; `|| null` on + // either would have erased both. + expect(item.data.temperatureC).toBe(0); + expect(item.data.windDirectionDeg).toBe(0); + expect(item.summary).toContain('wind 0° at 5 kt'); + expect(item.tags).toContain('below-vfr'); + expect(item.tags).toContain('ifr'); + }); + + test('an observation with no station or no time is not an observation', () => { + expect(metarItem({ icaoId: 'KORD' })).toBeNull(); + expect(metarItem({ reportTime: '2026-09-09T17:00:00.000Z' })).toBeNull(); + }); + + test('a box is split until its answer is not at the cap', () => { + expect(parseBox('24,-125,50,-66')).toEqual([24, -125, 50, -66]); + expect(parseBox('24,-125,50')).toBeNull(); + expect(parseBox('north,west,south,east')).toBeNull(); + + /* The continental United States answers a single box with exactly 400 + * stations while its two halves answer with 247 and 240. The quartering is + * what stops that silent 87-station loss, so the quarters must tile the + * box exactly: no gap between them is an airport nobody reads. */ + const box = [24, -125, 50, -66]; + const qs = quarters(box); + expect(qs).toHaveLength(4); + expect(Math.min(...qs.map((q) => q[0]))).toBe(24); + expect(Math.max(...qs.map((q) => q[2]))).toBe(50); + expect(Math.min(...qs.map((q) => q[1]))).toBe(-125); + expect(Math.max(...qs.map((q) => q[3]))).toBe(-66); + const area = (b) => (b[2] - b[0]) * (b[3] - b[1]); + expect(qs.reduce((sum, q) => sum + area(q), 0)).toBeCloseTo(area(box), 6); + }); +}); diff --git a/test/consumer-finance.test.js b/test/consumer-finance.test.js new file mode 100644 index 0000000..3123f8a --- /dev/null +++ b/test/consumer-finance.test.js @@ -0,0 +1,268 @@ +import { describe, expect, test } from 'bun:test'; +import { + toItem as complaintItem, + daysBetween, + nextDay, + productFamily, + utcDay, +} from '../packages/adapters/src/cfpb.js'; +import { + changeItem, + familyOf, + fdicDate, + institutionItem, + money, + thousands, +} from '../packages/adapters/src/fdic.js'; +import { adapterByName } from '../packages/adapters/src/index.js'; +import { normaliseItem, slugify } from '../packages/core/src/adapter.js'; + +process.env.DATABASE_URL ??= 'postgres://test:test@localhost:5432/test'; +process.env.SITE_URL ??= 'https://nichedb.test'; +const { COLLECTIONS, DEFAULT_FEEDS } = await import('../packages/core/src/seed.js'); + +/* A complaint as the CFPB search returns one. */ +const hit = (over = {}) => ({ + _id: '26830447', + _source: { + complaint_id: '26830447', + company: 'TRANSUNION INTERMEDIATE HOLDINGS, INC.', + product: 'Credit reporting or other personal consumer reports', + sub_product: 'Credit reporting', + issue: "Problem with a company's investigation into an existing problem", + sub_issue: 'Investigation took more than 30 days', + date_received: '2026-09-09T03:59:29.000Z', + date_sent_to_company: '2026-09-09T03:59:50.000Z', + state: 'IN', + zip_code: '46307', + company_response: 'In progress', + timely: 'Yes', + submitted_via: 'Web', + has_narrative: false, + complaint_what_happened: '', + ...over, + }, +}); + +describe('the consumer finance collection', () => { + test('exists, and every adapter in it is registered there', () => { + expect(COLLECTIONS.map((c) => c.slug)).toContain('consumer-finance'); + for (const name of ['cfpb-complaints', 'fdic-institutions', 'fdic-structure-changes']) { + const a = adapterByName(name); + expect(a).not.toBeNull(); + expect(a.collection).toBe('consumer-finance'); + } + }); + + test('a complaint and an institution can be joined on the same company key', () => { + /* This is the whole reason the FDIC feeds are in this collection. The CFPB + * names a company and identifies it with nothing at all, so the only join + * available is the name, and both sides have to normalise it the same way + * or the join silently matches nothing. */ + const complaint = complaintItem(hit({ company: 'Navy Federal Credit Union' })); + const bank = institutionItem({ data: { CERT: '5536', NAME: 'Navy Federal Credit Union' } }); + expect(complaint.data.companyKey).toBe(bank.data.nameKey); + expect(complaint.data.companyKey).toBe(slugify('Navy Federal Credit Union')); + }); + + test('every consumer-finance feed queries kinds these adapters emit', () => { + const kinds = new Set( + ['cfpb-complaints', 'fdic-institutions', 'fdic-structure-changes'].flatMap( + (n) => adapterByName(n).kinds, + ), + ); + const feeds = DEFAULT_FEEDS.filter((f) => f.collection === 'consumer-finance'); + expect(feeds.length).toBeGreaterThan(0); + for (const feed of feeds) { + for (const kind of feed.query.kinds ?? []) expect(kinds.has(kind)).toBe(true); + } + }); +}); + +describe('CFPB complaints', () => { + test('a complaint becomes a row that names the company, the product and the issue', () => { + const item = normaliseItem(complaintItem(hit())); + expect(item.externalId).toBe('cfpb-26830447'); + expect(item.kind).toBe('complaint'); + expect(item.publishedAt.toISOString()).toBe('2026-09-09T03:59:29.000Z'); + expect(item.data.productFamily).toBe('credit-reporting'); + expect(item.tags).toContain('in-progress'); + expect(item.tags).toContain('in'); + }); + + test('an unpublished narrative is absent, not an empty string', () => { + // has_narrative is false and complaint_what_happened is '' on most rows. + // Storing that '' would claim the consumer said nothing. + const quiet = complaintItem(hit()); + expect(quiet.data.narrative).toBeNull(); + expect(quiet.data.hasNarrative).toBe(false); + expect(quiet.tags).not.toContain('has-narrative'); + + const spoken = complaintItem( + hit({ has_narrative: true, complaint_what_happened: 'They never answered.' }), + ); + expect(spoken.data.narrative).toBe('They never answered.'); + expect(spoken.tags).toContain('has-narrative'); + expect(spoken.summary).toContain('They never answered.'); + }); + + test('the product family is short enough to be a tag and stable enough to be a feed', () => { + expect(productFamily('Credit reporting or other personal consumer reports')).toBe( + 'credit-reporting', + ); + expect(productFamily('Debt collection')).toBe('debt-collection'); + expect(productFamily('Checking or savings account')).toBe('bank-account'); + expect(productFamily('Vehicle loan or lease')).toBe('auto-loan'); + expect(productFamily('Payday loan, title loan, personal loan, or advance loan')).toBe( + 'payday-loan', + ); + expect(productFamily('Something new the Bureau invented')).toBe('other'); + expect(productFamily(null)).toBeNull(); + }); + + test('a row with no company, id or date is not a complaint', () => { + expect(complaintItem(hit({ company: null }))).toBeNull(); + expect(complaintItem(hit({ complaint_id: null }))).toBeNull(); + expect(complaintItem(hit({ date_received: null }))).toBeNull(); + }); + + test('the day walk covers every day and stops where it is told', () => { + /* The search accepts an offset, echoes it, and ignores it: frm=0, frm=500 + * and frm=1000 return identical pages. So the adapter walks days instead of + * offsets, and a gap in the walk is a day of complaints nobody would ever + * see again. */ + expect(nextDay('2026-09-09')).toBe('2026-09-10'); + expect(nextDay('2026-09-30')).toBe('2026-10-01'); + expect(nextDay('2026-12-31')).toBe('2027-01-01'); + expect(nextDay('2028-02-28')).toBe('2028-02-29'); + expect(daysBetween('2026-09-07', '2026-09-09', 10)).toEqual([ + '2026-09-07', + '2026-09-08', + '2026-09-09', + ]); + expect(daysBetween('2026-09-07', '2026-09-30', 2)).toEqual(['2026-09-07', '2026-09-08']); + expect(daysBetween('2026-09-09', '2026-09-09', 5)).toEqual(['2026-09-09']); + expect(daysBetween('2026-09-10', '2026-09-09', 5)).toEqual([]); + expect(utcDay('2026-09-09T23:59:59.000Z')).toBe('2026-09-09'); + }); + + test('the narratives source sweeps rather than tails, because narratives arrive late', () => { + // Complaints received in the last month carry one narrative between them; + // the same query from four months back returns 42,516. A source following + // the newest rows would be permanently empty. + const sweeper = adapterByName('cfpb-complaints').defaultSources.find( + (s) => s.config?.narrativesOnly === 'yes', + ); + expect(sweeper).toBeDefined(); + expect(sweeper.config.sweepDays).toBeGreaterThan(90); + }); +}); + +describe('FDIC', () => { + test('the register reads its dates whichever way they are written', () => { + expect(fdicDate('01/20/1910')).toBe('1910-01-20'); + expect(fdicDate('7/17/2026')).toBe('2026-07-17'); + expect(fdicDate('2026-09-03T00:00:00')).toBe('2026-09-03'); + expect(fdicDate('20260717')).toBe('2026-07-17'); + expect(fdicDate('0')).toBeNull(); + expect(fdicDate(null)).toBeNull(); + }); + + test('assets are reported in thousands, and a bank with none is not a bank with zero', () => { + expect(thousands(276713)).toBe(276_713_000); + expect(thousands(null)).toBeNull(); + expect(thousands('')).toBeNull(); + expect(thousands(0)).toBe(0); + expect(money(412_620_000_000)).toBe('$412.62bn'); + }); + + test('an institution carries the charter class the API actually publishes', () => { + /* BankFind's institution endpoint has no CLASS field; the charter class is + * BKCLASS. Asking for CLASS files every bank in the country as + * unclassified and raises no error doing it. */ + const item = normaliseItem( + institutionItem({ + data: { + CERT: '14', + NAME: 'State Street Bank and Trust Company', + BKCLASS: 'SM', + REGAGNT: 'FED', + ASSET: 412_620_000, + CITY: 'Boston', + STALP: 'MA', + ACTIVE: 1, + ESTYMD: '01/01/1792', + RUNDATE: '09/04/2026', + SPECGRPN: 'All Other Over 1 Billion', + }, + }), + ); + expect(item.data.class).toBe('SM'); + expect(item.tags).toContain('class:sm'); + expect(item.tags).toContain('assets:100bn-plus'); + expect(item.data.establishedOn).toBe('1792-01-01'); + }); + + test('a structure change is keyed on the row, not on the transaction', () => { + /* One merger writes a row for every institution and office it touches: + * 1,702 changes read in a single run carried only 1,169 distinct + * transaction numbers. Keyed on TRANSNUM, a third of the register would + * overwrite the rest, and the loss would look like a quiet quarter. */ + const base = { + TRANSNUM: 2026024801, + INSTNAME: 'The Fidelity Bank', + PROCDATE: '2026-09-03T00:00:00', + EFFDATE: '2026-08-11T00:00:00', + CHANGECODE: '713', + CHANGECODE_DESC: 'Branch Acquired in Merger/Consolidation/Failure', + OFF_PCITY: 'Greensboro', + OFF_PSTALP: 'NC', + }; + const a = changeItem({ data: { ...base, ID: 'row-one', OFF_NAME: 'WESTPORT BRANCH' } }); + const b = changeItem({ data: { ...base, ID: 'row-two', OFF_NAME: 'PISGAH BRANCH' } }); + expect(a.externalId).not.toBe(b.externalId); + + // And with no ID the office and institution numbers have to rebuild one. + const c = changeItem({ data: { ...base, UNINUM: '1', OFF_NUM: '7' } }); + const d = changeItem({ data: { ...base, UNINUM: '1', OFF_NUM: '9' } }); + expect(c.externalId).not.toBe(d.externalId); + }); + + test('the event is the change code, which is the field the FDIC actually fills in', () => { + /* The register also carries twenty-odd boolean flags, and they are zero on + * the great majority of transactions, mergers included. A reader built on + * them reports almost nothing happening. */ + expect(familyOf('713')).toBe('branch'); + expect(familyOf('223')).toBe('merger'); + expect(familyOf('520')).toBe('name-or-location'); + expect(familyOf(null)).toBe('other'); + + const item = normaliseItem( + changeItem({ + data: { + ID: 'x', + TRANSNUM: 1, + INSTNAME: 'Santander Bank, N.A.', + PROCDATE: '2026-08-28T00:00:00', + CHANGECODE: '721', + CHANGECODE_DESC: 'Branch Closing', + OFF_NAME: 'WESTPORT BRANCH', + OFF_PCITY: 'Westport', + OFF_PSTALP: 'CT', + }, + }), + ); + expect(item.data.change).toBe('Branch Closing'); + expect(item.tags).toContain('branch-closing'); + expect(item.tags).toContain('ct'); + expect(item.title).toContain('WESTPORT BRANCH'); + }); + + test('a change with no name, transaction or date is not a change', () => { + expect(changeItem({ data: { TRANSNUM: 1, PROCDATE: '2026-09-03T00:00:00' } })).toBeNull(); + expect( + changeItem({ data: { INSTNAME: 'A Bank', PROCDATE: '2026-09-03T00:00:00' } }), + ).toBeNull(); + expect(changeItem({ data: { TRANSNUM: 1, INSTNAME: 'A Bank' } })).toBeNull(); + }); +}); diff --git a/test/water.test.js b/test/water.test.js new file mode 100644 index 0000000..a6201fa --- /dev/null +++ b/test/water.test.js @@ -0,0 +1,276 @@ +import { describe, expect, test } from 'bun:test'; +import { toItem as coopsItem, exceedance } from '../packages/adapters/src/coops.js'; +import { + byClass, + CLASSES, + toItem as droughtItem, + STATE_FIPS, + worstClass, +} from '../packages/adapters/src/droughtmonitor.js'; +import { adapterByName } from '../packages/adapters/src/index.js'; +import { + boxFor, + category, + FLOOD_CATEGORIES, + toItem as gaugeItem, + REGIONS, + stage, +} from '../packages/adapters/src/nwps.js'; +import { normaliseItem } from '../packages/core/src/adapter.js'; + +process.env.DATABASE_URL ??= 'postgres://test:test@localhost:5432/test'; +process.env.SITE_URL ??= 'https://nichedb.test'; +const { COLLECTIONS, DEFAULT_FEEDS } = await import('../packages/core/src/seed.js'); + +/* A gauge as the NWPS API returns one, including the sentinel it uses for a + * reading it does not have. */ +const gauge = (over = {}) => ({ + lid: 'PISt2', + name: 'Pine Island Bayou near Sour Lake', + wfo: { abbreviation: 'LCH', name: 'Lake Charles' }, + rfc: { abbreviation: 'WGRFC', name: 'West Gulf River Forecast Center' }, + state: { abbreviation: 'TX', name: 'Texas' }, + latitude: 30.13, + longitude: -94.4, + status: { + observed: { + primary: 25.79, + primaryUnit: 'ft', + secondary: -999, + secondaryUnit: 'kcfs', + floodCategory: 'minor', + validTime: '2026-09-09T16:45:00Z', + }, + forecast: { + primary: -999, + primaryUnit: '', + floodCategory: 'fcst_not_current', + validTime: '0001-01-01T00:00:00Z', + }, + }, + ...over, +}); + +describe('the water collection', () => { + test('exists, and every water adapter is registered in it', () => { + expect(COLLECTIONS.map((c) => c.slug)).toContain('water'); + for (const name of ['nwps-river-gauges', 'coops-water-levels', 'drought-monitor']) { + const a = adapterByName(name); + expect(a).not.toBeNull(); + expect(a.collection).toBe('water'); + } + }); + + test('every water feed queries kinds these adapters emit', () => { + const kinds = new Set( + ['nwps-river-gauges', 'coops-water-levels', 'drought-monitor'].flatMap( + (n) => adapterByName(n).kinds, + ), + ); + const feeds = DEFAULT_FEEDS.filter((f) => f.collection === 'water'); + expect(feeds.length).toBeGreaterThan(0); + for (const feed of feeds) { + for (const kind of feed.query.kinds ?? []) expect(kinds.has(kind)).toBe(true); + } + }); + + test('the regions tile the country, and each has a usable box', () => { + for (const [key, region] of Object.entries(REGIONS)) { + const [xmin, ymin, xmax, ymax] = region.bbox; + expect(xmin).toBeLessThan(xmax); + expect(ymin).toBeLessThan(ymax); + expect(boxFor({ region: key })).toEqual(region.bbox); + } + expect(boxFor({ bbox: '-100,30,-95,35' })).toEqual([-100, 30, -95, 35]); + expect(boxFor({ bbox: 'not,a,box' })).toBeNull(); + expect(boxFor({ region: 'atlantis' })).toBeNull(); + }); +}); + +describe('river gauges', () => { + test('the -999 sentinel is a missing reading, not a river below datum', () => { + // It is finite, so Number.isFinite lets it through; only naming it stops + // "out of service" being published as a stage of minus 999 feet. + expect(stage(-999)).toBeNull(); + expect(stage('-999.0')).toBeNull(); + expect(stage(0)).toBe(0); + expect(stage(25.79)).toBe(25.79); + expect(stage(null)).toBeNull(); + }); + + test('only a real flood category counts', () => { + for (const c of FLOOD_CATEGORIES) expect(category(c)).toBe(c); + for (const c of ['no_flooding', 'not_defined', 'obs_not_current', 'out_of_service', '']) { + expect(category(c)).toBeNull(); + } + }); + + test('an observed gauge in flood becomes a row with the number in it', () => { + const item = normaliseItem(gaugeItem(gauge(), 'observed')); + expect(item.kind).toBe('river-gauge'); + expect(item.data.stage).toBe(25.79); + expect(item.data.floodCategory).toBe('minor'); + expect(item.data.secondary).toBeNull(); + expect(item.data.place.state).toBe('TX'); + expect(item.tags).toContain('flood:minor'); + expect(item.summary).toContain('was in minor flood at 25.79 ft'); + }); + + test('a forecast with no reading and a year-zero timestamp is not a row', () => { + expect(gaugeItem(gauge(), 'forecast')).toBeNull(); + }); + + test('observed and forecast are different rows for the same gauge', () => { + const g = gauge({ + status: { + ...gauge().status, + forecast: { + primary: 25.7, + primaryUnit: 'ft', + floodCategory: 'minor', + validTime: '2026-09-09T18:00:00Z', + }, + }, + }); + const observed = normaliseItem(gaugeItem(g, 'observed')); + const forecast = normaliseItem(gaugeItem(g, 'forecast')); + expect(observed.externalId).not.toBe(forecast.externalId); + expect(forecast.kind).toBe('river-forecast'); + expect(forecast.summary).toContain('is forecast to be'); + }); + + test('a gauge whose only news is that it is fine is dropped', () => { + const fine = gauge({ + status: { + ...gauge().status, + observed: { ...gauge().status.observed, floodCategory: 'no_flooding' }, + }, + }); + expect(gaugeItem(fine, 'observed')).toBeNull(); + }); +}); + +describe('coastal water levels', () => { + const thresholds = { + nos_minor: 10.19, + nos_moderate: 11.12, + nos_major: 12.39, + nws_minor: 10.49, + nws_moderate: 11.74, + nws_major: 13.24, + action: 10.29, + }; + + test('the worst threshold passed is the one reported', () => { + // A reading over the major stage is over the minor one too; reporting the + // minor would understate a flood by two categories. + expect(exceedance(13.5, thresholds).category).toBe('major'); + expect(exceedance(11.9, thresholds).category).toBe('moderate'); + expect(exceedance(10.6, thresholds).category).toBe('minor'); + expect(exceedance(10.3, thresholds).category).toBe('action'); + expect(exceedance(4.2, thresholds)).toBeNull(); + }); + + test('the NWS stage is preferred, so a reading and a flood warning agree', () => { + // 10.3 clears NOAA's own minor stage of 10.19 but not the NWS's 10.49. + // The alerts in the weather collection are written against the NWS one. + expect(exceedance(10.3, thresholds).category).toBe('action'); + expect(exceedance(10.6, thresholds).threshold).toBe(10.49); + }); + + test('a station with no published thresholds says so instead of guessing', () => { + expect(exceedance(9.9, null)).toBeNull(); + const item = coopsItem({ + station: '8518750', + name: 'The Battery', + state: 'NY', + lat: 40.7, + lon: -74.01, + reading: { t: '2026-09-09 17:06', v: '0.65' }, + thresholds: null, + datum: 'MLLW', + units: 'english', + }); + expect(item.data.floodCategory).toBeNull(); + expect(item.summary).toContain('No flood thresholds are published'); + }); + + test('a reading is stamped as GMT, because that is what was asked for', () => { + // datagetter writes `2026-09-09 17:06` with no zone at all. Stored without + // the Z, a Pacific gauge reads as having reported eight hours early. + const item = normaliseItem( + coopsItem({ + station: '8518750', + name: 'The Battery', + state: 'NY', + lat: 40.7, + lon: -74.01, + reading: { t: '2026-09-09 17:06', v: '10.62' }, + thresholds, + datum: 'MLLW', + units: 'english', + }), + ); + expect(item.publishedAt.toISOString()).toBe('2026-09-09T17:06:00.000Z'); + expect(item.data.floodCategory).toBe('minor'); + expect(item.data.overThresholdBy).toBe(0.13); + expect(item.tags).toContain('flooding'); + }); +}); + +describe('the drought monitor', () => { + const row = { + mapDate: '2026-09-01T00:00:00', + stateAbbreviation: 'DE', + none: 5.51, + d0: 94.49, + d1: 48.8, + d2: 10.79, + d3: 0, + d4: 0, + validStart: '2026-09-01T00:00:00', + validEnd: '2026-09-07T23:59:59', + }; + + test('the classes are cumulative, and the per-class share is derived, not assumed', () => { + // d0 is "D0 or worse". Adding d0..d4 would report 154% of Delaware in + // drought, which is the standard mistake with this dataset. + const inClass = byClass(row); + expect(inClass.d0).toBe(45.69); + expect(inClass.d1).toBe(38.01); + expect(inClass.d2).toBe(10.79); + expect(inClass.d3).toBe(0); + const total = Object.values(inClass).reduce((a, b) => a + b, 0) + row.none; + expect(total).toBeCloseTo(100, 1); + }); + + test('rounding that crosses never produces a negative share', () => { + expect(byClass({ d0: 10.0, d1: 10.01, d2: 0, d3: 0, d4: 0 }).d0).toBe(0); + }); + + test('the worst class with any area in it is the headline', () => { + expect(worstClass(row)).toMatchObject({ key: 'd2', label: 'severe drought' }); + expect(worstClass({ d0: 0, d1: 0, d2: 0, d3: 0, d4: 0 })).toBeNull(); + expect(CLASSES).toHaveLength(5); + }); + + test('a state row carries both readings of the numbers', () => { + const item = normaliseItem( + droughtItem(row, { area: 'Delaware', areaType: 'state', fips: '10-delaware' }), + ); + expect(item.title).toContain('94.49% abnormally dry or worse'); + expect(item.data.cumulative.d0).toBe(94.49); + expect(item.data.inClass.d0).toBe(45.69); + expect(item.data.cumulativeNote).toContain('OR WORSE'); + expect(item.tags).toContain('drought:d2'); + }); + + test('the areas of interest are FIPS numbers, because postal codes return nothing', () => { + // `aoi=IA` is accepted and answers `[]`, which looks exactly like a week + // with no drought in Iowa rather than like a wrong request. + const keys = Object.keys(STATE_FIPS); + expect(keys).toHaveLength(52); + for (const k of keys) expect(k).toMatch(/^\d{2}$/); + expect(STATE_FIPS['19']).toBe('Iowa'); + }); +});