Skip to content

Commit f8bbd36

Browse files
cschanhniemcschanhniem
committed
feat: add DATASETS.md global cross-pack dataset registry
Index 40+ open datasets across climate, health surveillance, food security, education, energy access, population, disaster risk, and air quality — linked to relevant problem packs and annotated with grain, coverage, and use cases. Includes dataset quality requirements and instructions for adding new datasets. This gives agents and researchers immediate discovery of what data is available before they pick up a task.
1 parent d3289b4 commit f8bbd36

1 file changed

Lines changed: 114 additions & 0 deletions

File tree

‎DATASETS.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Global Dataset Registry
2+
3+
A cross-problem reference to open datasets used across Open Problem Lab problem packs. Every entry here is or should be documented in the relevant pack's `datasets.md` and `evidence.json`.
4+
5+
This file is maintained as a discovery aid. It is not canonical — the canonical dataset record lives in the problem pack. When a dataset is accepted into a pack, update the pack's `datasets.md` and `evidence.json`, then update this index.
6+
7+
---
8+
9+
## Climate and Earth Observation
10+
11+
| Dataset | What it covers | Packs |
12+
|---------|---------------|-------|
13+
| **ERA5-Land** (Copernicus/ECMWF) | Temperature, rainfall reanalysis globally at 9km, monthly/hourly from 1950 | malaria-early-warning-africa, dengue-heat-vietnam, heat-stress-urban-south-asia, drought-early-warning-horn-of-africa |
14+
| **CHIRPS** (Climate Hazards Group) | Rainfall estimates at 0.05°, daily from 1981, strong Sub-Saharan Africa coverage | malaria-early-warning-africa, locust-outbreak-east-africa, drought-early-warning-horn-of-africa |
15+
| **MODIS LST** (NASA) | Land surface temperature at 1km, Terra/Aqua 8-day and monthly | heat-stress-urban-south-asia, coral-bleaching-great-barrier-reef |
16+
| **Sentinel-2 / Landsat** (ESA / USGS) | Multispectral land cover, 10m–30m, free access | deforestation-amazon, locust-outbreak-east-africa |
17+
| **PROBA-V / MODIS NDVI** | Vegetation indices weekly/monthly for drought and locust monitoring | locust-outbreak-east-africa, drought-early-warning-horn-of-africa |
18+
| **NOAA Coral Reef Watch (CRW)** | Sea surface temperature, thermal stress alerts for bleaching | coral-bleaching-great-barrier-reef |
19+
| **NASA GRACE / GRACE-FO** | Satellite gravimetry for groundwater mass changes monthly | groundwater-depletion-india, glacial-melt-hindu-kush |
20+
| **ICIMOD glacier inventories** | Hindu Kush Himalaya glacier mass balance and extent | glacial-melt-hindu-kush |
21+
| **NASA WorldWind / Copernicus DEMs** | Digital elevation models for sea-level and flood exposure | sea-level-rise-small-islands, cyclone-early-warning-bangladesh |
22+
| **ECMWF SEAS5** | 7-month seasonal forecast ensemble for rainfall and temperature | malaria-early-warning-africa, drought-early-warning-horn-of-africa |
23+
24+
## Health Surveillance
25+
26+
| Dataset | What it covers | Packs |
27+
|---------|---------------|-------|
28+
| **WHO Global Health Observatory** | Country-level mortality, disease burden, intervention coverage | malaria-early-warning-africa, dengue-heat-vietnam, stunting-sub-saharan-africa |
29+
| **DHIS2** | Sub-national health management information systems, varies by country | malaria-early-warning-africa, dengue-heat-vietnam |
30+
| **DHS (Demographic and Health Surveys)** | Nationally representative household surveys, 90+ countries | stunting-sub-saharan-africa, open-defecation-india, indoor-air-pollution-sub-saharan-africa |
31+
| **WHO Global Database on Child Growth** | Stunting, wasting, underweight estimates by country and year | stunting-sub-saharan-africa |
32+
| **WHO/UNICEF JMP** | Water, sanitation, hygiene coverage at country and sub-national level | open-defecation-india |
33+
| **AIMS (Australian Institute of Marine Science)** | Great Barrier Reef monitoring including bleaching surveys | coral-bleaching-great-barrier-reef |
34+
| **NHMRC Lead Exposure Databases** | Blood lead level surveys in LMICs | lead-exposure-urban-global |
35+
36+
## Food Security and Agriculture
37+
38+
| Dataset | What it covers | Packs |
39+
|---------|---------------|-------|
40+
| **FEWS NET** | Food security early warning for Sub-Saharan Africa, Central America, South Asia | drought-early-warning-horn-of-africa, locust-outbreak-east-africa |
41+
| **IPC (Integrated Food Security Phase Classification)** | Food security phases at country and sub-national scale | drought-early-warning-horn-of-africa |
42+
| **FAO GIEWS** | Global crop monitoring and early warning | drought-early-warning-horn-of-africa, locust-outbreak-east-africa |
43+
| **ECMWF CAMS** | Atmospheric dust and wind for locust wind trajectory modeling | locust-outbreak-east-africa |
44+
| **World Food Programme VAM** | Market prices, food access monitoring | drought-early-warning-horn-of-africa |
45+
| **APHLIS** | Post-harvest loss data for Sub-Saharan Africa cereals | food-security context |
46+
47+
## Education
48+
49+
| Dataset | What it covers | Packs |
50+
|---------|---------------|-------|
51+
| **USAID EdData** | Learning assessments in LMICs | learning-loss-post-pandemic |
52+
| **PASEC / SACMEQ** | Regional learning assessments in Sub-Saharan Africa | learning-loss-post-pandemic, girls-education-sub-saharan-africa |
53+
| **UNESCO Institute for Statistics (UIS)** | Enrollment, completion, literacy rates nationally | girls-education-sub-saharan-africa, learning-loss-post-pandemic |
54+
| **World Bank MICS** | Multiple Indicator Cluster Surveys including education | girls-education-sub-saharan-africa |
55+
56+
## Energy Access
57+
58+
| Dataset | What it covers | Packs |
59+
|---------|---------------|-------|
60+
| **Global Solar Atlas** (World Bank) | Long-run average solar irradiance globally | mini-grid-rural-sub-saharan-africa |
61+
| **ESMAP / SE4All Global Tracking Framework** | Electrification rates and access tiers nationally | mini-grid-rural-sub-saharan-africa, clean-cooking-sub-saharan-africa |
62+
| **WHO Household Energy Database** | Fuel type and cooking technology surveys | indoor-air-pollution-sub-saharan-africa, clean-cooking-sub-saharan-africa |
63+
| **Open Street Map settlements** | Rural settlement location data | mini-grid-rural-sub-saharan-africa |
64+
| **WorldPop** | Population distribution at 100m globally | mini-grid-rural-sub-saharan-africa, cyclone-early-warning-bangladesh |
65+
66+
## Population and Poverty
67+
68+
| Dataset | What it covers | Packs |
69+
|---------|---------------|-------|
70+
| **WorldPop** | Gridded population at 100m globally | most packs |
71+
| **GPW (Gridded Population of the World)** | Population count and density globally | most packs |
72+
| **World Bank PovcalNet / Poverty and Inequality Platform** | Poverty rates and distribution | stunting-sub-saharan-africa, lead-exposure-urban-global |
73+
| **GHSL (Global Human Settlement Layer)** | Built-up area, urban form globally | heat-stress-urban-south-asia, lead-exposure-urban-global |
74+
75+
## Disaster Risk
76+
77+
| Dataset | What it covers | Packs |
78+
|---------|---------------|-------|
79+
| **NOAA IBTrACS** | Global tropical cyclone track archive | cyclone-early-warning-bangladesh |
80+
| **GEBCO / ETOPO** | Bathymetry and coastal topography | sea-level-rise-small-islands, cyclone-early-warning-bangladesh |
81+
| **Global Earthquake Model (GEM) OpenQuake** | Seismic hazard models globally | earthquake-vulnerability-nepal |
82+
| **NDRRMA Nepal building surveys** | Post-Gorkha earthquake building damage classification | earthquake-vulnerability-nepal |
83+
| **NASA SRTM / Copernicus DEM** | Elevation for flood and surge exposure | cyclone-early-warning-bangladesh, sea-level-rise-small-islands |
84+
85+
## Air Quality
86+
87+
| Dataset | What it covers | Packs |
88+
|---------|---------------|-------|
89+
| **Atmospheric Composition Analysis Group (ACAG)** | Global PM2.5 satellite-derived estimates | pm25-monitoring-south-asia |
90+
| **OpenAQ** | Aggregated low-cost sensor and reference monitor data | pm25-monitoring-south-asia |
91+
| **WHO Global Ambient Air Quality Database** | Country-level PM2.5 concentrations and monitoring coverage | pm25-monitoring-south-asia, indoor-air-pollution-sub-saharan-africa |
92+
| **Global Burden of Disease (IHME)** | Health burden estimates attributable to air pollution | pm25-monitoring-south-asia, indoor-air-pollution-sub-saharan-africa |
93+
94+
---
95+
96+
## Dataset Quality Requirements
97+
98+
Before a dataset is accepted into a problem pack's `evidence.json`, it must have:
99+
- Verified URL with permanent or archived access
100+
- Source date and access date
101+
- Geographic grain documented
102+
- Case definition or variable definitions documented
103+
- License or reuse permission documented
104+
- Known limitations listed
105+
- Denominator source (where applicable)
106+
107+
Use `pnpm verify:sources` to check URL availability after adding or modifying evidence records.
108+
109+
## Adding a Dataset
110+
111+
1. Add the dataset to the relevant pack's `datasets.md` with source, grain, status, and intended use.
112+
2. Add an `evidence.json` record with URL, date, method, and limitations.
113+
3. Update this file under the appropriate category.
114+
4. Run `pnpm validate` and `pnpm verify:sources`.

0 commit comments

Comments
 (0)