offline-dns

Offline DNS backed by static datasets.

Collector

The collector/ image runs a self-contained pipeline that keeps a fresh, flattened snapshot of OpenINTEL fDNS data in S3:

1. download — fetches .gz.parquet partitions for the recent N days from the public OpenINTEL bucket (object.openintel.nl) via ListObjectsV2.
2. aggregate — flattens every record into a tidy long-format parquet: domain | record_type | record_value | count, zstd-compressed, sorted.
3. upload — pushes the result to an S3-compatible bucket with curl --aws-sigv4.

The same image runs on a laptop (minio), in Kubernetes, or in CI — all behavior is driven by environment variables.

Repository layout

collector/
├── aggregate_dns.ts      DuckDB flatten step (npm:@duckdb/node-api)
├── download_openintel.ts ListObjectsV2 fetch step (fetch)
├── entrypoint.sh         orchestrates download → aggregate → upload
└── Dockerfile            deno + curl(aws-sigv4) image
.github/workflows/
└── collector.yml         builds the image and pushes it to GHCR

Output dataset

One row per resource record:

column	description
domain	record owner with labels reversed, trailing dot stripped (www.example.com. → com.example.www) for sort locality and compression
record_type	RR type as it appeared in the answer
record_value	the record data, flattened to one string (e.g. A → IP, MX → "10 host.", CAA → "0 issue \"digicert.com\"")
count	occurrences across all input files (deduped mode)

The deduped output is sorted by domain, record_type, record_value and zstd-compressed (.zst.parquet). The object key is <S3_KEY_PREFIX><YYYY/MM/DD>/aggregated.zst.parquet (UTC run date), overridable with S3_KEY.

Configuration

Required:

Variable	Description
S3_ENDPOINT	S3-compatible endpoint, e.g. http://localhost:9000 (minio)
S3_BUCKET	destination bucket
AWS_ACCESS_KEY_ID	access key (used by curl --aws-sigv4)
AWS_SECRET_ACCESS_KEY	secret key

Optional:

Variable	Default	Description
S3_REGION	us-east-1	region for the sigv4 signature
S3_KEY_PREFIX	aggregated/	object key prefix
S3_KEY	derived	full object key override
AWS_SESSION_TOKEN	none	temporary-credentials session token
OPENINTEL_DAYS	7	number of recent days to fetch
OPENINTEL_END_DATE	today	window end (YYYY-MM-DD)
OPENINTEL_PARTITIONS	radar, umbrella, crux{sg,us,cn}	space-separated partition prefixes
AGGREGATE_DEDUPE	1	1 = dedupe + count, 0 = all rows
AGGREGATE_ALL_SECTIONS	0	1 to include authority/additional sections
AGGREGATE_MEMORY	none	DuckDB memory_limit, e.g. 512MB
AGGREGATE_THREADS	auto	DuckDB thread count (defaults to 2 when AGGREGATE_MEMORY is set)

AGGREGATE_MEMORY caps DuckDB's memory and spills the aggregate/sort to a temp file, so the image runs on small-RAM hosts (slower, but bounded). Setting it also lowers the thread count to 2, since per-thread hash-tables must fit.

Run against minio

docker run --rm --network host \
  -e S3_ENDPOINT=http://localhost:9000 \
  -e S3_BUCKET=my-bucket \
  -e AWS_ACCESS_KEY_ID=minioadmin \
  -e AWS_SECRET_ACCESS_KEY=minioadmin \
  -e OPENINTEL_PARTITIONS="fdns/basis=toplist/source=crux/country-code=cn/" \
  -e OPENINTEL_DAYS=1 \
  -e AGGREGATE_MEMORY=256MB \
  ghcr.io/<owner>/offline-dns/collector:latest

Use --network host when the bucket is on localhost (e.g. minio). The default partitions fetch ~5.5 GiB across the last 7 days.

Build

docker build -t offline-dns-collector collector/

CI

.github/workflows/collector.yml builds the image on pushes that touch collector/** and publishes it to the GitHub Container Registry as ghcr.io/<owner>/offline-dns/collector with :latest (on main) and :sha tags. Pull requests build without pushing.

Development

The two TypeScript scripts also run standalone with Deno (no container):

deno run -A collector/download_openintel.ts ./data --days 7 --dry-run
deno run -A collector/aggregate_dns.ts ./data out.zst.parquet --dedupe --memory 1GB

Permissions the image grants: download needs --allow-net --allow-read --allow-write; aggregate additionally needs --allow-ffi for the native DuckDB binding.