SEC Filing Intelligence Lakehouse

A Databricks Lakehouse that ingests SEC EDGAR filing metadata and XBRL financial facts for a universe of public companies, transforms them through a Bronze → Silver → Gold medallion architecture, and surfaces financial KPIs, sector comparisons, and anomaly signals.

Built on Databricks with PySpark, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines, and Databricks Asset Bundles for code-as-infrastructure deployment.

Demo

An interactive dashboard (financial KPIs, sector comparisons, a composite health score, peer clusters, and net-income-growth predictions) runs live on Databricks Apps.

The live app runs inside a Databricks workspace and requires a workspace login, so it isn't publicly viewable. The screenshots below show the full app — live demo available on request.

Screenshots

Financial KPIs & sector comparison

ML insights — health score, peer clusters, and growth predictions

Architecture

SEC EDGAR APIs (submissions + companyfacts)
        │  Python ingestion notebook — the only imperative step
        ▼
Raw     SEC JSON files in a Unity Catalog Volume · ingestion audit table
        │  ─── Lakeflow Declarative Pipeline (SQL) from here down ───
        ▼
Bronze  raw submissions · raw company facts (read_files from the Volume)
        │
        ▼
Silver  companies · filings · financial facts · standardized concepts
        │
        ▼
Gold    company KPIs · financial trends · sector summaries · outliers
        │
        ▼
Dashboards + anomaly intelligence

Ingestion is deliberately the single imperative step: a notebook fetches each company's filings over HTTP and lands the responses verbatim in a governed Volume. Everything downstream is declarative SQL, so the medallion layers are reproducible and lineage-tracked.

Data sources

• https://data.sec.gov/submissions/CIK##########.json — filing metadata
• https://data.sec.gov/api/xbrl/companyfacts/CIK##########.json — XBRL financial facts

Every request sends a descriptive User-Agent per SEC fair-access policy and is rate-limited well under the published request ceiling. Raw SEC data is not committed to this repository.

Project layout

config/
  company_universe.csv        companies to ingest (ticker, CIK, name, sector)
  financial_concepts.yml      friendly KPI names → US-GAAP XBRL tags
src/
  notebooks/ingest_sec_raw.py SEC ingestion → raw JSON landed in a UC Volume
  pipelines/medallion/        Lakeflow SQL transformations (Bronze/Silver/Gold)
resources/
  job.yml                     Databricks Jobs (ingestion)
  pipeline.yml                Lakeflow Declarative Pipeline (medallion)
databricks.yml                Asset Bundle definition + deploy target

Configuration

• config/company_universe.csv — the company universe. CIKs are stored zero-padded to 10 digits to match the SEC URL format.
• config/financial_concepts.yml — maps each KPI to the ordered list of US-GAAP tags that may represent it, since companies tag the same concept differently across filings and years. The Silver layer uses this to standardize metrics.

Getting started

Prerequisites

• A Databricks workspace. Everything here runs on Databricks Free Edition (serverless + Photon) — no paid compute required. Tables land in the default workspace catalog.
• The Databricks CLI authenticated to your workspace. This bundle's dev target reads its host from a CLI profile named sec-lakehouse (see databricks.yml). Either create one with that name (databricks configure --profile sec-lakehouse) or change that single line to your own profile.

No personal values are committed. Two bundle variables must be supplied locally:

• user_agent_email — your contact email, sent to the SEC in the User-Agent header (their fair-access policy requires a real contact).
• warehouse_id — the SQL warehouse that backs the dashboard. Find it with databricks warehouses list -p sec-lakehouse (Free Edition has a default serverless one).

Recommended — a git-ignored variables.local.yml (set once, never re-typed). databricks.yml pulls in *.local.yml via its include: list, and .gitignore excludes it, so your values stay out of version control. Create it in the repo root:

# variables.local.yml  (git-ignored)
targets:
  dev:
    variables:
      warehouse_id: <your-sql-warehouse-id>
      user_agent_email: you@domain.com

databricks bundle deploy then just works in any shell — no exports. Confirm with databricks bundle validate (should print Validation OK!).

Alternative — per-shell environment variables (handy for CI):

export BUNDLE_VAR_user_agent_email="you@domain.com"  # SEC fair-access contact
export BUNDLE_VAR_warehouse_id="<your-sql-warehouse-id>"  # for the dashboard

You can also pass them inline with --var user_agent_email=... --var warehouse_id=....

Deploy and run

From the repository root:

databricks bundle validate -t dev
databricks bundle deploy -t dev

# 1. Ingest raw SEC JSON into the Volume + audit table
databricks bundle run sec_ingest_job -t dev

# 2. Land the sector reference CSV where the Gold layer reads it (one time)
databricks fs cp config/company_universe.csv \
  dbfs:/Volumes/workspace/raw/sec/reference/company_universe.csv \
  --profile sec-lakehouse

# 3. Build Bronze -> Silver -> Gold
databricks bundle run sec_medallion_pipeline -t dev

The ingestion job reads company_universe.csv, calls the SEC APIs, and writes raw JSON files into the raw.sec Volume plus an append-only raw.sec_ingestion_audit table recording the status, size, and timing of every request. The medallion pipeline then transforms that raw JSON into the Bronze/Silver/Gold tables.

Dashboard

The AI/BI dashboard is bundle-managed as code (src/sec_filing_intelligence.lvdash.json). databricks bundle deploy creates or updates the draft; publish the snapshot to make it viewable — either click Publish in the dashboard UI, or run databricks lakeview publish <dashboard-id>.