FinDataPipeline

Company fundamentals from two vendors — EODHD and FMP — reconciled into a single schema, with the disagreements made visible, plus a Streamlit dashboard and screener over the result.

Two vendors describe the same balance sheet with different field names, different ticker conventions, and different gaps. Anyone using both ends up writing this translation layer. This is that layer, extracted and tested.

Scope is fundamentals only — income statement, balance sheet, cash flow. No prices, dividends, splits, or ESG, and no derived indicators or ratios: this project unifies vendor data, it does not analyze it.

What it does

• Fetches fundamentals from both vendors
• Harmonizes their field names through a 71-row mapping (data/mapping.csv)
• Matches tickers across vendors (AAPL.US ↔ AAPL) through data/tickers.csv, with a company-name fuzzy fallback for the ones that file leaves unmapped
• Merges into one table, primary vendor first, secondary filling gaps
• Reports where the vendors disagree, what each is missing, and which one supplied every value
• Serves all of that in a Streamlit dashboard, including a screener whose results show which vendor supplied each field

The mapping file

The core asset. Each row says what one financial concept is called at each vendor:

internal_name;eod_field;fmp_field
cfo;totalCashFromOperatingActivities;netCashProvidedByOperatingActivities
accountsPayable;accountsPayable;accountPayables
equity;;totalEquity

A blank means that vendor does not carry the field.

Install

git clone https://github.com/vesaias/FinDataPipeline
cd FinDataPipeline
python -m venv .venv
.venv/Scripts/activate      # Windows
# source .venv/bin/activate # macOS / Linux
pip install -e ".[dashboard,dev]"

Configure

You need your own API keys to fetch data — this repo ships no vendor data.

cp settings.example.toml settings.toml

Edit settings.toml and fill in eodhd_api_token and fmp_api_key. Every setting can also be given as an environment variable (FINDATAPIPELINE_RETRIES=5) or a CLI flag (--set retries=5), which take precedence in that order. An unknown key or a malformed --set argument is rejected immediately (exit code 2) rather than silently ignored.

List the companies you want in data/tickers.csv — see data/tickers.example.csv for the format.

Run

findatapipeline fetch       # download raw vendor JSON
findatapipeline normalize   # convert to long-form parquet
findatapipeline unify       # harmonize and merge
findatapipeline reconcile   # compare the vendors
streamlit run dashboard/app.py

Each stage writes to disk, so you can re-run any of them without refetching. fetch accepts a repeatable --ticker flag to refresh a single symbol instead of the whole ticker file, e.g. findatapipeline fetch --ticker AAPL.US. A --ticker that matches nothing in the ticker file is an error, not an empty run.

Trying it without API keys

The dashboard is read-only: it only renders parquet files an earlier CLI run already produced and never calls a vendor API itself, so it needs no API keys.

A small sample is committed at data/output/sample/ — one company, two fiscal years and three quarters, built from the test fixtures — so you can explore the UI before fetching anything. Point the dashboard at it:

FINDATAPIPELINE_OUTPUT_DIR=data/output/sample streamlit run dashboard/app.py

$env:FINDATAPIPELINE_OUTPUT_DIR = "data/output/sample"   # Windows PowerShell
streamlit run dashboard/app.py

It is generated from tests/fixtures/, and a test regenerates it and compares, so it cannot quietly fall out of step with the code.

Tests

pytest

The suite runs with no API keys. Vendor behaviour is covered by small trimmed JSON fixtures in tests/fixtures/; the network layer is deliberately untested.

License

MIT