Automate, harmonize, and standardize reference data dimensions across global enterprise systems with AI-assisted semantic matching and intuitive human-in-the-loop curation workflows.
1. AI Semantic Matching & Playground |
2. Canonical Reference Library |
3. Source System Connections & Profiling |
4. Match Insights & Curation Workflow |
RefData Hub is an open-source Reference Data Management (RDM) and Data Harmonization platform designed to solve reference data fragmentation across modern enterprise architectures.
In inconsistent multi-system environments, raw reference values—such as country codes, employment statuses, education levels, or marital statuses—vary widely between source databases, analytics pipelines, and third-party APIs. RefData Hub establishes a single source of truth by:
- Ingesting & Profiling raw source data from upstream databases.
- Semantically Matching disparate raw values to canonical dimensions using AI embedding vectors or LLM-powered re-ranking.
- Empowering Reviewers with a modern human-in-the-loop dashboard to inspect, approve, or refine suggestions.
- Publishing Canonical Mappings to downstream data warehouses, business intelligence tools, and microservices.
- 🧠 AI-Assisted Semantic Matching: Harmonize messy reference data automatically using local sentence-transformer embeddings (fastembed), or configurable LLM providers (OpenAI API or local offline Ollama models) — including optional LLM escalation that re-ranks near-tied embedding results.
- 🛡️ Human-in-the-Loop Curation: A streamlined dashboard for data stewards and reviewers to inspect confidence scores, approve auto-suggestions, or manually map complex edge cases.
- 🔗 Source System Connectors: Connect directly to upstream databases — PostgreSQL, MySQL, ClickHouse, Snowflake, SQL Server, Oracle, MongoDB, SQLite, DuckDB, or a generic SQL driver — inspect live schemas, and sample unstandardized values in real time.
- 📐 Canonical Library & Custom Schemas: Define flexible reference dimensions with custom JSON metadata schemas (e.g., ISO codes, regional attributes, sorting priority).
- 📊 Match Insights & Coverage Analytics: Track overall harmonization health, percentage of standardized values, and unmapped raw value frequencies across source systems.
- 🌳 Dimension Hierarchies & Relations: Model multi-level parent/child relationships (e.g., Country → State → District).
- 📁 Bulk Import & Export: Effortlessly import/export canonical values and field mappings via CSV, TSV, or Excel formats.
- 🎨 12-Palette Themeable UI: Built with React 18, Vite, and Tailwind CSS, with twelve palettes plus a
Systemoption that follows your OS. The selection persists inlocalStorageand is shared with the documentation site, so both stay on the same theme.
RefData Hub acts as an intelligent middleware layer between operational source systems and downstream reporting/data warehouses:
┌─────────────────┐ ┌────────────────────────────────────────────────────────┐ ┌─────────────────────┐
│ Source Systems │ │ RefData Hub │ │ Downstream Consumers│
│ (DBs, APIs, │ ────> │ ┌──────────────┐ ┌───────────────┐ ┌───────────┐ │ ────> │ (Data Warehouses, │
│ Legacy Data) │ │ │ Schema & │ │ AI Semantic │ │ Reviewer │ │ │ BI & Analytics, │
└─────────────────┘ │ │ Connectors │ ─>│ Match Engine │ ─>│ Dashboard │ │ │ Microservices) │
│ └──────────────┘ └───────────────┘ └───────────┘ │ └─────────────────────┘
└────────────────────────────────────────────────────────┘
- Extraction & Sampling: Source connectors pull distinct raw values and record counts.
- Semantic Vector Matching: Matcher scores candidates using cosine vector similarity or LLMs.
- Approval Thresholds: High-confidence matches auto-approve while uncertain values route to human reviewers.
- Central Repository: Approved canonical mappings are persisted and accessible via REST APIs.
Get the complete RefData Hub platform running with a single command:
docker compose up --build- 🎨 Reviewer UI (Dashboard): http://localhost:5274
- ⚡ Backend REST API Docs (Swagger): http://localhost:8000/docs
- 🗄️ Primary Application DB (PostgreSQL 15):
localhost:5432 - 🧪 Bundled Demo Target DB (PostgreSQL 15):
localhost:5433
💡 Lightweight by Default: The default
docker compose upstack starts PostgreSQL, the FastAPI backend, the React frontend, and a seeded target database. It requires minimal memory and CPU resources.Note the port difference: compose serves the UI on 5274, while
npm run devserves it on 5173.
Heavy preloaded LLM images are not downloaded by default to conserve bandwidth and system memory.
If you want to use local offline LLM semantic matching, start the optional Ollama service profile:
# Launch the core stack WITH the optional local Ollama LLM runtime
docker compose --profile ollama up --buildWhen running the optional Ollama container, pull a compact, lightweight model such as smollm:135m or tinyllama:
docker compose exec ollama ollama pull smollm:135mYou can then select Offline Ollama in the Settings workspace of the Reviewer UI or configure REFDATA_LLM_MODEL=smollm:135m in your environment.
A single end-to-end pass through the reviewer console — dashboard metrics, the canonical library, source profiling, and the approval queue.
| Component | Technology | Description |
|---|---|---|
| Backend API | FastAPI + Python 3.10+ | High-performance asynchronous REST API |
| ORM / Database | SQLModel + PostgreSQL 15 | Type-safe SQL querying and schema migrations |
| Frontend UI | React 18 + Vite + Tailwind CSS | Fast, themeable single-page application |
| Matching Engine | fastembed + OpenAI / Ollama | ONNX sentence-transformer embeddings & LLM re-ranking |
| Testing | Pytest & Vitest | Complete unit, integration, and UI test coverage |
| Containerization | Docker & Docker Compose | Multi-container service orchestration |
Run from the repository root. REFDATA_DATABASE_URL is required for the API
server; without it the app falls back to the compose-internal hostname db:5432,
which does not resolve outside Docker.
# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install dependencies
pip install -r api/requirements.txt pytest pytest-cov
# Point the API at a reachable database
export REFDATA_DATABASE_URL="postgresql+psycopg://refdata:refdata@localhost:5432/refdata"
export REFDATA_CORS_ORIGINS="http://localhost:5173"
# Run API server locally
uvicorn app.main:app --app-dir api --reload --port 8000# Navigate to UI directory
cd reviewer-ui
# Install dependencies
npm install
# Start Vite dev server (http://localhost:5173)
npm run dev
# Run UI test suite
npm testThe UI defaults to http://localhost:8000 for API calls on localhost. Set
VITE_API_BASE_URL to point it elsewhere.
The test suite lives in tests/ at the repository root, so run pytest from
there. Tests use an in-memory SQLite database and need no running Postgres.
# Backend — from the repository root
pytest
# With coverage
pytest --cov=app --cov-report=html
# Integration tests only
pytest -m integration
# Frontend
cd reviewer-ui && npm testpip install -r requirements-docs.txt
# Port 8000 is the API, so serve the docs on 8001
mkdocs serve -a localhost:8001The docs theme is generated from the app's stylesheet. After changing a palette
in reviewer-ui/src/styles.css, regenerate it with:
python scripts/generate_docs_tokens.pyFull documentation is published at → https://rezer-bleede.github.io/refdata-hub/
The hosted site is the canonical, always-current version of every guide below.
It ships the same 12-palette theme switcher as the Reviewer UI, so it reads as
part of the product rather than a separate website. The Markdown sources for
each page are committed in docs/ and build locally with
mkdocs serve.
| Guide | Description |
|---|---|
| 🚀 Quickstart Guide | Detailed walkthrough for Docker and local setup |
| Guide | Description |
|---|---|
| 🏠 Features Overview | What the platform does, capability by capability |
| 📊 Dashboard | Metric tiles and the semantic match playground |
| 🧱 Dimensions | Dimension definitions, attribute schemas, and parent/child relations |
| 📚 Canonical Library | Curating golden records, bulk import, and attributes |
| Guide | Description |
|---|---|
| 🔌 Source Connections | Register upstreams, test connectivity, introspect live schemas |
| 🗺️ Field Mappings | Bind source columns to dimensions and capture samples |
| Guide | Description |
|---|---|
| 📈 Match Insights | Coverage rates, top matched and unmatched values |
| 💡 Suggestions | Reviewing and approving semantic match suggestions |
| 🕓 Mapping History | Audit trail plus CSV/XLSX round-trip |
| Guide | Description |
|---|---|
| ⚙️ System Settings | Match thresholds, embedding models, LLM escalation |
| 🏛️ Architecture Guide | Deep dive into system components and vector matching |
| 🗄️ Database Schema Reference | Complete ERD, table specs, and migration guides |
| 📡 REST API Documentation | API endpoints, payloads, and integration examples |
| 🔧 Configuration Reference | Environment variables, matcher settings, and tuning |
| 🛠️ Developer Guide | Local development, testing, and contribution patterns |
| 🧪 Testing Guide | Test strategies and execution |
| ☁️ Deployment Guide | Production Docker, Kubernetes, and Cloudflare Pages setup |
| 🆘 Troubleshooting | Common issues and solutions |
We welcome contributions from the community! Please read our CONTRIBUTING.md guide for details on submitting pull requests, coding standards, and running tests.
This project is open-source software licensed under the MIT License.