feat(evaluation) 6/15: eval_datasets and eval_runs persistence

[Ahmath-Gadji]

Part 6 of 15 of the split of #811. Targets eval/05-promptfoo-config (#816).

What

core/ports/evaluation_repo.py	the EvaluationRepository contract
services/persistence/evaluation_repo.py	asyncpg implementation
services/persistence/schema.py	eval_datasets, eval_runs
migrations/.../a7c9e1f2b3d4_add_evaluation_tables.py	the migration
postgres_store.py, catalog_store.py, di/container.py	expose it on the store and the container

Notable

• The store enforces one run at a time, not the orchestrator. ux_eval_runs_single_active is a partial unique index over an always-true expression, so it admits exactly one row in QUEUED/INDEXING/EVALUATING; create_run maps the resulting UniqueViolationError to ConflictError. A read-then-insert in the service would be a race, and not a harmless one — starting a run regenerates a shared service user's token (part 8), so two racing starts that both passed a read check would revoke each other's credentials mid-indexing.
• Metrics are JSONB, not columns. The metric set is expected to grow, and the dataclasses in core/models/evaluation.py already define the shape. _row_to_run reconstructs them, including lifting samples back out of the indexing payload.
• _load exists because asyncpg returns JSONB as str unless a codec is registered on the pool; the repo does not own the pool, so it decodes defensively instead.
• created_by is ON DELETE SET NULL, matching how files.created_by behaves — deleting an admin should not cascade away the run history they produced. eval_runs.dataset_id is ON DELETE CASCADE: a run without its dataset is not meaningful.
• The migration is idempotent per this repo's alembic rules (CLAUDE.md): Base.metadata.create_all() runs at startup, so a freshly bootstrapped database already has these tables before alembic sees them. Every op is guarded by table_exists / index_exists, in both directions.

Testing

ruff, format check, the layer-import guard, and the full unit suite (2258 passed) — including the container and catalog-store wiring tests, which the new abstract property could have broken. The one failure, test_content_deduplication_can_be_disabled_by_env, reproduces on develop.

[coderabbitai]

Warning

Review limit reached

@EnjoyBacon7, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8a7f0b08-1b76-4b15-87ad-26f9bed97896

📥 Commits

Reviewing files that changed from the base of the PR and between 35e2f62 and 37d4787.

📒 Files selected for processing (7)

• openrag/core/ports/catalog_store.py
• openrag/core/ports/evaluation_repo.py
• openrag/di/container.py
• openrag/services/persistence/evaluation_repo.py
• openrag/services/persistence/migrations/alembic/versions/a7c9e1f2b3d4_add_evaluation_tables.py
• openrag/services/persistence/schema.py
• openrag/services/storage/postgres_store.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch eval/06-persistence

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Ahmath-Gadji]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

---

Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 57 minutes.