feat(evaluation) 2/15: test-set parsing and file_id normalisation

[Ahmath-Gadji]

Part 2 of 15 of the split of #811. Targets eval/01-config-and-models (#812) — merge that first.

What

The first two pieces of core/evaluation/, both pure functions with no I/O.

• parse_testset — reads the admin-uploaded CSV (question,expected_answer,expected_file_ids) into EvalTestCase rows, under a row cap. expected_file_ids is optional and ;-separated.
• sanitize_file_id — maps a corpus filename onto an id the indexing API accepts.

Notable

• Why identity.py is its own module. The indexing API allows only [A-Za-z0-9._:-] in a file_id, so a corpus file cannot be uploaded under a raw human filename. Both sides of the ground-truth comparison must therefore normalise: the uploader (part 9) and the metrics (part 4). A test set naming A B.pdf matches the stored A_B.pdf only because one function owns that mapping.
• Validation happens at upload, not at run time. A dataset whose CSV is malformed should fail while the admin is still looking at the upload dialog, not thirty minutes into an indexing phase. Part 7 calls this from create_dataset for that reason.

Testing

12 new unit tests over the parser: header validation, the optional third column, blank and malformed rows, the row cap, and encoding. ruff, format check and the layer-import guard pass.

Summary by CodeRabbit

• New Features
  • Added evaluation utilities to parse and strictly validate uploaded prompt-style CSV test sets.
  • Supports UTF-8 input (including BOM), header normalization, optional expected file IDs, and ignores fully blank rows.
  • Introduced consistent file ID sanitization for indexing compatibility.
• Bug Fixes
  • Improved validation behavior with clear, specific error codes for empty/invalid inputs, missing or duplicate columns, invalid encodings, and exceeding the row limit.
• Tests
  • Added unit tests covering successful parsing, edge cases, and error scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 17f62961-9e7b-4120-a860-137b0bf008e6

📥 Commits

Reviewing files that changed from the base of the PR and between cfa53f9 and 8e008f2.

📒 Files selected for processing (4)

• openrag/core/evaluation/__init__.py
• openrag/core/evaluation/identity.py
• openrag/core/evaluation/testset.py
• tests/unit/core/evaluation/test_testset.py

🚧 Files skipped from review as they are similar to previous changes (3)

• openrag/core/evaluation/identity.py
• tests/unit/core/evaluation/test_testset.py
• openrag/core/evaluation/testset.py

---

📝 Walkthrough

Walkthrough

Adds public evaluation exports, filename sanitization, strict UTF-8 CSV test-set parsing, validation errors, optional file identifiers, row limits, and comprehensive unit tests.

Changes

Evaluation test-set support

Layer / File(s)	Summary
Identity helper and public exports openrag/core/evaluation/identity.py, openrag/core/evaluation/__init__.py	Adds sanitize_file_id and re-exports it with parse_testset from the evaluation package.
Strict CSV test-set parser openrag/core/evaluation/testset.py	Defines the CSV schema, normalizes headers, parses optional file IDs, validates rows and encodings, enforces row limits, and returns ordered EvalTestCase objects.
Parser behavior validation tests/unit/core/evaluation/test_testset.py	Tests valid parsing, optional fields, normalization, blank lines, encoding, schema errors, row errors, duplicate headers, empty input, and row limits.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Upload
  participant parse_testset
  participant EvalTestCase
  Upload->>parse_testset: CSV bytes or text
  parse_testset->>parse_testset: validate headers and rows
  parse_testset->>EvalTestCase: build ordered cases
  parse_testset-->>Upload: cases or ValidationError

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the two main additions: test-set parsing and file_id normalization.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch eval/02-testset-parsing

⚔️ Resolve merge conflicts

• Resolve merge conflict in branch eval/02-testset-parsing

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@openrag/core/evaluation/testset.py`:
- Around line 44-45: Update _split_file_ids in
openrag/core/evaluation/testset.py at lines 44-45 to apply sanitize_file_id() to
each non-empty trimmed ID before returning the tuple. Add a unit test in
tests/unit/core/evaluation/test_testset.py at lines 20-27 asserting that “A
B.pdf” parses to “A_B.pdf”.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2336df91-037c-4fd7-918a-7fc3df9ca5ee

📥 Commits

Reviewing files that changed from the base of the PR and between 7a9a880 and 40c5277.

📒 Files selected for processing (4)

• openrag/core/evaluation/__init__.py
• openrag/core/evaluation/identity.py
• openrag/core/evaluation/testset.py
• tests/unit/core/evaluation/test_testset.py

[hedhoud]

The parser is going in the right direction, but I found two cases that can make the evaluation results unreliable.

Different filenames can currently produce the same file ID. For example, A B.pdf and A_B.pdf both become A_B.pdf. If both are in the corpus, one upload will fail and the metrics may match the wrong file. The generated ID should stay unique, or the upload should reject that collision with a clear error.

Malformed CSV rows can also be silently dropped. A row with only expected_file_ids, or a row with more values than its header, can be accepted without telling the user that some data was ignored. Please reject any non-empty malformed row and add tests for these cases. It would also be useful to report the CSV reader’s real line number when blank lines are present.

Once these cases are covered, this should be safe to move forward.