feat: curate records into trainable datasets (SFT/preference/rollout export) (#96)

[dcfocus]

Summary

Adds the downstream half of the post-training pipeline (#96): curate stored ContextRecords into a lifecycle-correct, deduped/decontaminated cut and export it as SFT / preference / RL-rollout JSONL plus a reproducible manifest.

Independent of the other open PRs — branches from main.

This is the first slice the issue calls for ("first implementation can target JSONL plus a manifest"). See Scope below for what's intentionally deferred.

Core (lance-context-core::export)

• Curation (selection only — never deletes/tombstones): keeps default-visible records and additionally drops contradicted; optional min_reward threshold (rejection sampling / Best-of-N); semantic dedup by cosine distance; decontamination against a holdout-embedding set.
• Grouping/aggregation by session_id / run_id / tenant / source / bot_id / external_id prefix, with deterministic ordering by (created_at, id).
• Three export shapes:
  • SftExample — ordered messages (also the rejection-sampling target via min_reward).
  • PreferenceExample — paired chosen/rejected (DPO/SimPO/ORPO), unpaired binary label (KTO), or ranked N-way (judge/RULER).
  • RolloutExample — prompt + rewarded responses with group_id (GRPO/RLVR groups), reward, reward_source.
• Provenance (context URI, version, record/external ids, tenant/source/bot/session/run, created_at range) and an ExportManifest (version, task, group_by, schema version, curation params, filters, created_at range, source record ids, per-stage counts).
• ContextStore::export_training(config, output_path) streams JSONL line-by-line to disk, writes a sibling <path>.manifest.json, and pins to a dataset version (time-travel) with restore-after.

Field conventions (documented on the module): reward/preference signals live in each record's metadata — reward, reward_source, group_id, label (chosen/rejected), rank. Messages are role + text_payload.

Python

manifest = ctx.export_training(
    "training/sft.jsonl",
    task="sft",                 # "sft" | "preference" | "rollout"
    group_by="session_id",
    filters={"tenant": "acme"},
    min_reward=0.7,             # rejection sampling
    dedup_threshold=0.02,
    version=ctx.version(),      # reproducible cut
)

Tests

• core (12): SFT grouping + ordering, rejection sampling, all three preference forms, rollout groups, dedup, decontamination, contradicted filtering, version pinning + restore, reproducibility, external-id-prefix grouping, manifest sibling file.
• python (8): mirrors the surface (SFT, rejection sampling, paired/unpaired preference, rollout, dedup, reproducibility, invalid-task error).

README gains a curate + export example.

Checks

• cargo test -p lance-context-core --lib — 59 passed
• cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings — clean
• ruff format --check, ruff check, pyright — clean
• test_export_training.py — 8 passed

Acceptance criteria (#96)

• Deduped/decontaminated, lifecycle-correct, versioned curated cut
• SFT JSONL grouped by session + rejection-sampling variant filtered by a reward threshold
• Preference JSONL in paired / unpaired (KTO) / N-way ranked forms
• RL rollout rows with prompt, response(s), reward/reward_source, optional group_id, multi-turn trajectory metadata
• Manifest with version, curation params, filters/selectors, schema version, created_at range, source record ids
• Streaming/chunked export (line-by-line writes; full JSONL never materialized)
• Tests cover dedup/decontamination, lifecycle filtering, version pinning, grouping order, reproducibility
• Namespace partial-selector fan-out — deferred (see below)

Scope / deferred follow-ups

Per the issue's own audit, these are intentionally out of this first slice and called out for follow-up PRs:

• Namespace partial-selector fan-out (phase-2 — needs cross-partition read design; the audit flags it as new work).
• User-supplied enrichment hooks (reward/verifier/judge/PII/summarization callbacks across the FFI boundary).
• Parquet/Arrow output (JSONL first, "once the schema stabilizes").
• REST POST /exports/training endpoint.

Note: curation is append-only selection (records are filtered for the cut, never deleted/tombstoned), matching the audit's guidance not to conflate curation with record removal.

Closes #96