docs: document the CSV data contract and the repeng consume-not-extract boundary

README.md

@@ -167,6 +167,11 @@ Consumes vectors produced by [repeng](https://github.com/vgel/repeng) —
 extraction is not reimplemented. Ships benign example concepts (formality,
 sentiment, verbosity).
 
+Writing your own sweep producer, or wiring steerbench to another extraction
+tool? [docs/data-contract.md](docs/data-contract.md) is the canonical reference:
+the exact columns of the three sweep CSVs, the raw-one-row-per-seed convention,
+the coherence directions, and where the repeng boundary sits.
+
 ## Related reading
 
 [Same recipe, different curves: a vector that lied](https://bamdad.substack.com/p/same-recipe-different-curves-a-vector) —

docs/data-contract.md

@@ -0,0 +1,189 @@
+# Data contract
+
+This page is the canonical description of what `steerbench` reads and what it
+refuses to produce. Two separate contracts live here:
+
+1. the **CSV schemas** a sweep producer has to emit for `steer-report` to render
+   a card, and
+2. the **repeng boundary** — steerbench *consumes* steering vectors, it does not
+   extract them.
+
+The module docstrings in [`report.py`](../src/steerbench/report.py) and
+[`vectors.py`](../src/steerbench/vectors.py) point here; if the two ever
+disagree, the code wins and this page is the bug.
+
+## 1. CSV schemas
+
+Three CSVs feed the four-part card. They are produced by the GPU sweep in
+`experiments/modal_app.py` (the `steerbench[gpu]` harness) and consumed on CPU
+by `report.load_dose_curve`, `report.load_layer_curve` and
+`report.load_side_effects`.
+
+Rules that hold for all three:
+
+- **Header row required.** All three are read with `csv.DictReader`.
+- **Extra columns are tolerated** and ignored. A producer may carry whatever
+  diagnostics it likes.
+- **Missing required columns fail loudly**, before any row is parsed:
+  `ValueError: <path> is missing columns: [...]`.
+- Every value in a required numeric column must parse — there is no
+  missing-value sentinel and no row skipping.
+
+### 1a. Dose response — `load_dose_curve`
+
+Default path `artifacts/dose_response.csv`; `steer-report --dose-csv`.
+
+One row per **(coeff, seed)** pair.
+
+| column | required | meaning |
+|---|---|---|
+| `coeff` | ✅ | injection coefficient (the dose); the x axis |
+| `seed` | ✅ | generation seed for this sample |
+| `formality` | ✅ | the effect axis — behaviour score for the concept |
+| `repetition` | ✅ | coherence axis, higher is worse |
+| `ppl` | ✅ | coherence axis (perplexity), lower is better |
+| `alpha_norm` | — | normalised dose; carried by the sweep, ignored here |
+
+The first rows of the committed `artifacts/dose_response.csv`, showing two seeds
+at the same coeff:
+
+```csv
+coeff,seed,alpha_norm,formality,repetition,ppl
+-60.0,0,-0.13114754098360656,2.0294648914518314,0.29126370083816894,10.909140628067348
+-60.0,1,-0.13114754098360656,2.848055555555556,0.10565149136577706,12.38439366631976
+```
+
+The required set is exactly `{coeff, seed, formality, repetition, ppl}` —
+`_SWEEP_COLUMNS` plus the x column.
+
+> **Known drift.** The effect column name is currently hardcoded to
+> `formality`, but the cross-model runs in `artifacts/` (sentiment, verbosity,
+> and the redosed layer sweeps) emit it as `effect`, so `steer-report` cannot
+> read them as committed. That is issue
+> [#30](https://github.com/bamdadd/steerbench/issues/30); this page will need a
+> one-line update when it lands.
+
+### 1b. Layer sweep — `load_layer_curve`
+
+Default path `artifacts/layer_sweep.csv`; `steer-report --layer-csv`.
+
+The same effect/coherence tail as the dose sweep, keyed on `layer` instead of
+`coeff` — one row per **(layer, seed)** pair, one vector injected at every
+depth.
+
+| column | required | meaning |
+|---|---|---|
+| `layer` | ✅ | injection layer index; the x axis |
+| `seed` | ✅ | generation seed for this sample |
+| `formality` | ✅ | the effect axis |
+| `repetition` | ✅ | coherence axis, higher is worse |
+| `ppl` | ✅ | coherence axis, lower is better |
+| `layer_pos` | — | ignored |
+| `dir_norm` | — | per-layer direction L2, ≈1.0 since repeng unit-normalises; ignored |
+| `resid_norm`, `coeff`, `alpha_norm` | — | ignored |
+
+The x column is a parameter (`load_layer_curve(path, x_column="layer")`), so a
+sweep keyed on something else can be read without changing the file.
+
+### 1c. Side effects — `load_side_effects`
+
+`steer-report --side-csv`; **optional**. If it is not passed, the CLI writes a
+header-only stub and the card renders an empty "Side effects" table rather than
+failing. If it *is* passed and does not exist, that is an error — a typo should
+not silently degrade to the stub.
+
+One row per benchmark slice. All three columns are required and there are no
+optional ones.
+
+| column | required | meaning |
+|---|---|---|
+| `benchmark` | ✅ | slice name, e.g. `mmlu`, `gsm8k` |
+| `unsteered_acc` | ✅ | accuracy with no steering applied |
+| `steered_acc` | ✅ | accuracy with the vector injected |
+
+```csv
+benchmark,unsteered_acc,steered_acc
+mmlu,0.62,0.55
+gsm8k,0.40,0.41
+```
+
+`SideEffect.delta` is `steered_acc - unsteered_acc`, so **negative means the
+vector degraded the capability**. The renderer colours it on that sign.
+
+### 1d. Raw-per-seed, not pre-aggregated
+
+This is the part that most often surprises a new producer:
+
+> **Emit one raw row per seed. Do not pre-aggregate.**
+
+`report.py` computes the statistics itself, in `_combine`:
+
+- the reported centre is `statistics.fmean` of the per-seed values;
+- the reported spread is the **population** standard deviation
+  (`statistics.pstdev`) across seeds, and is `0.0` for a single seed;
+- `n_seeds` counts *distinct* `seed` values in the group.
+
+Rows are grouped by the x column and the resulting points are sorted by x, so
+row order in the file does not matter and duplicate x values are expected.
+
+Handing the reader a pre-averaged file is not detected — it will silently be
+treated as a one-sample-per-x sweep and every error bar will collapse to zero.
+`CONTRIBUTING.md` asks for 3+ seeds for exactly this reason.
+
+### 1e. Coherence direction
+
+Coherence is carried as **two raw axes**, matching what
+`metrics.CoherenceScore` returns, with no sign convention imposed on the
+producer. Dump exactly what `metrics.py` gives you. The direction is owned by
+the consumer, in `report.COHERENCE_DIRECTION`:
+
+| axis | column | direction |
+|---|---|---|
+| perplexity | `ppl` | `lower_is_better` |
+| repetition | `repetition` | `higher_is_worse` |
+
+A point is **coherent** when it satisfies *both* axes — the cliff can appear in
+either one:
+
+- `analyze_dose` — perplexity within `perplexity_tol` (default `0.5`) of the
+  *baseline* point, i.e. `ppl <= baseline_ppl * 1.5`, **and**
+  `repetition <= repetition_cap` (default `0.5`). The baseline is the point
+  with `coeff` nearest zero, not the smallest coeff, so a two-sided sweep is
+  handled symmetrically.
+- `analyze_layers` — perplexity within `perplexity_tol` (default `1.0`, and
+  deliberately more generous) of the *best perplexity anywhere in the sweep*,
+  plus the same repetition cap. A degenerate trap means a catastrophic blowup,
+  not a hair above the tightest floor.
+
+## 2. The repeng boundary
+
+**steerbench consumes steering vectors. It does not reimplement extraction.**
+
+Producing a direction from contrastive prompts is
+[repeng](https://github.com/vgel/repeng)'s job. `steerbench.vectors` exists to
+read repeng's output durably and to describe it, and there is deliberately no
+training or extraction code anywhere in `src/`.
+
+What `load_vector` accepts:
+
+- **`.gguf`** — repeng's native `ControlVector.export_gguf` format: architecture
+  `controlvector`, KV fields `controlvector.model_hint` and
+  `controlvector.layer_count`, and one tensor per layer named
+  `direction.{layer}`.
+- **`.pt`** — a plain `dict[int, tensor]` mapping layer to direction, as a
+  fallback for anything that is not repeng.
+
+What `save_vector` writes is a **superset** of the native format: the same
+tensors and KV fields, plus `steerbench.concept` and
+`steerbench.repeng_version`. repeng's `import_gguf` ignores KV fields it does
+not recognise, so a file steerbench wrote stays natively loadable by repeng.
+That is the whole point of the superset — the boundary is one-way in code but
+round-trippable in data.
+
+Practically, for a contributor:
+
+- Adding a new **concept** means adding example prompts and a vector, not
+  extraction code.
+- Adding a new **model** means a new sweep producing the CSVs above.
+- If you find yourself writing PCA over hidden states, you are in the wrong
+  repo — that belongs upstream in repeng.

src/steerbench/report.py

@@ -10,9 +10,10 @@
 3. **Side effects** — steered vs unsteered accuracy on held-out benchmarks.
 4. **Layer sensitivity** — the same vector injected at every layer.
 
-CSV schema (the contract with the M0 ``m0-sweep`` worktree). Rows are RAW —
-one sample per seed, NOT pre-aggregated — and this module computes mean±std
-across seeds itself:
+CSV schema (the contract with the M0 ``m0-sweep`` worktree), summarised below;
+``docs/data-contract.md`` is the canonical write-up and covers the same ground
+for a sweep producer. Rows are RAW — one sample per seed, NOT pre-aggregated —
+and this module computes mean±std across seeds itself:
 
 * dose-response (``artifacts/dose_response.csv``) — one row per (coeff, seed)::
 

src/steerbench/vectors.py

@@ -4,6 +4,9 @@
 file) with no conversion pain, and also accepts a plain ``.pt`` mapping of
 ``layer -> tensor`` as a fallback. See ``load_vector`` for format detection.
 
+steerbench *consumes* vectors and does not reimplement extraction; that
+producer/consumer boundary is written up in ``docs/data-contract.md``.
+
 repeng stores directions as ``dict[int, np.ndarray]`` inside a GGUF file with:
 
 * architecture (``general.architecture``) == ``"controlvector"``