Judged eval: does the dialectical framework add value over a bare persona-prompted model?

[etaroza]

Summary

Build a judged evaluation that answers one question: does the dialectical framework (as embodied by the Advisor agent) produce better counsel than a strong base model given the same persona prompt and the same base model? This is a deliberate, cost-bearing eval — not a CI test — and it is a project in its own right. Filing now as backlog; we'll eventually need it to defend the framework's value (internally and externally).

The question, stated honestly

"Does the framework add anything?" reduces to: is the output better than the alternative? That is inseparable from judging quality against the alternative. There is no cheap proxy for it.

The framework's process virtues — persistent, systematic, transparent — are properties, not quality guarantees. A wrong T/A decomposition that is persistent is durably wrong; that is transparent is visibly wrong; that is systematic is consistently wrong. So verifying those properties (graph got built, graph got reused, personas share structure) confirms the machinery ran as designed — it does not demonstrate value. Those checks are integration tests, useful only for falsification, never for demonstration.

Therefore the real deliverable is the judged comparison below.

The eval

Comparison (ablation ladder), base model held constant within each row:

Arm	Persona preamble	Engine prompt	Graph as context	Live tools + persistence
A0 — bare persona call	✅	❌	❌	❌
A1 — persona + engine prompt, no tools	✅	✅	❌	❌
A1.5 — engine prompt + pre-built graph dumped as text	✅	✅	✅	❌
A2 — full Advisor	✅	✅	✅	✅

• A0→A1 isolates the value of the reasoning instructions (pure prompt engineering).
• A1→A1.5 isolates the value of the structured artifact as context — does a decomposed, quality-gated, ranked tetrad/wheel help even when it's just handed to the model as text, with no building or memory? (Build the graph offline via the real Advisor, dialectical_context-dump it, paste into A1's prompt.)
• A1.5→A2 isolates the value of the live process: on-the-fly decomposition, quality gates firing mid-conversation, and — the part context windows structurally can't fake — cross-turn/cross-session persistence and curation (discard/re-anchor/re-explore against an addressable artifact).
• Cross the base model across rows (e.g. Opus, Fable) to separate the framework effect from the model effect. Do not confound them by only running one model.

Why A1.5 matters: without it, an A1→A2 win is ambiguous — it could be "structure helps" (which a smart model could self-generate given a better prompt, a depreciating asset) or "the machinery does something the prompt can't" (persistence, search, verification — a durable asset). A1.5 is the cut between them. Skip it and you can't tell which asset you're defending.

Confound to state plainly: A2's final counsel is the same base model reading its own graph. So A1.5/A2 measure "does graph-as-context (static / live) beat free in-context reasoning," not "can the model reason at all." Keep the response-generation step identical across A1.5/A2 (same persona, same decode settings) so the only variable is where the structure came from.

Prediction to test: A0→A1 closes most of the perceived gap on a strong model; A1→A1.5 shows a modest gain (structure-as-context helps but a strong model half-generates it anyway); the framework's real, durable payoff is A1.5→A2 on cross-turn/cross-session coherence and curability — not on single-turn "the model can't reason dialectically." If we only compare A0 vs A2 we'll wrongly credit the machinery for gains that were really just the prompt.

Rubric (score per-dimension, blind, paired)

A naive "which response is more helpful?" judge favors the eloquent base model (warmth/fluency/empathy). Score the structural dimensions tied to the framework's actual claims instead:

• Blindspot specificity — names a genuine A+ (what the opposing force offers, invisible to the holder), vs generic "consider the other side."
• Entanglement — explains why championing T+ structurally creates A-, vs listing pros/cons.
• Paired recipe — a concrete do-and-reflect (Ac+/Re+) move, vs "find balance."
• Tension coverage — does the response surface the non-obvious tensions a systematic search finds, or just the first plausible one? This is the framework's strongest structural claim (BuildWheels ranks every valid arrangement; a base model satisfices on the first framing). Score against the ranked set the framework actually enumerated for that scenario — a base model that misses the 2nd/3rd tension the search ranked highly scores low here.
• Cross-turn coherence — turn N builds on the tension mapped in turn 1, vs drift. (This is where persistence should visibly win; a single-turn eval is blind to it.)
• Non-triviality — would this surprise the person, or is it what any thoughtful friend says?

Non-inferiority guard (regression dimension): also score warmth / actionability / conversational fit — the base model's home turf. The framework arm must not lose here to count as a win overall. A structurally superior answer that reads as colder or more clinical is a net loss for a counseling product. Report this as a non-inferiority bound, not a win-rate: A2 may tie on warmth and win on structure, but if it loses on warmth the trade-off is a product decision, not a slam dunk.

Judge design (the hard part — where these projects die)

• Blind, paired A/B; per-dimension scores; explicit instruction to ignore length and eloquence (framework arm is longer/more structured; judges are verbosity-biased).
• LLM judge as first pass, but human calibration on a sample is required — LLM judges have self-preference bias, and scoring fidelity ("is this a true reading of the situation?") tempts the judge to rubber-stamp any plausible structure.
• Multiple seeds per scenario (LLM non-determinism will swamp small effects).
• Report per-dimension deltas with the base model as a reported variable, not just an aggregate "win rate."

Scenario set

• Span first-person ("I'm in the situation") and third-party ("I'm mapping someone else's situation") — the Advisor adapts to these differently.
• Include poor-fit controls (pure info requests, sequential procedures, single-right-answer calculations). The framework should show no gain there. If it "wins" on those too, the judge is rewarding structure, not counsel — that invalidates the rubric.
• Conversations must be long/rich enough for the machinery to actually engage (see prerequisite).

Prerequisite that can silently invalidate A2

The Advisor only builds a graph if the model calls ingest/anchor/explore. In a short conversation it may never call them — in which case A2 collapses to A1 and we'd be measuring the prompt, not the framework, without noticing. Instrument tool calls (effect log / advisor.messages) and assert a graph exists before trusting any A2 result. This likely forces multi-turn scenarios rich enough for explore/synthesis to fire.

Cheap falsification pre-filter (do this first, it is not the deliverable)

Before building the judged harness, read ~5 graphs by eye. Run the Advisor over a handful of scenarios, dump the resulting graphs to readable text, and look. This is a real (manual, un-scaled) quality judgment and the highest information-per-hour step:

• If 5 hand-read graphs show nothing a bare model wouldn't produce → value is disproven cheaply; stop.
• If they look valuable → we've lost nothing and learned which dimension has signal, which scopes the judged eval.

The one non-tautological cheap probe worth keeping: inject a wrong framing → can we locate and correct it in the graph? That's a capability the bare model categorically cannot do (not a spec-check). But it demonstrates a capability, not that the capability is wanted.

Deliverables

• Falsification pre-filter script (tests/, --real-llm): run Advisor over N scenarios, dump graphs to readable text for eye review.
• Scenario set (first-person, third-party, poor-fit controls) with the multi-turn richness the machinery needs.
• Tool-call / graph-existence instrumentation to guarantee A2 ≠ A1 (assert ingest/anchor/explore fired and a committed graph exists).
• A0/A1/A1.5/A2 × base-model harness (tests/, --real-llm, reuses conftest DI wiring). A1.5 needs an offline graph-build + dialectical_context-dump step feeding A1's prompt.
• Blind paired judge on the per-dimension rubric (incl. tension-coverage and the warmth/actionability non-inferiority guard), length/eloquence-neutralized.
• Human-calibration pass on a sample; report self-preference/verbosity bias checks.
• Results write-up: per-dimension deltas by arm (esp. A1→A1.5→A2 to attribute gains to structure vs process) and by base model; non-inferiority result on warmth; poor-fit control results.

Notes / open questions

• Test vs eval: this lives under tests/ for conftest/DI wiring but is a deliberate --real-llm script, never a CI gate.
• Two claims, two tracks — decide before scoping the full build:
  1. "Better responses" → the rubric eval above. The A1.5 arm is what keeps this honest: it separates "structure helps as context" (a strong model can partly self-generate this — depreciating as models improve) from "the live machinery helps" (durable).
  2. "Durable reasoning substrate" (inspectable, reproducible, reusable across sessions/personas/UIs) → a single-response quality comparison structurally undersells it. This track measures cross-session consistency (same input → same framing across runs, where a stateless model drifts), auditability (can we trace why a framing won?), and curability (inject-wrong-framing → locate-and-correct, which the bare model categorically cannot do). As models get smarter this is the track that strengthens — the model fills the substrate with better content; the substrate makes that content addressable, searchable, and correctable. Track 1 alone risks conceding the framework's most durable value.
• Framing for the write-up: position the framework as memory + systematic search + verification + audit infrastructure for a strong model — analogous to a spreadsheet for an accountant or a proof assistant for a mathematician — not as a reasoning crutch for a weak one. Track 2 is where that story lives; scope it deliberately rather than letting Track 1's single-response comparison define "value."
• Cost: real-LLM, multi-seed, multi-arm (now 4), multi-model → non-trivial spend per run. Budget deliberately; the A1.5 arm adds an offline graph-build cost per scenario.

---

Context: distilled from a design discussion. Filed as backlog — "eventually we'll have to do something like this."