Notes from a read-only pass: the compile path, and one gap between declared and measured context cost

[bprzybysz]

I went through the public Rosetta repo and docs after a technical conversation with the team.
Read-only pass: no install, no runtime test. So every negative below is "I did not find it", not
"it is not there" — if any of this is already solved somewhere I did not open, I have simply missed it.

Short version: the compile path is the strongest part. The one thing I could not place: the token
budget you declare is estimated, while the token cost you measure is exact but goes away with the run.
One optional thing follows: keep that measured cost across runs. The join that makes it per-rule is
mostly already built and can surface in debug. Two questions at the end.

---

The compile path is the strongest part.

One instruction source, and Rosettify-Plugins turns it into IDE-native plugin trees instead of parallel prompt trees drifting apart per IDE. The typed aliases on top (USE SKILL, APPLY PHASE, READ RULE) mean instruction content never learns whether it is running as a plugin, over MCP, or locally. The instructions-repo / target-repo split is the same instinct applied to state: you ship behaviour, the project keeps its own files.

HITL as explicit policy with an exact opt-out is a stronger promise than "the agent will probably ask" — and durable resume reads like it was built by someone who has had a run die at 80%.

For context on where I am coming from — I keep a small Claude Code–native context harness as a side project, which is the only reason the next section jumped out at me at all.

---

What I could not place: the estimated number and the measured one never meet.

There are two halves of context accounting in the repo. Both are built. Both are good. What I could not find is the edge between them.

Half one, at authoring time — the Cognitive Budget rubric. A prompt's input + reasoning + output, plus the prompt itself, should stay under 60% of the context window. It is one of the gates the prompt-quality audit scores on pull requests — and worth being precise about how it bites: the gate score is reported, not enforced. What fails the check is the auditor tagging an issue at severity 3 or higher, which is what trips the exit 1.

This side is estimated, not measured. The auditor is a model reading a diff, with no token count in front of it, and I did not find a per-model context-window size in the tree to take a percentage of. For catching an obviously bloated prompt that is a good trade — a real gate for the price of a prompt file. It just cannot catch a slow drift, because the number it judges against was never measured.

Half two, at measurement time — the cost rollup in Curiocity. Input, output, reasoning, cache-write, cache-read and total, per model and per source. Exact, and better-shaped than most: splitting cache-write from cache-read is the difference between knowing your bill and knowing why your bill moved. The JSON reporter writes all of it to suite.json.

Then the run output directory goes away. run-<timestamp>/ is created fresh per invocation, and I did not find a step in the Curiocity workflow that uploads it. So the accurate number exists, machine-readable, exactly once. It never reaches the place the estimate is made.

flowchart LR
  A["<b>Authoring time</b><br/>Cognitive Budget rubric<br/><i>estimated: under 60%?</i>"]
  S["shipped prompt / skill"]
  B["<b>Measurement time</b><br/>cost rollup → suite.json<br/><i>exact, per model × source</i>"]
  X["run output directory discarded"]
  A -->|scored on the PR| S
  S -->|instrumented| B
  B --> X
  B -. "I could not find this edge" .-> A

Loading

---

One optional thing follows, with a small addition that makes it usable.

Part	What it gives you	What it costs
Primary — retain cost	Cost that outlives the run — a spread across many runs instead of one number per run	keeping a file that is already being written; eval and CI only
Addition — surface the join	Attributable loads — which rule was actually in context, and what it cost	one debug flag, default off

This is not a gate. budgetUsd already does the right thing at suite level, and your P7, "cost: track + warn, never abort", is the right policy — I would not want a token count failing builds either.

The primary offer asks for nothing at runtime — no flag on the serving path, no bytes in anyone's
context window; it is retention of a file the eval harness already writes. The addition is a debug mode,
default off. A stock install behaves exactly as it does today under both.

---

Cost that outlives the run

The token cost numbers do not outlive the run, so nobody can look at twenty runs of one rule and see the spread.

Why retain those records. One run gives a rule a single reading. Keeping the per-run cost records lets an
audit compute a mean and a standard deviation, then read the tail, across many runs of one rule. That is what
turns the Cognitive Budget check from a judgement into a measurement, and it answers three questions the audit
cannot ask today:

• Does this rule actually fit? Compare its mean input + reasoning + output with 60% of the window, then
  inspect the high-percentile tail — or a tail expressed as mean plus k standard deviations — rather than asking
  whether it merely looks bloated in a diff.
• If it does not fit, by how much, and since when? Standard deviations say how far an overshoot sits from a
  rule's usual cost; a mean that moves says when the rule's normal cost changed. That sizes the problem and points
  at the change that caused it.
• Which shape of problem is it? A rule that spikes on some inputs is a different problem from one that sits
  high on every input, and the two want different fixes. Today they look identical to the auditor, because both
  arrive as a model's estimate on a diff.

The maths for that is already in the tree, one subsystem over. stats/ already has mean and stddev,
and they already compute exactly what those three questions need: mean gives a rule's usual cost, stddev
gives how far it moves around that, and together they give you the tail. They take a plain list of numbers, so a
list of token costs is as valid an input as anything else — no new maths has to be written. What they are pointed
at today is verdict scores — how well a run did, not what it cost — and only across the repeats inside a
single curiocity run, so the spread they measure is a spread within one run rather than across many.

So the missing piece is not the computation. It is that the numbers those functions would read are thrown away at
the end of the run: suite.json already holds the cost blocks in machine-readable form, and the directory it
sits in is created fresh each invocation. Keep that file, point mean and stddev at the cost blocks instead of
the scores, and the 60% stops being a number someone picked and becomes one the data chose.

This is not hypothetical for you, and it is not a runtime question either. The repo already treats the
token cost of an authored instruction as something worth acting on — there is a skill whose whole job is
compressing an instruction before it ships, on the stated grounds that every token it carries becomes
context in someone else's session.

What is missing underneath that is the measurement. The size a rule
gets held to is a number a person chose, so "small enough" stays a judgement call. With the cost blocks
kept across runs it becomes a distribution instead: here is what this rule normally costs, and here is
the run that sat three standard deviations out. That is something you would look at while developing a
rule.

The obstacle, honestly: I did not find a per-rule id on a Curiocity cost record. The finest grain I saw there is (source, model) inside a (case, agent) group, and a case names an eval scenario, not the rule the rubric scores.

The identity itself is not missing from the repo — the publishing side already derives a release-stripped resource path, a content hash and a deterministic document id per rule. What I did not find is a path by which any of that reaches a cost record. So the gap looks like wiring between two subsystems rather than a new id space.

---

What makes that usable — attributable loads

Why this addition is smaller than it first looks. Closing the obstacle above sounds like inventing provenance — a new id space, a new accounting layer, something that has to be designed before it can ship. What I found is narrower.

The publishing side already derives a stable identity per rule. Bundling already keeps merged documents distinguishable. What I did not find is a path that carries that identity onto the load (and from there onto a cost record).

So the ask is not a new subsystem. It is projecting metadata you already compute into a wrapper you already emit, optionally, in a debug mode. That is why this sits as an addition to retaining cost, and not as a second proposal.

On the wrapper itself: each source document already gets its own element carrying an id, a path and a name. What I did not find on that element is the rest of the identity the publishing side already computes for the same document — the content hash, the original path, the deterministic document id. Those exist in the stored metadata; I just did not find them projected into the bundle.

The immediate payoff is debugging. "Which version of that skill was actually in context?" is the question I can never answer from a transcript three sessions later, and it is usually the only one that matters.

A content hash on a bundled document is the join key — once a load is attributable, its measured token cost is attributable too, and declared-versus-measured becomes a join on data you already collect rather than a new accounting layer. A typed alias is already the moment a choice gets made. So the same tag is the natural place to record which path was taken, not only which document arrived.

The cost is real and worth measuring rather than assuming: tokens, plus the risk the wrapper becomes noise the model narrates back. I would run off versus debug and keep it only if the token delta is tolerable and one real "which instruction caused this?" debug gets faster. Default off either way.

---

Two questions:

1. Is cost data being per-run-only a deliberate call — the suite warning is enough and history is out of scope — or is keeping it on a list somewhere?
2. The shipped plugins default to --deterministic-hooks false, so a stock install gets SessionStart bootstrap and not the advisory hooks. Is that a deliberate "let people opt into the guardrails" stance, or mostly a first-run footprint decision? I went the other way — hooks on, always — and I am genuinely not sure I am right.

Thanks for publishing architecture that is actually readable. That is rarer than it should be.

— Blaise

Sources (public):
https://github.com/griddynamics/rosetta · https://griddynamics.github.io/rosetta/ · ARCHITECTURE.md · USAGE_GUIDE.md

Method: read-only review of the public tree and docs. No Rosetta install; no functionality or runtime testing.

[isolomatov-gd]

@bprzybysz Thank you for the read.

There is too much text restating back existing documentation.

Confirming what you are asking:

1. Modify .gitignore to keep the cost numbers in curiocity-results/run-*/suite.json, so those can be compared across runs. What would change once the spread is visible?

2. We recommend developing with local plugins, not MCP — DEVELOPER_GUIDE.md, then TESTING-PLUGINS.md. We can also add an option to disable MCP caching. What problem are you solving with the content hash?

3. Deterministic hooks are disabled deliberately. Advisory hooks run before and after every tool call, so they are opt-in and will be enabled in the future one-by-one.

4. Anything else I missed?