A test needs two things: something to try, and a way to tell whether it worked.
In an ordinary example-based test, we supply both: these inputs should produce this result. That works well for individual cases. But as operations interact—insert, update, delete, retry, reconnect—the combinations grow beyond the examples we can reasonably write.
What if we could generate those combinations and still know the right answer?
That is the practical value of oracle-based testing. An oracle supplies the judgment independently of how we choose the test cases. It might be another implementation whose behavior should match, an executable specification, a simple reference that recomputes the answer, or a rule relating several executions.
For an incremental query engine, production might maintain complex indexes while the reference simply filters and sorts an array. Or we might compare compatible queries against another database. Either way, we can explore many histories without hand-calculating every expected result.
The payoff is broader bug detection and a stable check during refactoring. The challenge is making sure the reference, generated cases, and observations actually represent the behavior we promise.
Start with one small oracle. Follow the later sections when your contract needs richer state, controlled timing, or more observations. The review card is a short way to apply the guide to an existing test. Historical cases and research are collected in the companion notes. Use the project glossary for terms shared with production code.
For help discovering or challenging the law itself, use the repository's oracle-authoring or oracle-review skill and its design instruments. These aids support inquiry; the numbered requirements below continue to define guide conformance.
The numbered requirements in this section are the complete checklist for an
oracle-tests.md conformance audit. Repository policies such as AGENTS.md and
the coverage map apply separately. Later sections explain the requirements and
give techniques and examples. They do not create additional requirements unless
they cite one of these IDs.
MUST, SHOULD, and MAY have their usual specification meanings:
MUSTis required when the stated trigger applies.SHOULDis expected when the trigger applies. A review may accept an explicit, technically grounded reason for doing something else.MAYis optional.
Only these uppercase keywords inside numbered requirement blocks define guide
conformance. Later prose may use lowercase contract terms such as must and
may, but it does not create a new guide requirement unless it cites an ORC
ID. Words such as “useful,” “ask,” and “consider” remain explanatory.
An important generated property is a generated property used to protect a reusable product law, claim coverage beyond named examples, or demonstrate a bug-class repair. A fixed regression, bounded enumeration, or diagnostic stress probe is not an important generated property merely because it uses test data.
The executable oracle MUST keep its contract, model, history grammar, production driver, and refinement check visible in the oracle file or in companion modules that the file names directly. Pull-request evidence and a versioned review record tied to the exact reviewed head may supplement those five responsibilities, but cannot replace them. Unless a requirement says otherwise, other conformance evidence MAY live in the executable test, an oracle-file comment, the pull-request description, or a versioned review record linked from the change. Reviewers MUST score the stated obligation, not the presence of a preferred heading, class, comment template, or helper.
- Trigger: A test computes or constrains an expected product result.
- Obligation: The oracle MUST identify the promised law, its authority, and the limits of the claimed result.
- Acceptance evidence: A reviewer can trace the expected result to an approved API, architecture document, established contract, or design decision, and can identify what the oracle does not establish.
- Not required: One particular comment format or a complete list of every behavior outside the subsystem.
- Trigger: An oracle compares production with an expected result.
- Obligation: The expected result MUST be derived independently of the production semantic machinery whose behavior it judges.
- Acceptance evidence: The model, alternate implementation, or relation does not import or reproduce the relevant production classifier, transition, comparator, or state machine without a separately justified trusted base.
- Not required: Independence of the production driver. Exercising the real production entry point is expected and does not violate this requirement.
- Trigger: Any oracle or generated-history test.
- Obligation: The oracle MUST combine explanatory prose and executable code as a literate program. Opening prose MUST state the promised law and its limits before test mechanics. Prose beside the model MUST explain why its independent rule predicts the promised result. The oracle MUST explain its legal histories, production path, public observations, and comparison checkpoint beside the relevant code. These explanations and the five responsibilities MUST remain visible in the executable oracle or directly named companion modules. The contract, model, history grammar, production driver, and refinement check MUST remain distinguishable. Changes to the executable law MUST update its prose in the same change.
- Acceptance evidence: A reviewer can learn the subsystem contract by reading the prose and code together. The explanations connect the law to the model and the model to observations at the named checkpoint. A compact oracle may use one opening comment and short explanations beside its code.
- Does not satisfy: Code alone, five headings without explanations, or explanations available only in a pull-request description or separate review record.
- Not required: Five headings, five classes, five files, one fixture per law, a long essay, or a review card embedded in every oracle.
- Trigger: A generated property claims coverage of a legal history or input grammar.
- Obligation: The change MUST account for reconstruction, ablation, range, and exclusion for that grammar.
- Acceptance evidence: The evidence names every known valid witness within the claimed scope and shows that each can be reconstructed; accounts for the semantic contribution of each declared axis, rule, or overlap under ablation; states the bounded domains and marginal cases; and names a nearby invalid history or state the grammar rejects.
- Evidence location: These controls MAY be executable calibration checks, comments, pull-request evidence, or a versioned review record tied to the exact reviewed head. Stable executable checks are preferred when the claim is cheap to preserve.
- Not required: One test per control, a Cartesian product of unrelated axes, or a permanent source mutant.
- Trigger: Any oracle used as product evidence.
- Obligation: The test MUST exercise the named production entry point and MUST compare the promised public observation at the named checkpoint.
- Acceptance evidence: A positive execution witness shows that the intended path and comparison ran. The recorder can represent the violations relevant to the law, including duplicates, omissions, order, or partial output when those are contractual.
- Not required: Observation of internal facts that the contract does not expose.
- Trigger: An important generated property or a claimed oracle repair.
- Obligation: The evidence MUST name at least one plausible wrong answer or design and demonstrate that the relevant comparison rejects it.
- Acceptance evidence: A wrong-result control, hostile fixture, fault injection, or production mutant reaches the intended checkpoint and preserves the distinguishing failure.
- Not required: Modifying production, checking in a mutant, or running a production mutation campaign for every oracle.
- Conditional obligation: When a mutant is run, the evidence MUST classify its outcome as assertion failure, timeout, setup failure, runtime failure after setup outside the intended comparison, an unreached path, survival, or equivalence within the tested domain.
- Trigger: An important generated property.
- Obligation: The package's oracle campaign MUST execute the same property,
generators, observation recorder, refinement check, and run budget twice:
once with a documented fixed seed and once without a seed. A replay interface
MUST accept both the seed and shrink path and MUST run the requested replay
directly. When the property uses
fc.commands, the interface MUST also accept the commands arbitrary's reportedreplayPathand pass it back tofc.commands. - Acceptance evidence: The normal test command reaches both campaigns, each
campaign records its execution, and a checked replay command reproduces a
captured seed-and-path failure, including the command replay path when the
property uses
fc.commands. - Does not satisfy: Fixed examples do not replace the fixed-seed campaign. Different generators, observation recorders, refinement checks, or budgets do not establish campaign parity.
- Not required: Running either normal campaign before an explicit replay.
- Trigger: A change introduces, removes, combines, or splits state in a stateful reference model.
- Obligation: The change MUST explain whether a legal next action can distinguish states the model proposes to treat as equal.
- Acceptance evidence: A distinguishing history justifies retained state, or a reasoned argument shows why removed state cannot change a promised observation or action's legality.
- Not required: An executable mutant for every model field or application of this requirement to a stateless recomputation.
- Trigger: A model combines or splits production concepts, or introduces a model-only term that could be mistaken for a production concept.
- Obligation: The oracle MUST map the differing concepts explicitly and MUST use the project glossary's term for a shared concept.
- Acceptance evidence: A reviewer can translate model actions, states, and checkpoints to production boundaries without guessing.
- Not required: A mapping entry for plainly local data holders whose meaning and observation are already unambiguous.
- Trigger: Shrinking, capture, or cleanup can replace or erase an oracle failure.
- Obligation: The harness MUST preserve the original violated law and checkpoint, retain distinguishable secondary cleanup diagnostics, and release its resources.
- Acceptance evidence: The failure report separates the primary mismatch from each cleanup failure and says whether a reduction reproduced the same violation.
- Not required: Throwing cleanup failures as separate top-level exceptions.
An
AggregateErrorsatisfies this requirement when itscauseanderrorspreserve those distinctions.
- Trigger: A reviewer names a plausible semantic fault that production and the primary model could share, and a meaningfully different formulation could distinguish it.
- Obligation: The change SHOULD add that formulation. If it does not, the evidence MUST state why it is impractical or out of scope and track the remaining risk.
- Acceptance evidence: The alternate path states its equivalence or metamorphic relation, including relevant differences in ordering, projection, duplicates, and empty results.
- Not required: A second formulation without a named shared-fault hypothesis.
- Does not satisfy: Agreement between two structurally identical paths does not establish independent evidence.
- Trigger: A change claims that an oracle is new, repaired, or comprehensively audited against this guide.
- Obligation: The review evidence MUST record the outcome of every other applicable numbered requirement and MUST identify the requirements that do not apply.
- Acceptance evidence: Each applicable requirement has concrete evidence or an explicit unresolved gap. Each non-applicable requirement has a reason tied to its trigger.
- Conditional bug-class closure: When a change claims to eliminate a bug class, the review evidence MUST name its contract × history × production-path × observation boundary, identify the original and adjacent distinguishing witnesses, show a relevant wrong design rejected at the intended checkpoint, and list unresolved in-scope cells with their coverage-map owner. A known reachable counterexample prevents a closure claim. One oracle need not own every boundary.
- Evidence location: At closeout, verdict-critical evidence outside the executable oracle MUST live in a versioned review record tied to the exact reviewed head. An editable pull-request description alone does not satisfy this requirement.
- Not required: Copying the layer-comparison questions or review card into every test, answering them in their printed order, or placing the audit in the oracle file.
- Trigger: A fixed or generated oracle is cited as protecting a reusable conditional, threshold, or range law beyond its named examples.
- Obligation: The evidence MUST show a legal witness that reaches the law's premise and a nearby in-domain witness whose observation at the claimed checkpoint distinguishes the stated law from a plausible wrong boundary or consequence. If those witnesses are unavailable, the claim MUST be limited to the tested cases, and the reusable-law gap and its owner MUST be recorded.
- Acceptance evidence: The test or review record names the wrong rule and shows which existing case rejects it. For a numeric threshold, cases on opposite sides are insufficient if a plausible misplaced threshold still produces every expected result in the tested matrix.
- Not required: Random generation, a permanent mutant, or exhaustive tests of every adjacent value for a deliberately bounded example.
- Trigger: A controlled provider or host supplies a premise material to an oracle result, and that result is cited as evidence for behavior with a real provider or host.
- Obligation: The evidence MUST identify that premise and either point to a receiving witness in which the real provider or host supplies the same premise, or mark the cross-boundary claim unresolved with an owner.
- Acceptance evidence: A reviewer can trace the specific event shape, ordering, error, or lifecycle condition from the controlled fixture to the receiving witness. A real-provider test of an unrelated ordinary case does not establish the handoff.
- Not required: An end-to-end run inside every component oracle. An oracle whose claim is explicitly limited to the controlled boundary does not need a real-provider witness to satisfy this requirement.
One way to supply that judgment is a small reference model. For a simple stateful test, choose a legal action, apply it to production and the reference, and compare the promised result at the agreed point. “After the action returns” may be the right point for one API; a callback during that action may matter for another.
For example, suppose a query promises IDs in rank order. We expect [1, 2]; production returns [2, 1]. A length check passes. A membership check passes. An ordered comparison fails. We exercised the query in all three tests, but only one observed the difference we cared about.
An oracle is the rule that judges that behavior. The generator chooses what to try. Keep those jobs separate: more executions cannot repair a comparison that ignores the promised order.
Before writing the loop, answer six questions:
| Question | What to write down |
|---|---|
| What is promised? | The rule, its source and its limits. |
| What can happen? | Values, actions and relevant action orders. |
| How will we know the answer? | A simpler reference, another implementation, or a relation that must hold. |
| What are we actually exercising? | The production entry point and what the fixture supplies. |
| What will we compare, and when? | Rows, order, events or other promised behavior, at a named checkpoint. |
| Why should we trust this check? | Evidence the path ran, the comparison rejects the relevant wrong answer, and failures can be replayed. |
These are responsibilities, not six required classes. A small property may answer them in a comment and twenty lines of code. A lifecycle suite may need a separate model and driver. Do not build a framework just to fill the table.
An evidence card is an optional way to collect this information; it is not a required file format:
Law and source:
Domain and legal histories:
Reference or relation:
Production path and checkpoint:
Observed result and known omissions:
How to prove it ran and rejects a wrong answer; how to rerun:
An intentionally partial oracle is still useful. A rule that rejects duplicate completion cannot prove the rows are correct, but it can protect a real promise. A fixed regression can preserve a valuable history. Prefer an oracle that generalizes a bug's missing distinction where practical; keep the fixed witness when it adds clarity or reach. Randomness is not what makes either test trustworthy.
Some concurrent contracts allow several results. In those cases, the reference must allow that freedom rather than invent one required order. We will return to that after establishing how to record an execution.
ORC-003 requires literate programming for every oracle and generated-history test. Prose and executable code together teach humans and agents the subsystem contract. State what the system promises and explain why each observable result follows from the model. The guidance below explains this requirement.
This does not make the model the source of product policy. Derive the contract from an approved API, architecture document, established behavior, or design decision. The file then keeps that contract, its model, and its production evidence together.
Two small examples show the form:
load-subset-transaction-refinement-oracle.test.tsexplains when an abort can still cancel an on-demand load.fifo-retry-oracle.property.test.tsexplains why a ready transaction waits behind a delayed FIFO head.
Use oracle in filenames that own an independent expected-result computation,
state model, differential or metamorphic comparison, or reusable law checker.
A separate module that defines that model, checker, or its history grammar also
qualifies. Identify the mechanism in the code before applying the name. A mixed
test file can qualify through a specific oracle inside it; its other tests do
not thereby become separate oracles.
An ordinary example or regression does not need this suffix merely because it asserts a contract. The same applies to concrete type assertions, fixtures, registration wrappers, and generic runner utilities. A driver that only invokes production or records observations remains support. Coverage-map inclusion, parameterization, and contract prose alone do not establish oracle ownership.
Count owners, model/checker definitions, drivers, and other support separately.
Several files may execute one shared oracle, and one file may contain several
oracle mechanisms. A filename is a discovery aid, not evidence of guide
conformance. Preserve runner suffixes such as .property.test.ts, .test-d.ts,
and .spec.ts. Update imports, test commands, replay selectors, and current
coverage links whenever an oracle file moves.
ORC-003 requires these responsibilities to remain distinguishable even when they share one file:
| Layer | What it must answer |
|---|---|
| Contract | What does the subsystem promise, and where is the boundary? |
| Model | What is the smallest independent rule that predicts public results? |
| History grammar | Which values, actions, relationships, and schedules can occur? |
| Production driver | Which real entry point and event boundary does the test exercise? |
| Refinement check | Which public observations must agree with the model, and when? |
Do not create five classes merely to match this table. A short file can use one opening comment, one pure model function, a generated input, a production fixture, and assertions. A large state machine can split these layers into separate modules when that makes each layer easier to review.
Under ORC-003, state the contract and its limits before introducing test mechanics. Start with a question or a direct statement of the problem. Explain why a normal example can miss the failure.
Put the central law beside the model too. The opening comment supplies context. The local comment lets a reader check the model without searching the file. These comments are not duplicates when they serve those separate jobs.
For example:
// Publication is the boundary. An abort before publication rejects the load
// and discards the row. An abort after publication starts resolves the load
// and keeps the row visible.
function expectedOutcome(phase: AbortPhase): ExpectedOutcome {
// ...
}Prefer a pure function or a small state transition. Keep production queues, caches, classifiers, and helpers out of the expected result. A reader should be able to challenge the model without first learning the implementation.
Name partial models honestly. If a model predicts rows but not callback counts, say so. If several outcomes are legal, return the permitted set or relation. Do not hide policy in scattered assertions outside the model.
Check every modeled public fact at each relevant boundary. During the FIFO rewrite, prose exposed that the first draft modeled calls but checked outbox ownership only at the end. Moving both observations into one state model made the test and the documentation agree.
A reference model can become as hard to trust as production. Do not keep adding state until it becomes a second implementation of the subsystem.
Treat the oracle as a small graph when the contract has independent laws:
- A node owns one coherent rule and the least state needed to predict it.
- An edge records a real dependency between two rules.
- The production driver can feed the same action to several nodes.
- The refinement check composes their observations at the named checkpoint.
For example, demand generations, facade identity, and callback coherence can use separate models. They share actions, but they do not need one controller that reproduces every production queue and cache. A failure then names the law that diverged. A reviewer can also inspect each rule without learning the rest of the subsystem.
Do not split state that a legal next action can observe only as a whole. Use the distinguishing-history test: if two states look equal to the proposed nodes, can one legal next action produce different promised results? If yes, add the missing edge or keep that state in one model.
Decomposition does not mean one model per assertion. Group facts that form one state machine. Keep independent policies separate. When an integrated promise spans several nodes, add a relational check across their outputs instead of merging their internal machinery.
This graph is a reasoning tool, not a required test framework. Plain functions and Maps are often enough.
Before choosing arbitraries, describe the language of legal cases. A useful grammar separates three kinds of fact:
- Dynamics: actions and transformations that move the system.
- Constraints: invariants, interfaces, and forbidden combinations.
- Boundary conditions: starting state, scale, provider behavior, and value domains that limit where the rules apply.
Freeze the contract first. Mark which rules come from an API or architecture document and which rules the test author inferred. Then map containment, overlap, and dependency. Do not force overlapping concerns into a tree merely to make the generator neat.
ORC-004 requires four controls before a generated property claims grammar coverage:
- Reconstruction: Can the grammar rebuild every known valid witness?
- Ablation: Does removing each axis, rule, or overlap lose a promised case or admit a forbidden one?
- Range: Does an independent marginal case require only new parameter values, or does it expose a missing rule?
- Exclusion: Can the grammar reject a nearby invalid history or state?
Generate adjacent valid forms from this grammar. Do not take a flat Cartesian product of unrelated axes merely because the tool makes that easy. When the contract does require a product, add a calibration assertion that proves every declared cell appears once.
This approach exposes two common false greens. A grammar that cannot reconstruct a known bug omits a path. A grammar that generates valid and invalid histories without distinction makes skips and classifiers carry hidden policy.
Write prose that another agent can parse without asking what a term means:
- Use active voice and short sentences.
- Give one state or event one name. Do not rotate synonyms.
- Define necessary domain terms when they first appear.
- Keep lowercase
must,may, anddoes notexact when they describe a product contract. Only the uppercase keywords in the numbered ORC blocks define guide conformance. - Use a list for three or more phases, actions, or conditions.
- Avoid metaphors when a boundary or transition has a precise name.
- Explain laws, causes, omissions, and checkpoints. Do not narrate clear code.
Aim for no more than 25 words in a descriptive sentence. Keep a longer sentence when splitting it would lose a condition or change its force. Clarity is the goal. A low word count is not.
The model and production code must use the same term for the same concept. Read the project glossary and the subsystem architecture before naming model states, actions, or observations.
Shared vocabulary does not weaken model independence. Reuse a production term, not its queue, cache, transition helper, or semantic implementation. If the model deliberately combines or splits production concepts, declare that mapping beside the model. Never give a model-only convenience the name of a stronger production concept.
Check the causal grammar as well as the nouns. Demand starts an acquisition attempt. Adapter acceptance establishes a physical acquisition and its lease. Committing source writes produces an applied receipt. Publication exposes one coherent public snapshot. Adapter acceptance, promise settlement, applied settlement, and publication are different boundaries even when one synchronous execution crosses all four.
Avoid unqualified words that hide those distinctions. Name a sync run,
public snapshot, logical subset owner, or window-operation generation
rather than a generic session, state, owner, or generation. When production
renames a concept, update its glossary entry and the models that represent it in
the same change.
Treat the layers as separate claims. Compare each pair during review:
- Does every promise in the prose have a model rule?
- Does the model retain every distinction that a legal next action can expose?
- Can the history grammar reach each modeled transition?
- Does the driver exercise the named production path and event boundary?
- Does each modeled output reach an assertion at the promised checkpoint?
- Can the recorder represent duplicate, missing, reordered, or partial output?
- Does any assertion impose behavior that the contract and model do not state?
- Do prose, model, driver, and production use the glossary's canonical term and transition grammar for each shared concept?
- Does every model-only term declare how it maps to production, or that it has no production counterpart?
This comparison is a useful audit instrument. ORC-012 requires outcomes for the applicable numbered requirements, not these questions or their answers in the test file. If the prose cannot explain an assertion through the model, the model may be incomplete. If the model predicts a fact that the test never observes, the test may be false green. If the driver cannot create a named phase, the prose claims more reach than the test has.
ORC-003 requires explanatory prose and five visible responsibilities in every oracle and generated-history test. A focused regression that is not an oracle can remain short when its name and setup already state the whole contract, but it does not replace applicable oracle coverage.
Mandatory structure does not mean five classes or a long essay. A compact oracle can state its contract and limits in one opening comment, keep a pure model beside it, define a small grammar, drive the production entry point, and compare the promised observations at a named checkpoint.
As a starting budget, add only prose that helps a reader answer one of the five layer questions. After the first draft, remove comments that only translate the next line of code into English. Run the test after the rewrite. A documentation edit that weakens the executable law changes behavior.
We'll build a small reference model as one example of the approach. The following is an illustrative contract, not a specification for every TanStack query:
- Rows have unique integer
idvalues and finite numericrankvalues. - Rows sort by ascending rank, then ascending ID.
- A nonnegative integer width chooses the first rows; width zero returns none.
- Each action inserts an absent ID, changes the rank of a present ID, or deletes a present ID.
- After an action has been applied, the ordered visible IDs must match a full recomputation.
Here is the completed card for this example:
Law: sort by rank then ID; return the first width IDs after each action.
Source: the illustrative contract above.
Domain: finite numeric ranks, unique integer IDs, legal insert/update/delete.
Reference: independent Map, fully sorted after every action.
Production path: the ordered-query entry point, after each action is applied.
Observe: the complete ordered IDs; not row values or callback counts.
Prove it ran: record applied actions and executed comparisons.
Challenge: the comparison must reject [2, 1] when [1, 2] is required.
Replay: retain the action list and width, plus the harness's replay inputs.
The production path still needs its concrete API name when this card becomes a real integration test. The example does not establish row-value, notification, locale or asynchronous-loading behavior.
Our expected answer is an ordered list of IDs. We may ignore internal tree nodes. We may not sort the actual IDs before comparison: doing so would erase the ordering bug.
The reference can store all rows in a Map and sort from scratch. It does not need production's tree, incremental caches or rank maintenance:
function expectedIds(rows, width) {
return [...rows.values()]
.sort((a, b) => a.rank - b.rank || a.id - b.id)
.slice(0, width)
.map((row) => row.id);
}Keep the reference's records separate from production's records. If production mutates a shared object and thereby changes the expected answer too, the two sides can agree for the wrong reason. Here, copying the two numeric fields is enough. Richer values need a copying or identity policy suited to their contract—not an automatic appeal to JSON serialization.
Independent code also needs independent reasoning. Calling production's comparator from this reference would make comparator defects invisible to this particular comparison. Reuse mundane test mechanics when useful, but name any semantic helper both sides trust. A small model is easier to inspect; it is not correct merely because it is small.
Take two rows: (id: 1, rank: 2) and (id: 2, rank: 1), with width two. The expected IDs are [2, 1]. Change row 1's rank to zero and they become [1, 2]. Delete row 1 and the answer becomes [2].
That is a history: a starting state followed by actions. The generator can produce many such histories while the same reference computes each answer.
The integration loop below is pseudocode; the named driver operations are not TanStack APIs:
create independent model and production fixture
for each generated action:
require action to be legal in the model
apply action through the production driver
apply action to the model
reach the contract's applied checkpoint
compare production's visible IDs with expectedIds(model, width)
finally:
clean up the fixture without hiding the original failure
For this settled-result law, no intermediate publication claim follows. If atomic publication is also promised, record callbacks during the action; a read afterward cannot recover a transient tear.
For this example, choose only actions that are legal now:
| Current model | Available choices |
|---|---|
| Empty | Insert an unused ID. |
| Contains ID 1 | Insert another ID; update 1; delete 1. |
| Contains IDs 1 and 2 | Insert another ID; update either live ID; delete either live ID. |
After choosing an action kind, choose its key from the applicable set. This is constructive generation: build a legal case rather than repeatedly discard illegal ones. Another valid approach generates commands with applicability checks and skips those that cannot run. Either way, count executed actions; a long list of skipped commands may do little work.
Shrinking means reducing a failing input or action list while keeping its failure. Design action dependencies so the smaller histories remain meaningful; don't disable shrinking merely because the test is stateful. The fast-check model guide describes commands and replay; check the installed version before copying an API recipe.
Pin the structural cases that matter: no rows, width zero, a boundary tie, and repeated changes to one key. Exhaust a small domain where that is cheap. Then randomize values and longer legal histories. Fixed cases, bounded enumeration and random exploration do different jobs; overlap between them is not a defect.
A fixed fast-check seed always generates the same cases. This makes a useful history stable, but it does not explore new histories on later runs. Do not describe a fixed-seed property as random coverage.
Under ORC-007, every important generated property runs in two campaigns:
- Run a fixed seed that preserves a known useful campaign.
- Run without a seed so fast-check chooses a new seed.
When the random campaign fails, retain the reported seed and shrink path. Give the suite environment variables or another checked replay interface. The replay input must select both the seed and the path. A seed alone reruns the campaign but might not stop at the same reduced counterexample.
fc.commands reports another value named replayPath. It records which
generated commands were eligible as the model changed. Capture that command
replay path separately from fc.assert's shrink path and pass it back through
the replayPath option of fc.commands; the assert seed and path alone do not
fully specify a command-model replay.
When replay inputs are present, run only the requested replay. Do not spend time on the fixed campaign before reaching the failure the developer asked to reproduce.
The fixed and random campaigns must use the same property, generators, observation recorder, refinement check, and run budget. Only their seed source may differ. This keeps a random failure eligible for promotion into a pinned example or fixed campaign.
fifo-retry-oracle.property.test.ts
shows this shape. Its fixed run preserves one scheduler campaign. Its second
campaign uses a random seed by default and accepts OFFLINE_ORACLE_SEED with
OFFLINE_ORACLE_PATH for replay.
Do not use a larger random run count as a substitute for structural reach. Pinned examples force rare boundaries. Bounded enumeration proves small finite domains. A stress job increases sampling depth. Record each form separately.
Suppose the update test only asserts that the result changed. Returning [] passes that assertion. Keep the update—it is a useful cause—but compare against [1, 2].
Now test the check against a known bad actual answer:
function sameOrderedIds(actual, expected) {
return (
actual.length === expected.length &&
expected.every((id, index) => actual[index] === id)
);
}
sameOrderedIds([1, 2], [1, 2]); // true: the allowed answer
sameOrderedIds([2, 1], [1, 2]); // false: correct members, wrong order
sameOrderedIds([], [1, 2]); // false: an arbitrary changed answerIn the test harness, assert those outcomes rather than merely calling the function. This calibrates the comparison; a production mutant that reverses results would test whether the whole fixture carries that fault to the assertion. Separately, confirm that the integration test reached the production boundary it names. Checker sensitivity and path reach are different claims: a perfect comparison on an unused code path protects nothing there.
When production fails, keep the original history and the first mismatching checkpoint. Reduce the history while requiring that same violation, then retain the small witness with the broader property. That gives us both a readable explanation and more ways to encounter the class of failure.
The Map works because the answer is cheap to compute independently. Other promises support other judgments:
| Method | Useful judgment | Important limit |
|---|---|---|
| Simple model | Recompute the allowed result or step a small abstract state. | The model can omit a meaningful distinction or encode the wrong rule. |
| Differential comparison | Run equivalent work through another implementation or formulation. | Both paths can share a bug; their semantics must actually agree. |
| Metamorphic relation | Transform an execution and check a relation between the answers. | The transformation's premises must hold. |
| Partial invariant | Check a specific promise such as no duplicate completion. | Passing says nothing about unobserved properties. |
| Recorded regression or snapshot | Preserve a known execution and expected result. | The stored result needs justification; recording it does not make it right. |
Choose the smallest authority that answers the question. Combining complementary checks can help; making every test compute every possible observation usually does not.
A reference becomes suspicious when it starts to look like production. But removing fields until it looks simple is not a sound design method either. Which distinctions can we safely discard?
Consider this illustrative support model. An acquisition handle supports a row. Replacing handle a with b transfers that support. Releasing a retired handle is a legal no-op. We observe support, not whether a provider physically deletes a source row.
| History | Current supported rows | Live support count |
|---|---|---|
Acquire a supporting x |
{x} |
1 |
Acquire a, then replace it with b supporting x |
{x} |
1 |
The rows and count look identical. Now release a. The first history loses support for x; the second retains it through b. A model containing only rows and counts cannot answer both correctly.
This gives the reasoning required by ORC-008: find two states the model treats as equal, then try a legal next action that could distinguish them. If their promised observations diverge—or an action is legal in only one—the model erased something relevant. The evidence may be an argument or a preserved witness; a separate executable test for every state field is not required.
The witness tells us to retain the ownership distinction here. It does not prove the revised model handles every retry, pending completion or replacement race. Nor does it require every real API to accept retired handles: if the actual contract rejects them, model that rule instead.
There are two questions behind “the model says so.” Who established the behavior? And which state determines the observable?
An example from the archive makes the risk concrete: a finite model represented scores only up to three, turning score > 3 into an empty predicate. The model had lost permitted values; that was not a reason to change production semantics. A separate ordering correction concerned which state owned the position: the model used locally edited values where the relevant contract used source state. The case notes retain that distinction; neither historical correction defines today's operator contract. See the case notes.
Before changing production to satisfy a red test, check the contract, representation and comparison. A genuinely unresolved product choice belongs with its owner. A missing value in a finite model belongs in the model. References—including newly written specifications—can be wrong.
Review the accusation and suggested fix separately. In one recorded adapter episode, returning a Promise created an acquisition lease even when it later rejected. Clearing ownership on rejection looked like cleanup but reportedly leaked that release obligation. The concern was useful; that repair did not follow from it. Other adapters may use different acquisition rules.
Projection deserves the same care. Removing internal metadata from a public-value comparison can be valid. Removing a contractual virtual field because it is called “metadata” cannot. Write down one difference the comparison may ignore and one it must retain.
Once the reference can distinguish relevant states, the generator must put production into them. Random payloads alone do not create ownership reuse, failed startup or work after recovery.
Look for relationships and transitions: same key versus fresh key; one owner versus a surviving sibling; reject before versus after acquisition; release before versus after completion; recover, then perform another operation. Keep healthy peers alive where isolation is part of the promise. A test that ends at “recovery succeeded” may never reveal broken next-use state.
Fresh disjoint rows can simplify a fixture while excluding the retired-key interaction it needs. Repeated true leadership reports do not cover true → false → true while earlier work is pending. These are different histories, not merely different random values.
Async operations can have several boundaries:
request invoked → response delivered → writes applied → view published
↘ returned Promise settles
That diagram names possible events, not a universal required order. The API contract must say which events precede which others. The test must control and observe the ones its claim depends on.
If setWindow applies work eagerly, delaying its returned Promise does not delay that work. In an earlier harness, an unconditional await let a queued microtask repair a same-turn mismatch before the assertion ran. A test that reads only settled state could not expose the earlier observation. Likewise, an equal initial window change can repair startup before startup is checked.
Put an explicit hold or release at the event you need to control. For example, a fixture testing visibility before application can arrange this illustrative sequence:
production requests data
fixture receives the response but holds delivery of its writes
test records the visible state while that delivery is held
test releases write delivery and lets production apply it
test records the visible state at the promised applied checkpoint
The fixture must really hold write delivery. If it delivers the writes and only delays returning from its load function, it controls a different event. Whether the returned Promise settles before or after application is a separate contract question. Record publications during the held interval if the promise concerns what users can see then. A scheduler is helpful only within the boundaries it controls; a seed cannot schedule arbitrary external I/O. The fast-check scheduler documentation explains the distinction between an operation's lifecycle and delivery of its result.
The historical four-receipt settlement fixture shows why enumerating more positions is not enough. A receipt here is a completion obligation for a portion of applied work. The tested contract required success to await the relevant receipts and failure not to wait for unrelated later work.
For each selected receipt, two schedules asked different questions:
| Question | Hold and release pattern | Observation |
|---|---|---|
| Can settlement ignore this receipt? | Keep it pending; apply every peer. | The request must still await this obligation. |
| Does its rejection wait for a later peer? | Apply earlier receipts; leave later ones pending; reject this one. | Observe the exact rejection while later work remains pending, and compare the intact partial rows with the expected applied prefix. |
Number the four receipts r0 through r3 and select r1. In the omission branch, apply r0, r2 and r3, hold r1, and check that the request remains pending. In the rejection branch, apply only r0; hold r2 and r3; reject r1 with error E. Before releasing either later receipt, check rejection with E and the exact rows from the applied prefix. The fixture's application action must control applied work, not merely rename a Promise resolution.
Resolving every peer first isolates omission but removes the pending sibling needed to expose delayed rejection. At the terminal receipt there is no later peer, so that cell cannot distinguish this delay. An empty expected prefix at the first receipt is valid; do not require nonempty rows just to prove the test ran.
The archive reports that an allSettled mutant failed by timeout in this fixture. That is evidence about delayed rejection, not proof that every row or error assertion rejected a wrong value. Keep the two claims separate.
Widening a generator can lose reach in two ways. A rewrite may remove a previously possible complete history, even while adding a dimension. Or a larger domain may make a useful history rarer at the same run budget. Neither means a true superset contains fewer cases; it means possibility and sampling frequency need separate checks.
A test provider can accidentally fix the consumer. One reported pagination fixture removed already-delivered rows before applying the requested limit. That supplied progress the consumer had not requested and hid repeated-page behavior.
Contrast a provider whose job is to gather several backend pages to satisfy one requested offset and limit. Its internal draining loop may be legitimate. The location of code in a test helper does not decide which case it is. State what the request asks, what the provider promises, and which progress production must arrange.
The same discipline applies to reentry. If a callback conditionally triggers the second operation, assert that the callback and trigger occurred. “If reached, check it” is a useful conditional claim, but it is not evidence that reentry was tested.
ORC-013 asks a different question from whether the fixture ran: would the tested cases reject a plausible wrong interpretation of the reusable law? A conditional assertion needs a legal case where its premise is true. A finite threshold matrix needs a case close enough to the boundary to distinguish the promised cut from a plausible displaced cut. The claim can instead remain about the named cases; record the broader law as open rather than stretching the matrix's result.
ORC-014 applies only when evidence crosses from a controlled premise to a real-provider or host claim. Name the exact premise being transferred. For example, a controlled callback can judge a component's response to an event; claiming that a real provider emits that same event under the relevant conditions needs a receiving witness. The two witnesses may live in different tests. A declared controlled-boundary limit is an honest scope, not a failed component oracle.
Reaching the right state still leaves a choice: what evidence survives the test harness?
Select observations from the actual promise. Membership, values, order, multiplicity, callback boundaries, downstream views, error identity, settlement timing, durable storage and resource ownership are not interchangeable. No suite needs all of them merely because they exist.
If an unprojected sort field changes, compare the expected new ordering, not just whether the array changed. If a callback count matters, assert the count before looping through callbacks; a loop over no callbacks checks nothing. If a failed write must not reach storage later, reconstruct persisted data after a later successful operation. The assertion should separate the specific wrong behavior from the allowed one.
It is useful to make generated inputs legal. It is dangerous to make recorded outputs incapable of being illegal.
Suppose production completes the same request twice. A request-keyed map that stores only the last completion collapses the violation. Record both events, then assert the promised count. Keeping the raw list alone is not enough if the final report folds it back into a map and loses the duplicate.
Similarly, do not reset a delta tracker before checking unchanged anchors, discard negative weights merely because the final relation should be nonnegative, or normalize away contractual order. Normalize only what the chosen comparison permits.
This section and the next are deeper branches for incremental-query and cross-formulation tests. If you are building an ordinary collection test, continue with testing the test and return when these comparisons become useful.
An incremental engine emits changes. A reference often computes a whole result. Compare compatible objects: accumulate the changes, then compare that state with full recomputation at the promised logical point. Inspect individual batches too when the contract makes their boundaries visible.
DBSP formalizes the computational relation as QΔ = D ◦ Q ◦ I: accumulate input changes, run the ordinary query, then take output differences. Equivalently, accumulating the incremental output should agree with querying the accumulated input. This is a useful source of laws for db-ivm. It does not choose whether a client may see a partial publication, when readiness changes, or what cancellation means. DBSP paper.
A final correct snapshot cannot establish that every earlier callback was coherent. The recorded graph-history case used asymmetric changes to expose an old-left/new-right publication that a later settled check would miss. Choose observation points that can distinguish the promised failure, rather than adding delays until the result is stable.
When scheduling legitimately allows several results, check for one permitted explanation of the observed history. Do not approve each field against a different allowed execution: those individually permitted pieces may never coexist. In specification terms, refinement means implementation-visible behaviors are permitted by the specification. A bounded test checks sampled histories; it is not a refinement proof. Nor can an accepted finite prefix prove eventual completion. Refinement mappings.
Even a plain reference can copy an incorrect semantic assumption. A second formulation gives another way to disagree.
This is the conditional requirement in ORC-011, not a universal requirement for every oracle. It applies after a plausible shared semantic fault and a meaningfully different formulation have been identified.
For a suitably scoped includes query, compare nested results with per-parent standalone queries or a flat join regrouped by parent. First state how empty parents, duplicates, ordering and projection match. A transformation that changes the answer is not an oracle for equivalence.
SQLancer names ternary logic partitioning, or TLP, as one such strategy: compare a query with recomposed predicate partitions. Here is a simple illustration of the partition relation, not a verified TanStack recipe: give every occurrence in an input exactly one label—true, false or unknown—according to the same predicate. Recombining all three groups, retaining every occurrence once, recovers that input. NULL behavior, aggregation and ordering need explicit treatment before turning this into a library recipe. It is not a license to split an arbitrary limited query into independently limited pieces. SQLancer's oracle inventory.
Metamorphic testing is the broader idea: transform an execution and check a known relation between the answers. Incremental-versus-recompute checks and legal history transformations can serve similar roles. They need not know one answer in advance, but they still need a justified relation.
Agreement across two paths is not proof of independence. NoREC's database experiments report shared operator faults and a copied join path that concealed a bug; count comparison can also miss wrong rows with the right count. Use differing formulations for the faults they can separate, not as an automatic second vote. NoREC paper.
“The suite passed” compresses several claims. Was the property selected? Did it run? Did it reach the intended path? Did its comparison execute? Would that comparison reject the fault under discussion?
Treat those as separate evidence. A replay registry can contain a property nobody invokes. A multiplier can be parsed without changing the run count. A no-op assertion can leave a campaign green. Use a positive execution witness and a deliberate wrong-result control where those claims matter.
Mutation testing changes production code deliberately to challenge tests. Record what happened, not just whether a mutant was “killed”:
- A value assertion rejected the intended wrong answer.
- A timeout exposed a missing progress obligation.
- Setup failed before reaching the comparison.
- Execution failed outside the intended comparison after setup, such as a crash.
- The changed code never ran.
- The mutant survived, or the change was equivalent within the tested domain.
These results call for different conclusions. If deleting one clause leaves the same obligation enforced elsewhere, survival does not show the obligation unnecessary. If the test kills a compound change, it may not distinguish each part. Research on mutation testing does not supply a conversion from a score to the probability that this library is correct; the reading notes preserve those limits.
Shrinking makes failures explainable by removing irrelevant values and actions. A smaller legal history that fails during setup is not a reproduction of an original torn publication. Require the original failure predicate: the violated law, the relevant observation point, and the distinguishing evidence.
Keep the original trace beside the reduction. Record which property ran and which checkpoint replay reached. “Ran, but did not reproduce the original mismatch” and “failed before reaching the comparison” are useful outcomes; neither should be labeled successful reproduction.
Capture evidence before cleanup can change it. The historical publication fix copied the batch list before rollback; that preserved the needed local evidence, not universal deep immutability. An outer copy may still hold mutable values. A serializer may lose object identity or reorder arrays. Choose capture rules for the law and retain an original representation when normalization would erase the disputed distinction.
Cleanup also has three separate jobs: preserve the primary failure, retain
secondary cleanup diagnostics, and release resources. Reporting the first error
does not prove the other two happened. Avoid letting a cleanup exception replace
the mismatch that started the investigation. “Separate” means distinguishable
in the report; an AggregateError may preserve the primary error as its cause
and each cleanup diagnostic in its errors.
A useful failure report contains:
Law and source; requested property and actually executed target/path
Code/dependency version; relevant environment
Seed/path and explicit action or schedule trace
Reached checkpoint; expected versus actual observation
Original trace; reduced trace; reproduction result
Primary failure; separate cleanup diagnostics
Check replay commands against the installed tool. A seed is not a recording of the network, operating system or every Promise. If a challenge influenced the design, keep it as a regression or tuning case; it is no longer a held-out test of that design.
When a reviewer finds a bug beside a green oracle, ask why the test missed it before adding another isolated example. Was the expected answer wrong? Was the history absent? Did the fixture drive another path or supply the missing work? Did the observation admit the wrong answer? Did capture or replay erase the failure?
These categories overlap. They are prompts for a concrete explanation, not an exhaustive taxonomy or a grade.
Begin a portfolio audit with known product promises as well as existing test claims. Otherwise ten honest settled-row tests can hide that nobody claimed startup rejection in the first place. Credit a witness at contract × history × production path × observation. A green component tested under a mock assumption does not establish that its real provider satisfies that assumption.
The historical React tie-group test illustrates precise credit. It exercised a real QueryClient/Query DB path with a finite test provider and checked visible prefixes and exhaustion. That could fill a portfolio-level gap without improving another test's weak assertion. It did not establish page-array contents, every framework or arbitrary server ordering. Preserve useful partial tests without lending them their neighbors' scope.
Before removing or combining tests, identify the valid history, expected outcome, production path and observation that will remain. Name the receiving test. “Retry the parent” becoming “delete the parent” changes the behavior, even if the replacement is shorter and green. The archive records exactly that kind of loss during reduction.
Other reductions are legitimate: an expectation can contradict the adapter contract, or two branches can enforce the same established law. Removing a redundant branch need not remove its law. Keep valuable fixtures and semantic assertions even when their old helper or duplicate implementation goes away.
Share stable mechanics such as replay parsing or resource cleanup where they are genuinely interchangeable. Keep domain truth small and separately reviewable. Several independent models should not grow into a universal miniature implementation merely to reduce duplication.
Sometimes the right correction is to narrow a claim. A reachable-object count is not a construction count: traversal may itself construct objects, and discarded objects are unseen. The historical repair both renamed that metric and checked source delivery before and after traversal. Renaming alone would omit part of the repair.
Narrowing evidence does not retire the larger promise. Keep two lines when necessary:
Observed: 100 irrelevant payload reads versus zero in the recorded probe.
Open: elapsed slowdown is unverified; the extra-work finding remains open
in that historical ledger.
Likewise, “assertions passed; process exited nonzero” is neither a clean green run nor an absence of passing assertions. An observed compatibility failure and an unresolved support policy can coexist. State both rather than squeezing them into one status.
A weak observer, a missing protocol signal, and a measurement not yet taken are different limits. None alone grants permission to drop a valid product obligation. The guide does not prescribe a universal run count, fault rate or stopping score.
This card is an optional presentation for the ORC-012 evidence. For a new oracle or a claimed repair, ask:
- What exact promise authorizes this expected result?
- Which legal history distinguishes the proposed model from a weaker one?
- Does the fixture make production do the work being tested?
- Which concrete wrong answer can the comparison reject—and which can it miss?
- What proves the path and assertion ran? What did the mutant or fault injection actually show?
- Can capture, cleanup or shrinking turn this into a different failure?
- Which larger promises remain outside this test, and where are they tracked?
- If a reusable boundary law is claimed, which case rejects a nearby wrong boundary? If a controlled premise supports a real-provider claim, where is that exact premise received?
Adapter and lifecycle oracles should consider these laws when the contract has the corresponding boundary. They are prompts, not universal requirements. State why an inapplicable law does not belong to the owner instead of adding a vacuous case.
- Real-provider conformance: freeze representative values from each supported provider version. Prove the fixture accepts those values before it stands in for that provider.
- Minimal ambiguity: include the smallest valid input for every classifier branch. Rich values that carry several redundant signals do not cover a one-field collision.
- Name invariance: changing a user-controlled name or SQL alias must not change envelope classification unless the public contract assigns that name structural meaning.
- Representation symmetry: equivalent array/object forms and coexisting carriers must produce the same public result or the same documented error.
- Await-boundary transitions: hold each relevant
await, change ownership, leadership, generation, abort, cleanup, or restart state, then release it. Compare the result with the contract for that transition. - Local/transport refinement: immutable transported data must have the same meaning on local and remote paths. Live local references must remain local, and cleanup must receive the exact lifecycle object delivered locally.
- Partial-construction cleanup: fail each construction step after it acquires a resource. Preserve the primary error and prove every acquired resource is released exactly once.
- Value-and-work refinement: when bounded work is promised, check the exact result and a deterministic work/cardinality measure. Correct rows alone do not establish the work law.
Every owner should also state its known omissions beside the contract. The coverage map tracks open reusable laws; an unchecked item is not evidence that the neighboring laws are absent.
The payoff is not a bigger test framework. It is a smaller distance between “this test is green” and a precise account of what that green result protects.