Add resident speculate: test an instantiation hypothesis on a retained query - #36
Conversation
…d query A `speculate` request checks a retained query as usual, observed for matching loops, then again with one hypothesis sent in the query's own scope through cvc5's `(speculate ...)`: instantiate a quantifier at given terms, give it one more trigger, or refuse its instantiations that fit a fingerprint. `(get-info :speculation)` is read in the same batch, and finish_query pops the hypothesis with the scope. The reply says whether the hypothesis closed the query (failed before, held after, failed again on a recheck, since queries near the resource limit flip between checks of their own), whether it introduced a matching loop, the instances it made (LLM_DIRECTED), and source to paste: an assert of the instance with its type guards dropped, or a #![trigger ...] annotation. Without a hypothesis it lists the quantifiers written in source that the query's scope asserts. Terms may be Verus expressions or SMT terms. air::speculate finds the quantifier by qid in the retained AIR; the lowering resolves source names among the scope's declarations and boxes into Poly (or unboxes) where a function or variable takes the other. The journal now keeps the bucket's setup batches, which the solver held below its scopes, so declarations there can be read. A term cvc5 cannot read is refused in the early flush, before any check-sat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The listing stops at MAX_CANDIDATES but read as complete; every toyDB query hit the cap. The note now gives the count listed and the count the scope asserts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # source/air/src/smt_verify.rs # source/rust_verify/src/resident.rs # source/rust_verify_test/tests/resident.rs
|
BasisResearch/cvc5#14 changed after a review, and this branch is now merged with cvc5#14 changes that reach this PRcvc5#14 is at
Merge with
|
|
cvc5#14 changed again (head
I have not rerun this PR's |
ReviewReviewed at 60cfcc6 against cvc5#14 at 47d666d, built and run on a separate machine (arm64 macOS). What holds. fmt and clippy are clean. Bugs. Each one was reproduced with a fixture test.
Fixed on cvc5#14 at 47d666d: a directed instance over a constant cvc5 had substituted away never connected. At 4cf5707, Nits.
I am pushing fixes for all of the above to this branch, with regression tests, and will follow up with the results. |
… kg/speculative-probe The journal's base batches follow main's record_base, which twin added for the same purpose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review fixes. An instantiation's terms stand outside the quantifier, so they no longer resolve names through its variables: a parameter the quantifier shadows is the parameter. A hypothesis's terms are now read after the check without it, at the goal it failed at (the last goal when it names none): Verus terms through the scaffold lowerer, which reads method calls, indexing, casts and old, then as the lowered query reads them there (air::GoalScope), so a mutable local is its version at the goal. SMT-spelled terms map a variable's AIR name to the same version, declare every version, and ask the AIR context about the prelude's functions instead of seeding four boxes by hand, so (Add a! 1) is boxed. A snippet no longer names a variable at another version than the one where it is pasted; QueryNames::pasteable checks the versions live at the goal, for speculate and for the egraph's verus_assert alike. An echoed marker precedes (speculate ...), so only an error after it counts as the hypothesis refused. The status docs name the three reasons for rejected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes pushedTwo commits on top of 60cfcc6:
The fixes.
The nits. An echoed marker now precedes Tests (arm64 macOS; cvc5#14 at 47d666d for the ignored ones):
The toyDB acceptance runs in the description predate these fixes. Their terms were all in the solver's spelling, which reads the same now, but they were not rerun. The description is updated. The cvc5 pin still has to move to a release with #14 at 47d666d or later before the speculate tests can be un-ignored. |
|
cvc5#14 changed again, after a third review: head
Still stale from the last round: the Not done: |
…robe Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cvc5 eliminates a variable an equality in a formula's body fixes, and the formula it holds then binds the rest; since da4b2b0073 it keeps the :qid, and answers an instantiation that names the eliminated variable with mismatch. The resident now sends the instance again without each variable cvc5 reports unbound, lists them in eliminated, and offers as the snippet the instance cvc5 made, since the requested term for an eliminated variable need not be the one its equality fixes. A trigger's fallback is built only when its terms line up with the variables. A no_quantifier answer for a formula the scope asserts now says why: cvc5 registers alpha-equivalent formulas once, under the first qid. The status docs name the fourth rejected reason (the instance simplifies to true) and what pending means since 47d666d. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Side-by-side additions in the module docs, COMMANDS and the listed requests; speculate reads the journal's base and prefix as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Verus fixture's equality guard is between boxed values under type guards, and cvc5 keeps both of its variables, so the retry is covered by a unit test of unbound_variable; the fixture's comment says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Addressed in f553280 and 2ba14e6, with main merged in twice (8551697 for #41, eff2440 for #33), and tested against cvc5#14 at da4b2b0073.
Tests (arm64 macOS, on the merged tree):
|
|
cvc5#14 is merged, so this PR now pins its release and runs its own tests.
Tests (arm64 macOS, pinned cvc5 fetched by
Ready to merge once CI passes on this head. Nothing is waiting on cvc5 any more. The toyDB acceptance numbers in the description predate the review fixes and were not rerun; the plan's toyDB criterion was not met, as the description says. |
Implements the Verus side of
plans/speculative_probe.md. It uses BasisResearch/cvc5#14,(speculate ...)and(get-info :speculation). cvc5#14 is merged, and the cvc5 pin here isbasis-e68dc63e37, cvc5 main at that merge.What a
speculaterequest does{"command":"speculate","session":"...","bucket":0,"query":3, "hypothesis":{"instantiation":{"qid":"user_crate__f_3","subst":{"i":"a"}}}}hypothesisis one of the following. Without one, the reply lists the quantifiers written in source that the query's scope asserts (qid, variables with source and SMT names and sorts, triggers), the first 40;notessays when the scope asserts more.instantiation{qid, subst}: instantiate once at the given terms.trigger_pattern{qid, pattern}: add one more trigger;patternis one term or a list.block_cycle{qid, fingerprint}: refuse the instantiations that fit a shape.A term is a Verus expression (
s.len() - 1,f(y)) or an SMT term in the solver's spelling ((I y@1),(Add a! 1)). It is read at the goal the query fails at, so a mutable local means its value there.The worker checks the query as usual (
before), then with the hypothesis sent in the query's own scope (after). Ifafteris valid, it checks again without the hypothesis (recheck). Queries near the resource limit flip between two checks of their own; the egraph work saw this. So a close counts only if the recheck still fails.The reply carries:
closed;introduced_loopandnew_loops: a quantifier whose deepest instantiating term got deeper in at leastloop_thresholdrounds after the hypothesis, where it did not without it;new_provenance.closing_instantiations: the instances it made, taggedLLM_DIRECTED;verus_snippetafter a close: anassertof the instance for an instantiation, or#![trigger ...]for a trigger, which also getsfallback_snippet, an assert of one instance it made;readings: how names in Verus terms were read, where more than one reading was possible.A hypothesis that cannot be applied comes back with a
statusand a reason (no_quantifier,mismatch,could_not_lower,unusable,rejected). The session goes on serving.Design
air::speculate. It holds the request, the reply parser, and a finder that locates a quantifier by qid in the retained AIR (prefix declarations and the query) in the solver's spelling. It also substitutes binders for rendering.(speculate ...)command's outer syntax is fixed by the worker: the qid and variable names come from the AIR quantifier the worker found (so a qid the caller made up is answeredno_quantifierand never sent), and every caller-written term, pattern and fingerprint goes as a string literal that cvc5 parses on its own and can refuse without ending the session.Context::set_speculationsends(speculate ...)just before the early flush (get-info :version), after the query's assertions, with an echoed marker in front of it. A term cvc5 cannot read therefore comes back as an(error ...)line after the marker; an error before it is the query's own and stays unexpected output. The check then returnsCanceledwithout acheck-sat, and the reply carries the error.(get-info :speculation)is read in the same batch as thecheck-sat.supports_speculationasks(get-info :speculation)once, because a cvc5 without the command would end at it.sst_to_air. There is no VIR at resident time, so the worker reads the text itself. It checks the request's variable names first, then runsbefore, and reads the terms at the goalbeforefailed at (the query's last goal when it names none).scaffold::lower_term). It reads calls, method calls, fields, indexing,@,old, casts and the operators, and boxes as the encoders do.air::GoalScopethen reads the result as the lowered query does at the goal, so a mutable local is its SSA version there.$,a!), or an application headed by an SMT operator or a declared function. It is resolved symbol by symbol. A variable's AIR name reads as its version at the goal, every version is declared, and the prelude's functions come from the AIR context.x == y ==> ...), and the formula it holds then binds only the rest. It answers an instantiation naming the eliminated one withmismatch. The worker sends the instance again without each such variable, lists them ineliminated, and offers as the snippet the instance cvc5 made (its:bodies), since the requested term need not be the one the equality fixes. A trigger's fallback is built only when its terms line up with the quantifier's variables.no_quantifierfrom cvc5. cvc5 registers alpha-equivalent formulas once, under the first qid, so a hypothesis on the second qid finds nothing there even though the scope asserts it.notessays so.base. The journal keeps the bucket's setup batches (fuel, traits, datatypes, function declarations), which the solver holds below its scopes. They are only read, never replayed. Add the resident twin request: a query and one edit of it, compared #34's twin added the samebaseandrecord_base; this PR now uses those.has_typeguards dropped. Where that does not read as source, it is an assert that mentions the trigger's instance instead.QueryNames::pasteable, factored out of the egraphverus_assert, applies the same rules: every symbol must read as source, no variable may appear at two versions, and none at another version than the one where the snippet goes. The egraph'sverus_assertgets the last rule too.Testing
Run on arm64 macOS against the pinned cvc5 (
basis-e68dc63e37), with the solver version check on. The two speculate tests run with the rest of theresidentsuite.resident_speculate_probes_and_leaves_the_session_unchangedcovers:assert(((crate::g(a) > 0) && (crate::f(a) > 0)));(recheck stillinvalid). A fresh worker with that assert pasted verifies.f(i)closes with#![trigger crate::f(i)]; pasted into the quantifier, it verifies.h(x) > h(s(x)),beforeisresource_limitwith the quantifier rising past the threshold. Withblock_cycleh(s(s(_))),afterhas no loop and instances were refused.gnever fires. Given the triggerh(x), it climbs, andintroduced_loopis true.no_quantifierwith candidates,mismatch,could_not_lower, and an out-of-rangeloop_threshold. A last check answers exactly as the first did, one solver process served everything, and pushes and pops balance.resident_speculate_reads_terms_at_the_goalcovers:proof fn shadowed(i: int) requires forall|i: int| ...,{"i": "i"}is the parameter and closes the goal.let mut y = a; y = y + 1;,{"i": "y"}closes, and its snippet, pasted before the goal, verifies in a fresh worker.(Add a! 1)is boxed and closes the same goal.f(i)matches cvc5's term overy's first version, so an assert of it is offered only when it names no stale version, and one offered verifies pasted.{"i": "s.len() - 1"}closesassert(s[s.len() - 1] > 0).forall|x: A, y: A| x == y ==> p(x)) closes at{x: a, y: a}. cvc5 keeps both of its variables, since the equality is between boxed values under type guards. The retry for an eliminated variable is therefore covered by the unit testan_eliminated_variable_is_read_from_the_refusal, and the elimination itself by cvc5#14'sspeculate-varelimregression; no Verus fixture reaches it yet.residentsuite against the pinned cvc5: 52 passed, none ignored, the two speculate tests included.airunit tests: 216 passed, including request printing, the reply parser, binder substitution, and goal scopes. Therust_verifylib tests: 48 passed, including sort-directed boxing, ground terms and versions at the goal, SMT-or-Verus reading, stale-version snippets, and type-guard dropping. fmt and clippy are clean.toyDB acceptance: not met
These runs predate the review fixes: they were measured at d2dcec3 with cvc5#14 at 4cf5707, through the MCP server (
speculative_probe) against toyDB d79db20. Every term was given in the solver's spelling, which reads the same after the fixes.Targeted hints. Five toyDB functions each had one hint removed. Each hint is an instance of the function's own quantified
requires, and each was probed with that instance:keycode::lemma_lex_lt_atassert(a[0] == b[0]);j := 0, applied, validkeycode::lemma_allffassert(p[0] == 255);j := 0, applied, validkeycode::lemma_prefix_end_fwdassert(e[j] == p[j]);verified_lexer::lemma_scan_digits_end_runassert(is_digit(input[pos]));i := pos, applied, validverified_lexer::lemma_scan_ident_end_runassert(is_ident_cont(input[pos]));i := pos, applied, validSweep. Every single-line
assertin eight keycode and lexer proof functions was removed one at a time, 53 variants in all. Only one broke its proof.lemma_allffwithoutassert(k.subrange(0, p.len() as int) =~= p);fails its postcondition. Probing it with vstd'slemma_seq_ext_equalatA&. := $,A& := (UINT 8),s1 := k.subrange(0, p.len()),s2 := papplied the instance, but the query stayedinvalid. The type binders and boxed terms were given in the solver's spelling. The removed assert is a proof subgoal: Verus proves the pointwise equality behind=~=itself. A single axiom instance cannot stand in for that.So toyDB's hints of this kind are redundant under cvc5, and the one necessary one is not an instantiation. The plan's criterion (at least three toyDB unknowns flipped by a pasted snippet) is not met, and this PR does not claim it. What these runs do show:
The flips themselves are shown on the fixtures in the resident tests.
Review changes
First round:
serve_speculate: the candidate listing is capped atMAX_CANDIDATES(40) but said nothing about it, so a scope with more looked fully listed;notesnow says "the first 40 of the N quantifiers" when it is cut. The toyDB numbers above are restated accordingly.(speculate ...)per query scope, popped byfinish_query, with cvc5 holding all of it in that user context); and theCanceledpath for a refused command, which pops like any canceled check.Second round (see the review comment):
could_not_lower. They are now read outside the quantifier.Var, which cvc5 does not declare. Terms are now read at the goal the query fails at, where the local is one SSA version.g((y + 1))overy's first version, which pasted before the goal reads asa + 2.pasteablenow requires the version where the snippet goes, for the egraph'sverus_asserttoo.(Add a! 1)was never boxed. It now asks the AIR context.(errorline anywhere in the early flush counted as the hypothesis's refusal; an echoed marker now separates the query's output from the command's.rejected's documentation names all three reasons cvc5 gives.@are read.y@1 = Add(y@0, 1)) was applied but never connected. That was cvc5's, and cvc5#14 fixes it at 47d666d.Third round (after cvc5#14 moved to da4b2b0073):
:qidthrough variable elimination, and answers an instantiation naming the eliminated variable withmismatch. The worker retries without it, as above, so such a formula can be instantiated at all; da4b2b0073 is now the minimum.rejectedhas a fourth reason, "the instance simplifies to true", and the docs inresident.rsandair::speculatelist it.pendingnow means that no round reached e-matching (for instance because conflict-based instantiation closed every check first), in both docs.no_quantifieranswer for a formula the scope asserts explains alpha-equivalence innotes.basis-e68dc63e37, cvc5 main at that merge, and the two speculate tests are no longer ignored. The one cvc5 change after da4b2b0073 reads an integer term given for a real variable as a real, which Verus's encoding never sends.Overlap
mainat da8fa53 (#33's ablate, #34's twin, #35's ladder, #41's scaffold placement, and the cvc5 pin at the pending-lemmas release) is merged in. Its conflicts were side-by-side additions incontext.rs,lib.rs,smt_verify.rs,resident.rsand the resident tests, kept both; the journal'sbaseandverifier.rsfollow main's.COMMANDSislist, check, bisect, ablate, egraph, scaffold, close, inst_graph, ladder, twin, speculate. The cvc5 pin isbasis-e68dc63e37, cvc5 main at #14's merge, which also carries #11's:branch-profile.🤖 Generated with Claude Code