feat: implement the case matrix — records, reachability, methods, and the mark's exit - #13
Merged
Merged
Conversation
… the mark's exit
`test/spec/` was written before the implementation: 174 behaviours of which 111 were
`@test_broken`. This fills them in. 176 behaviours now, all live assertions, 0 broken.
The five questions the package answers, and what was added for each:
* **what this run entered** — unchanged, and now the floor rather than the whole thing.
* **how often, by which paths, how much of the run** — `record`. It emits nothing new: opening
a block clears every probe's flag, so the short-circuit fails and the *write* side, which is
a call rather than an inlined store, does the counting. Counts are exact, survive inlining,
and are correct under threads (per-thread padded counters sized by `maxthreadid()`). Time
comes from Julia's sampler through a new `Profile` extension, so `inclusive`/`exclusive` are
`missing` — never `0.0` — when nobody measured. `assert_clean`, `write_record`, `stamp`.
* **what a caller depends on without naming it** — `reach`, over inferred un-optimised IR.
Three-valued: `:depends`, `:clean`, and `:unknown` for a call site that cannot be pinned to a
method. A site with several candidates is walked through all of them, so "cannot tell which"
only becomes `:unknown` when one of them is actually marked. Module and script entry points,
`ignore=`, `dependents`.
* **what is unfinished** — marks now record the signature they attached to, so `reach` reports
the marked dispatch path and not its siblings, while `audit` still reads the name. Qualified
definitions (`Base.show(io, ::T) = …`) are accepted as method-level marks and stored in the
module that wrote them; `mark_method!` is the imperative route. `audit` gained the method
half — `contributed_methods`, `unaccounted_methods` — which is the only half a package whose
surface is `fetch(model, quantity)` has. `verification`/`coverage` join marks against
`--code-coverage`, flushed in-process.
* **may this mark go** — `until=` states the exit condition next to the reason;
`ready_to_promote` calls it, `marks_without_exit` reports the marks that never said, `age`
and `stale_since` read `since`.
Defects the spec caught, all of them a mark silently recording the wrong thing:
* `(c::C)(x) = …` marked `:c`, the argument name — and the audit then reported `:c` dangling
AND `:C` unaccounted, telling the author to declare what that line declares. Both fixed.
* `since = "0.4.0"` and a non-string reason were refused by the field's own conversion, with a
`MethodError` naming neither the keyword nor `@experimental`.
* a name-keyed mark made every sibling method experimental; `stable` now keeps a name in the
covenant until every method behind it is marked.
* the spec's own `invoke` case named an entry signature no method matched, and its
`@eval`-in-a-loop fixture interpolated at the wrong level. Both invisible while Broken.
One requirement was **withdrawn**: a mark inside a function body cannot be refused with a message
naming `@experimental`. `const` in local scope fails during lowering, before any emitted code
runs, and Julia's message does not name the variable either — measured byte-identical for a
binding whose name is the whole explanatory sentence. `test/spec/README.md` records the three
routes and why each fails; what is kept and asserted is that the blame lands on the author's line.
Also: `@generated`, `Base.@kwdef`, `@inline` and the other annotating macros compose with the
mark; a Documenter extension renders `@experimental` blocks; the API reference is split by source
file so a new file with no page is a build failure.
Verified: every test file individually on 1.12.2 and 1.11.9, and one full `runtests.jl` — 977
assertions, green, 2m49s with coverage and four threads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
📚 Docs preview: https://codes.sota-shimozono.com/ExperimentalAPI.jl/previews/PR13/ (updates on each push to this PR) |
`marks_markdown` was edited after the formatting pass — the heading-free rewrite that the Documenter HTML writer's `!(node.element isa MarkdownAST.Heading)` assertion forced — so `format / format-check` was red on one function. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
`julia nightly — ubuntu-latest` is `continue-on-error`, which makes it allowed to fail and not
allowed to be invisible. It failed 6 of 977. Both causes are real, and one of them was a false
"fully covered" that would have reached a release.
**`Type{X}` is no longer a `DataType`** (measured on 1.14.0-DEV.3115), and `Core.Typeof(Float64)`
now returns the new `Core.TypeEgal{Float64}` rather than `Type{Float64}`. Two `ft isa DataType`
guards — one in `_is_callable_type`, one in `_ftype_identity` — therefore stopped recognising
every constructor call in the graph, and four otherwise-clean fixtures came back `:unknown`. Both
now ask about `t` rather than about how `t` is represented: `t <: Type` with one non-`TypeVar`
parameter, read through `_type_parameter`, and normalised to `Type{X}` before a signature is
built.
**`--code-coverage` now emits a counter for the definition line of a method nothing ever called.**
Up to 1.12 that line had no counter at all, which is what `unverified` was reading: no counters
anywhere in the span meant nothing had ever generated code for it. On 1.14-DEV a one-line
definition comes back at `1.0` — fully covered on the strength of having been *defined*. The two
versions disagree about the same file and only one of them can be read as "the suite ran this".
So `unverified` no longer rests on that. The exact answer was already in the package: the probe.
A marked definition whose flag never fired is `0.0` whatever the line counters say, `unverified`
needs no `--code-coverage` at all now, and coverage supplies only the partial fraction — which is
the question it can still answer. `test/spec/test_spec_verify.jl` lost a version-conditional
branch as a result.
Verified: full suite on 1.14.0-DEV.3115 (978 green), and the affected files on 1.12.2 and 1.11.9.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s on nightly `recording does not disturb Profile` failed on macOS and errored on nightly. Both are defects in what this repository wrote, not in the platforms. **The buffer was not reliably filled.** The test profiled `Sim.driver(M, 200_000)` — about one millisecond, which is *one sampling interval* at Profile's default rate — and then asserted the buffer was non-empty. On ubuntu it caught a sample; on macOS it caught zero, and `before > 0` failed with `0 > 0`. An assertion whose subject is a sample count has to be given a run that is long compared with the interval, so it now profiles `Hot.grind(2_000_000)` at `delay = 1e-5`. That fixture already existed for the attribution test and moved up the file. **`fetch(; include_meta = false)` strips metadata behind an `@assert`.** On 1.14.0-DEV.3115 that assertion fires — `metadata stripping failed` — on a buffer this test did not fill. Two changes: the test reads `Profile.len_data()`, which answers "how much is in the buffer" without fetching or stripping anything; and the `Profile` extension does the strip itself through `has_meta` / `strip_meta` with a fallback to the raw data, because an exception there would have turned `record`'s timing silently off rather than loudly wrong. Verified: `test/spec/test_spec_profile.jl` on 1.12.2, 1.11.9 and 1.14.0-DEV.3115. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`main` moved under this branch: #11 cut the README from 1372 words to 419 for the registry's LLM policy, and #12 replaced the four required matrix contexts with one aggregate. Three files conflicted, and the resolution is not "keep both halves" in any of them: * **README.md** — `main`'s trimmed version wins wholesale. Re-adding the long sections written here would undo the point of #11. What is added back is two things that are now *false* without them: `record` and `reach` did not exist when that README was written, and the closing line called `test/spec/` "the specification for the propagation and profiling work that is not built yet". 559 words, against the 1372 that #11 cut. * **src/audit.jl** and **docs/src/checking.md** — the extension blind spot has a workaround #11 documented (declare the function in the parent, let the extension add only the method) and a report this branch added (`Audit.extensions`). Both are true, so both are said. The "methods on other packages' functions" bullet keeps this branch's version, because it stopped being a blind spot. `docs/make.jl` and `.github/workflows/CI.yml` merged cleanly and correctly — #11's canonical URL with this branch's page list, and #12's `all-tests` job with this branch's corrected runtime comment. Verified after the merge: `test_readme.jl`, `test_dogfood.jl`, `test_audit.jl`, the formatter, and a full docs build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/spec/was written before the implementation: 174 behaviours of which 111 were@test_broken. This fills them in. 176 behaviours, all live assertions, 0 broken.What each layer is, and what it costs
recordreachaudit, the coverage joinuntil=,ready_to_promote,age,stale_sincerecordemits nothing newThe statement
@experimentalputs in a body did not change. Opening a block clears everyprobe's flag, so the short-circuit fails and the write side — a function call, not an inlined
store — does the counting. Counts are therefore exact, survive inlining (which is what ruled out
the sampling route: a definition worth marking is small enough to be inlined, and a sampler has no
frame left to attribute to), and are correct under threads via per-thread padded counters sized by
maxthreadid().One change to the emission is load bearing and was found the hard way: the probe statement now
carries the declaration's own
LineNumberNode. Without it the cold write branch inherits whateverstatement the optimiser leaves next to it, and
record's call paths came back reportingiterateand
+whereenergyandinnerbelonged.Time comes from Julia's sampler through a new
Profileextension, soinclusive/exclusivearemissing— never0.0— when nobody measured.reachhas three answers because Julia's call graph is not closed:depends,:clean, and:unknownfor a call site that cannot be pinned to a method. Walksinferred, un-optimised IR:
optimize = trueseesmul_floatand finds nothing. A site withseveral matching candidates is walked through all of them, so "cannot tell which method" only
becomes
:unknownwhen one of the candidates is actually marked — otherwiseconvert(::Type, x)would make every caller that formats a string unknown.
There is deliberately no
verdictfield: a stored one makes:cleanwith a non-emptyunresolvedrepresentable, and that is the single state this must never report.Defects the spec caught
All of them a mark silently recording the wrong thing rather than refusing:
(c::C)(x) = …marked:c, the argument name — and the audit then reported:cdanglingand
:Cunaccounted, telling the author to declare the very thing that line declares.since = "0.4.0"and a non-string reason were refused by the field's own conversion, with aMethodErrornaming neither the keyword nor@experimental.attached to;
stablekeeps a name in the covenant until every method behind it is marked.invokecase named an entry signature no method of its fixture matched, and its@eval-in-a-loop fixture interpolated at the wrong level. Both invisible while Broken.One requirement withdrawn, with the measurement
A mark inside a function body cannot be refused with a message naming
@experimental.constinlocal scope fails during lowering, before any emitted code runs, and Julia's message does not
name the variable either — measured byte-identical for
__EXPERIMENTAL_API_MARKS__and for abinding whose name is the whole explanatory sentence.
globalfails silently;Core.evalremovesthe error altogether.
test/spec/README.mdrecords all three routes. What is kept and asserted isthe part that was in this package's hands: the blame lands on the author's line, never inside this
package.
Verification
Every test file individually on 1.12.2 and 1.11.9, plus one full
runtests.jl: 977assertions, green, 2m49s with coverage and four threads. Docs build clean; the API reference is
split by source file so a new file with no page is a build failure.
Overlaps with the open PRs
CI.yml; this edits only the header comment (the "half a minute"runtime claim is now false). No conflict.
README.md,docs/make.jl,docs/src/checking.mdandsrc/audit.jl, all ofwhich this also changes. Merge docs: trim the README, disclose the LLM assistance, fix the canonical URL #11 first and I will rebase, or the other way round — but not
in parallel.
🤖 Generated with Claude Code