feat: implement the case matrix — records, reachability, methods, and the mark's exit

.github/workflows/CI.yml

@@ -1,10 +1,12 @@
 name: CI
 
-# A version matrix rather than a shard matrix, deliberately. This suite runs in about half a
-# minute, so splitting it would cost more setup than it saves — but the mechanism it tests sits
-# directly on three things the language has been moving under it: the `public` keyword (1.11),
-# `names()` reporting public names, and 1.12's rule about reading a binding created in the same
-# world age. Which Julia this runs on is the variable that actually finds bugs here.
+# A version matrix rather than a shard matrix, deliberately. This suite runs in about three
+# minutes with coverage on — measured 2026-09-04, 977 assertions — so splitting it would still
+# cost more setup than it saves. The mechanism it tests sits directly on things the language has
+# been moving under it: the `public` keyword (1.11), `names()` reporting public names, 1.12's rule
+# about reading a binding created in the same world age, and the compiler internals `reach` walks
+# (`CodeInfo.codelocs` on 1.11 became `debuginfo` on 1.12). Which Julia this runs on is the
+# variable that actually finds bugs here.
 #
 # `test/test_precompile.jl` spawns its own subprocesses with `--depwarn=error`, so the deprecation
 # that would precede a hard failure on the next Julia fails the job rather than scrolling past.

.gitignore

@@ -1,2 +1,7 @@
 docs/build/
 Manifest.toml
+# `flush_coverage` writes these next to the source it measured, so a coverage-enabled run
+# of the suite leaves them in `src/`.
+*.cov
+*.info
+lcov.info

Project.toml

@@ -7,22 +7,33 @@ authors = ["sota shimozono <shimozono-sota631@g.ecc.u-tokyo.ac.jp>"]
 TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 
 [weakdeps]
+Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
+Profile = "9abbd945-dff8-562f-b5e8-e1ebf5ef1b79"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [extensions]
+ExperimentalAPIDocumenterExt = "Documenter"
+ExperimentalAPIProfileExt = "Profile"
 ExperimentalAPITestExt = "Test"
 
 [compat]
 Aqua = "0.8"
+Documenter = "1"
 Profile = "1"
 TOML = "1"
 Test = "1"
 julia = "1.11"
 
 [extras]
 Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
+Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 Profile = "9abbd945-dff8-562f-b5e8-e1ebf5ef1b79"
+TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Test", "Aqua", "Profile"]
+# `TOML` is a dependency of the package, and `test/spec/test_spec_integration.jl` also imports it
+# directly to read a stamp back. Named here rather than relied on through the manifest: a test
+# file that imports a package's own dependency without declaring it is the shape that breaks the
+# day the dependency is dropped.
+test = ["Test", "Aqua", "Documenter", "Profile", "TOML"]

README.md

@@ -45,6 +45,16 @@ docstring, and that no mark points at a name that was never made public. **A mar
 substitute for prose** — it records that a shape is unsettled, which is never a reason to say
 nothing about what the name does.
 
+Two further questions, both opt-in.
+[`record(f)`](https://qatlashub.github.io/ExperimentalAPI.jl/dev/observing/#Recording:-counts,-paths-and-time)
+counts how often a run entered each mark, by which paths, and how much of the run it was — exactly,
+without emitting anything the flag above does not already emit.
+[`reach(f, T)`](https://qatlashub.github.io/ExperimentalAPI.jl/dev/analysing/) asks the other
+question, before running anything: what a caller depends on without naming it. Its answer is
+three-valued, because Julia's call graph is not closed — `:depends`, `:clean`, and `:unknown` for
+a call site that cannot be pinned to a method. Reporting that third case as `:clean` would not be
+a weaker claim, it would be a false one.
+
 ## Install
 
 ```julia
@@ -55,13 +65,17 @@ pkg> add https://github.com/QAtlasHub/ExperimentalAPI.jl
 
 [Declaring](https://qatlashub.github.io/ExperimentalAPI.jl/dev/declaring/) ·
 [Observing](https://qatlashub.github.io/ExperimentalAPI.jl/dev/observing/) ·
+[Analysing](https://qatlashub.github.io/ExperimentalAPI.jl/dev/analysing/) ·
 [Checking](https://qatlashub.github.io/ExperimentalAPI.jl/dev/checking/) ·
 [Release decisions](https://qatlashub.github.io/ExperimentalAPI.jl/dev/releases/) ·
 [Adopting it](https://qatlashub.github.io/ExperimentalAPI.jl/dev/adopting/) ·
 [API](https://qatlashub.github.io/ExperimentalAPI.jl/dev/api/)
 
-`test/spec/` is the specification for the propagation and profiling work that is not built yet,
-written as tests so it cannot drift from the code.
+`test/spec/` is the specification, written as tests before the implementation so it could not
+drift from the code — 176 behaviours across ten files, all of them live assertions. Its
+[README](test/spec/README.md) records the negative control each group has, the two requirements
+that were withdrawn and why one of them could not be met, and the four defects the exercise found
+in the shipped code.
 
 ## Development
 

docs/make.jl

@@ -22,10 +22,15 @@ makedocs(;
         "Home" => "index.md",
         "Declaring" => "declaring.md",
         "Observing" => "observing.md",
+        "Analysing" => "analysing.md",
         "Checking" => "checking.md",
         "Release decisions" => "releases.md",
         "Adopting it" => "adopting.md",
-        "API" => "api.md",
+        "API" => [
+            "Declaring" => "api.md",
+            "Observing and analysing" => "api-runtime.md",
+            "Checking and releasing" => "api-checks.md",
+        ],
     ],
 )
 

docs/src/analysing.md

@@ -0,0 +1,114 @@
+```@meta
+CurrentModule = ExperimentalAPI
+```
+
+# Analysing
+
+[`entered`](@ref) and [`record`](@ref) say what a run *did*. [`reach`](@ref) says what a caller
+*could* do — before running it, and including through code that never names the marked thing.
+
+```julia
+julia> r = reach(analyse, Tuple{Model,Float64});
+
+julia> verdict(r)
+:depends
+
+julia> r.reached
+1-element Vector{ExperimentalAPI.Reached}:
+ Reached(MyModel.energy via analyse → sweep → inner → energy)
+```
+
+The model is Lean's `sorry`: a proof that uses one is not a proof, however many layers down it
+sits. Julia's call graph is not closed, though, so the answer has to be three-valued.
+
+## Three answers, and why the third one exists
+
+| verdict | |
+|---|---|
+| `:depends` | a marked definition is reachable. Proved, not suspected |
+| `:clean` | the whole call graph was resolved and nothing marked is in it |
+| `:unknown` | at least one call site could not be pinned to a method |
+
+`:unknown` is the point of the design. Two shapes really can reach a marked function while being
+statically invisible:
+
+```julia
+struct Holder; f::Function; end
+top_field(h::Holder, x) = h.f(x)          # the callee is a value chosen at run time
+
+const TABLE = Function[unstable, solid]
+top_table(i, x) = TABLE[i](x)             # …and so is this one
+```
+
+Reporting `:clean` there is not a weaker claim, it is a false one. Every unresolved site comes
+back as an [`Unresolved`](@ref) carrying the file, the line, why it could not be resolved, and —
+when they are visible — the marked methods it could have reached.
+
+There is deliberately no `verdict` **field**: a stored one makes `:clean` with a non-empty
+`unresolved` representable, and that is the single state this analysis must never report.
+[`verdict`](@ref) derives it, the way [`isbreaking`](@ref) derives its answer from a
+[`Diff`](@ref). [`combine`](@ref) folds two verdicts (`:clean` < `:unknown` < `:depends`), which
+is what [`reach`](@ref)`(::Module)` does over a module's entry points.
+
+## What it resolves
+
+The walk is over **inferred, un-optimised IR**. Inference runs before inlining, so every call is
+still a call and every argument still has a type; `optimize = true` would show `mul_float` and
+find nothing.
+
+| call site | |
+|---|---|
+| a named call, however deep | resolved |
+| a function passed as a value | resolved — Julia specialises on `typeof(f)` |
+| a `@nospecialize`d callee, called with a concrete function | resolved |
+| `invoke(f, Tuple{Integer}, x)` | resolved to the method `invoke` pins, not the one dispatch would pick |
+| a marked `const` or `struct` used in the body | `:depends` — a const is not a call site, and reading globals out of the IR is how it is seen |
+| a `Union`- or abstract-typed argument whose candidates include a marked method | **`:unknown`** |
+| a callee read out of a field or a table | **`:unknown`** |
+
+A call site with several matching methods is not automatically unresolved: every candidate is
+walked, and if none of them reaches anything marked the site is resolved after all. That is not a
+guess — it is having checked all of them. Without it, `convert(::Type, ::UInt32)` (dozens of
+matching methods, none of them anybody's research code) would make every caller that formats a
+string `:unknown`.
+
+A more specific unmarked method shadowing a marked one is resolved as what actually runs: an
+`Int` goes to `more_specific(::Int)` and is clean, while a `UInt8` falls through to the marked
+`::Integer` method and is not.
+
+## Whole modules, and scripts
+
+```julia
+r = reach(MyPackage)
+verdict(r)                # one answer for the package
+r.affected_entries        # …and which public entry points are not clean
+```
+
+Function-by-function does not scale to a package, and "something in here is experimental" is not
+actionable. Each entry point gets its **own** walk: sharing one visited set would make the second
+entry that reaches a mark through an already-walked callee look clean.
+
+[`reach_script`](@ref) is the shape a researcher actually has — a file that produces a figure, not
+a package. Note what it costs: the script's top-level `using`, `const` and type definitions are
+evaluated in a scratch module, because the analysis has to resolve the names the script uses.
+
+## The exit, read backwards
+
+```julia
+dependents(MyPackage, :energy)                  # who reaches it
+verdict(reach(MyPackage; ignore = [:energy]))   # what removing the mark would change
+```
+
+`ignore` answers "what would removing this mark change?" without removing it. [`dependents`](@ref)
+is propagation read the other way: a mark gets deleted because somebody looked at the definition,
+not at who reaches it.
+
+## What it is not
+
+  * **Not a run.** It says what *could* be reached; [`record`](@ref) says what was. A path this
+    reports is not necessarily taken.
+  * **Not sound past a dynamic call.** That is what `:unknown` is for, and why
+    [`isclean`](@ref) answers `false` for it — the predicate means "may I rely on this", and the
+    honest non-answer is not a yes.
+  * **Not free.** It runs inference over the call graph. `maxdepth` and `maxcandidates` bound it,
+    and hitting either bound is reported as `:unknown`, never as `:clean`.

docs/src/api-checks.md

@@ -0,0 +1,13 @@
+```@meta
+CurrentModule = ExperimentalAPI
+```
+
+# API — checking and releasing
+
+The audit ([Checking](@ref)), the coverage join, the mark's exit, and the release layer
+([Release decisions](@ref)).
+
+```@autodocs
+Modules = [ExperimentalAPI]
+Pages = ["audit.jl", "verify.jl", "lifecycle.jl", "docsnote.jl", "release.jl", "ExperimentalAPI.jl"]
+```

docs/src/api-runtime.md

@@ -0,0 +1,12 @@
+```@meta
+CurrentModule = ExperimentalAPI
+```
+
+# API — observing and analysing
+
+What a run went through ([Observing](@ref)), and what a caller could reach ([Analysing](@ref)).
+
+```@autodocs
+Modules = [ExperimentalAPI]
+Pages = ["detect.jl", "record.jl", "reach.jl"]
+```

docs/src/api.md

@@ -2,26 +2,27 @@
 CurrentModule = ExperimentalAPI
 ```
 
-# API
+# API — declaring
+
+The mark itself, and reading marks back out. The rest of the surface is on
+[API — observing and analysing](@ref) and [API — checking and releasing](@ref); the split is by
+source file, listed explicitly, so a new file with no page is a build failure rather than a
+silently missing section.
 
 ```@autodocs
 Modules = [ExperimentalAPI]
+Pages = ["mark.jl", "query.jl"]
 ```
 
 ## Declared unfinished
 
-Generated from the package's own marks at build time, so it cannot go stale — and it is the
-same call any consumer would make:
+Generated from the package's own marks at build time by this package's own Documenter extension,
+so it cannot go stale — and it is the same call any consumer would make:
 
-```@example marks
-using ExperimentalAPI
-# `experimental` is `public`, not exported, so it is qualified — which is the visibility
-# convention this package leans on rather than duplicates.
-for mk in ExperimentalAPI.experimental(ExperimentalAPI)
-    println(mk.name, "\n    ", mk.reason, "\n")
-end
+```@experimental
+ExperimentalAPI
 ```
 
-A docstring says what a name does. This says whether it is finished. Nothing on this page is
+A docstring says what a name does. That block says whether it is finished. Nothing on this page is
 undocumented — the names above are documented **and** declared, which is the normal state for
 something that works but whose shape is still being argued about.

docs/src/checking.md

@@ -76,7 +76,12 @@ it had none:
   cannot tell you that you should have.
 - **Prose quality.** A docstring reading `TODO` counts as documented. [`isdocumented`](@ref)
   asks whether prose exists, never whether it is any good.
-- **Methods added to another package's function.** They are not in `names(m)` and never will be.
+- **Methods added to another package's function** are not in `names(m)` and never will be —
+  which is why the audit has a second half. [`contributed_methods`](@ref) finds them,
+  [`unaccounted_methods`](@ref) reports the ones with neither a docstring nor a mark, and
+  [`test_surface`](@ref) asserts on them. For a package whose surface *is* such methods —
+  `fetch(model, quantity)` with 570 of them — a clean name audit reports nothing while having
+  looked at none of them.
 - **Names public only inside an extension**, which is a separate module — audit it separately, or
   avoid the blind spot the way this package does: declare the function and its docstring in the
   parent (`function test_surface end` in `src/ExperimentalAPI.jl`) and let the extension add only
@@ -99,4 +104,55 @@ contradiction. What `test_surface` adds is `foreign` (a re-exported name whose p
 else's job) and `dangling` (a mark on a name that was never made public), neither of which
 Documenter has a notion of.
 
-Run both. Neither is a looser version of the other.
+Run both. Neither is a looser version of the other, and
+[`aqua_compatible_names`](@ref) computes the difference between them so a project running both can
+see exactly which names it would have to argue about. Empty means the two agree.
+
+## The method-level half
+
+```julia
+julia> audit(Downstream).contributed_methods
+4-element Vector{Method}:
+ fetch_value(::Ising, ::Energy) …
+ ⋮
+
+julia> unaccounted_methods(Downstream)      # neither a docstring nor a mark
+2-element Vector{Method}:
+ ⋮
+```
+
+Docstrings are keyed by signature, so [`isdocumented`](@ref)`(::Method)` is a real question and
+not the same one as [`isdocumented`](@ref)`(m, :name)`. Only the module that *wrote* the method is
+asked: the generic's own docstring upstream has the key `Tuple{Any, Any}`, and letting it count
+would make one docstring account for all 570 methods anybody ever contributed.
+
+[`test_surface`](@ref) asserts on these with `require_methods = :foreign` by default — methods on
+another *package's* generic, not on Base's. `Base.show(io, ::Audit)` implements a protocol whose
+documentation is Base's, and a default that reported every `show`, `==` and `getindex` method as a
+finding is a default that gets switched off wholesale. [`extends_base`](@ref) is the rule, stated
+as "who owns the generic" rather than as a list of interface functions — a list is not closed
+under the ones Julia adds next. `require_methods = :all` widens it.
+
+## How well is any of it exercised?
+
+The mark records where it was written, and `--code-coverage` records a count per line. Joining the
+two answers the worst case a marked definition can be in:
+
+```julia
+julia> unverified(MyPackage)          # marked AND never executed by the suite
+1-element Vector{ExperimentalAPI.Mark}:
+ Mark(MyPackage.never_called, "shipped without ever being called")
+
+julia> coverage(MyPackage, :half_exercised)
+0.6
+```
+
+[`coverage`](@ref) answers `missing`, never `0.0`, when the run has no coverage data: a run
+without `--code-coverage` has nothing to say, and reporting zero would flag every marked
+definition in every ordinary run. The counters are flushed from the running process rather than
+read from the files Julia writes at exit, because a test that has to wait for the process to end
+cannot assert anything.
+
+[`stale_marks`](@ref) is the check that keeps the join honest: an edit above a definition moves
+the code and not the record, and every downstream reader of `mk.line` then describes the wrong
+lines silently.