feat: report which experimental definitions a run entered

README.md

@@ -6,39 +6,71 @@
 [![Code Style: Blue](https://img.shields.io/badge/Code%20Style-Blue-4495d1.svg)](https://github.com/invenia/BlueStyle)
 [![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-`public` says who may call a name. Nothing says whether the name is finished.
+`public` says who may call a name. Nothing says whether the name is *finished* — and nothing tells
+you, after a twelve-hour run, that the number you are about to publish came out of code its author
+had not validated.
 
-Julia already checks half of that. `Docs.undocumented_names` has been public API in Base since
-1.11, and `Aqua.test_undocumented_names` ships it as a test: every public name must carry a
-docstring. What neither can express is the **third option** —
-
-> this name is public, it has no docstring, and that is deliberate: the shape is not settled,
-> and here is why.
-
-ExperimentalAPI adds that option at the definition site, and makes it something a tool reads
-rather than prose a human might happen to notice.
+`@experimental` is that mark, written at the definition site where the author is, and readable by
+a machine so the answer arrives without anyone remembering to ask for it.
 
 ```julia
 using ExperimentalAPI
 
-@experimental "reads Test's internal result tree; not dogfooded in CI yet" \
-function render_test_report(records)
-    # ...
-end
+@experimental "convergence not established below β ≈ 0.1" energy(β) = β * 1.0000001
+
+energy(0.5)
+```
+
+Run that file and it ends by telling you something you did not ask for:
+
+```console
+$ julia sweep.jl
+┌ ExperimentalAPI: this run entered 1 experimental definition
+│   Main.energy — convergence not established below β ≈ 0.1
+└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this
 ```
 
+It is on by default, it carries the **reason** rather than just the symbol, and a marked
+definition the run never entered is *absent* — not reported with a count of zero. The same answer
+is available programmatically:
+
 ```julia
-julia> ExperimentalAPI.audit(MyPackage)
-Public surface of MyPackage — 46 names
-  documented      45
-  experimental     1
-  unaccounted      0
+julia> ExperimentalAPI.entered()
+1-element Vector{ExperimentalAPI.Entry}:
+ Entry(Main.energy, "convergence not established below β ≈ 0.1")
 ```
 
-The mark costs nothing at run time: `@experimental` emits your definition unchanged plus one
-`push!` at load time. Calls are not wrapped.
+That example is executed verbatim by `test/test_readme.jl`, so it cannot rot.
 
-## The check is the point
+## What it costs
+
+A read, plus one write on the first call:
+
+| emitted into the body | 1 thread | 8 threads |
+|---|---|---|
+| nothing | 1.00× | 1.00× |
+| **the flag `@experimental` emits** | 1.03× | **0.985×** |
+| a counter, plain shared `Ref` | 1.03× | 3.76× — and loses 40% of its increments to races |
+| a counter, global atomic | 1.17× | 4.87× |
+| `@warn`, guarded so it fires once | 5.65× | — |
+
+10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. The
+flag is written once and only read afterwards, so it stops dirtying the cache line — which is why
+it is free at eight threads while every counting scheme is not, and why the notice is a summary at
+exit rather than a warning at the call. Counting, call sites and paths are a separate, opt-in
+layer that is not built yet.
+
+Only a definition **with a body** carries a flag — `function`, `f(x) = …`, parametric and
+return-type-annotated signatures alike. A mark written as a name list, or attached to a struct, a
+const or a module, is a declaration: queryable, and part of the audit below, but nothing observes
+it at run time. One flag per marked *name*, so two methods of a marked name share it.
+
+## It is also a check
+
+Julia already checks half of the surface question: `Docs.undocumented_names` has been public API
+in Base since 1.11, and `Aqua.test_undocumented_names` ships it as a test — every public name must
+carry a docstring. What neither can express is the third answer: *this name is public, it has no
+docstring, and that is deliberate, and here is why*.
 
 A marker nobody compares against anything is a claim. Put this in `runtests.jl` and it becomes a
 contract:
@@ -169,7 +201,7 @@ public, which is the module contradicting itself and needs no reference to be wr
 | type stability | DispatchDoctor | unrelated |
 | every public name has a docstring | `Docs.undocumented_names` (Base 1.11+), `Aqua.test_undocumented_names` | the same check, plus a third answer |
 | generating documentation | Documenter | only ever checks whether prose exists |
-| run-time behaviour | — | calls are untouched |
+| run-time behaviour | — | one short-circuit read in the body; see the table above |
 
 ## What the audit cannot see
 

docs/make.jl

@@ -16,6 +16,7 @@ makedocs(;
     pages=[
         "Home" => "index.md",
         "Declaring" => "declaring.md",
+        "Observing" => "observing.md",
         "Checking" => "checking.md",
         "Release decisions" => "releases.md",
         "Adopting it" => "adopting.md",

docs/src/index.md

@@ -4,46 +4,63 @@ CurrentModule = ExperimentalAPI
 
 # ExperimentalAPI.jl
 
-`public` says who may call a name. Nothing says whether the name is finished.
+`public` says who may call a name. Nothing says whether the name is *finished* — and nothing tells
+you, after a twelve-hour run, that the number you are about to publish came out of code its author
+had not validated.
 
-Julia already checks half of that. `Docs.undocumented_names` has been public API in Base since
-1.11, and `Aqua.test_undocumented_names` ships it as a test: every public name must carry a
-docstring. What neither can express is the **third option** —
-
-!!! note ""
-    this name is public, it has no docstring, and that is deliberate: the shape is not settled,
-    and here is why.
-
-ExperimentalAPI adds that option at the definition site, and makes it something a tool reads
-rather than prose a human might happen to notice.
+`@experimental` is that mark, written at the definition site where the author is, and readable by
+a machine so the answer arrives without anyone remembering to ask for it.
 
 ```julia
 using ExperimentalAPI
 
-@experimental "reads Test's internal result tree; not dogfooded in CI yet" \
-function render_test_report(records)
-    # ...
-end
+@experimental "convergence not established below β ≈ 0.1" energy(m::Model) = m.β * correction(m)
 ```
 
-```julia
-julia> ExperimentalAPI.audit(MyPackage)
-Public surface of MyPackage — 46 names
-  documented      45
-  experimental     1
-  unaccounted      0
+```console
+$ julia sweep.jl
+… your output …
+┌ ExperimentalAPI: this run entered 1 experimental definition
+│   MyModel.energy — convergence not established below β ≈ 0.1
+└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this
 ```
 
-## Three things, and only the third is a reason to have this
+Nobody asked for that summary. It is on by default, it carries the **reason** rather than just the
+symbol, and a marked definition the run never entered is *absent* — not reported with a count of
+zero. The same answer is available programmatically through [`entered`](@ref).
+
+## Three things, and the first is the reason to have this
 
 | | |
 |---|---|
+| an **observation** | which marked definitions the run went through — [`entered`](@ref), and the summary at exit |
 | a **declaration** | the reason travels with the name, in the source, where the author is — [`@experimental`](@ref) |
-| a **query** | a tool asks the module instead of reading prose — [`experimental`](@ref), [`stable`](@ref) |
 | a **check** | every public name is accounted for, or the test fails — [`audit`](@ref), [`test_surface`](@ref) |
 
-A marker nobody compares against anything is a claim. A marker something compares against the
-public surface is a contract. Everything in the first two rows exists to make the third possible.
+A docstring can carry the second row. Nothing a human writes can carry the first: the question is
+not "is this name experimental" but "did *this run* go through one", and it has to be answered
+after the run, about the run.
+
+## What it costs
+
+One short-circuit read in the body, and a write on the first call:
+
+| emitted into the body | 1 thread | 8 threads |
+|---|---|---|
+| nothing | 1.00× | 1.00× |
+| **the flag `@experimental` emits** | 1.03× | **0.985×** |
+| a counter, plain shared `Ref` | 1.03× | 3.76× — and loses 40% of its increments to races |
+| a counter, global atomic | 1.17× | 4.87× |
+| `@warn`, guarded so it fires once | 5.65× | — |
+
+10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. The
+flag is written once and only read afterwards, so it stops dirtying the cache line — which is why
+it is free at eight threads while every counting scheme is not, and why the notice is a summary at
+exit rather than a warning at the call.
+
+Only a definition **with a body** carries a flag; see [`@experimental`](@ref) for the
+form-by-form table. Counting, call sites and paths are a separate, opt-in layer that is not built
+yet.
 
 ## Why this is a separate axis
 
@@ -80,10 +97,11 @@ was never made public, which is the module contradicting itself.
 | type stability | DispatchDoctor | unrelated |
 | every public name has a docstring | `Docs.undocumented_names`, `Aqua.test_undocumented_names` | the same check, plus a third answer |
 | generating documentation | Documenter | only ever checks whether prose exists |
-| run-time behaviour | — | calls are untouched |
+| run-time behaviour | — | one short-circuit read in the body — see the table above |
+| how often a path ran | `Profile`, `@time` | not answered: the default layer knows *whether*, never how often |
 
-`@experimental` emits your definition unchanged plus one `push!` at load time. It does not wrap
-the call, does not add a method, and does not change dispatch.
+`@experimental` adds one statement to the body and nothing else. It does not add a method, does
+not change dispatch, and never allocates or logs.
 
 ## Install
 

docs/src/observing.md

@@ -0,0 +1,88 @@
+```@meta
+CurrentModule = ExperimentalAPI
+```
+
+# Observing
+
+A docstring can say a name is unfinished. It cannot tell you that *this run* went through it.
+
+That is the question behind the mark: not "is `energy` experimental", which the author already
+knows, but "did the number in this figure come out of code nobody validated" — asked after the
+run, about the run, by somebody who may not have written either.
+
+## Without asking
+
+```julia
+using ExperimentalAPI
+
+@experimental "convergence not established below β ≈ 0.1" energy(β) = β * 1.0000001
+
+energy(0.5)
+```
+
+```console
+$ julia sweep.jl
+┌ ExperimentalAPI: this run entered 1 experimental definition
+│   Main.energy — convergence not established below β ≈ 0.1
+└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this
+```
+
+Three properties, each of them a decision:
+
+  * **On by default.** The user who never asks is the one who needs telling. [`detecting`](@ref)
+    reports whether the hook is armed; the environment variable has to be set *before*
+    `using ExperimentalAPI`, because that is when `atexit` is registered.
+  * **Silent unless something was entered.** Loading a package that *has* marks prints nothing.
+    A package that cannot be quiet is one people vendor around.
+  * **Carries the reason.** The name says which line to open; the reason says whether the result
+    is affected.
+
+## As data
+
+[`entered`](@ref) returns the same thing the summary prints, as a `Vector{`[`Entry`](@ref)`}`:
+
+```julia
+julia> ExperimentalAPI.entered()
+1-element Vector{ExperimentalAPI.Entry}:
+ Entry(Main.energy, "convergence not established below β ≈ 0.1")
+```
+
+A marked definition the run never entered is **absent**, not reported with a count of zero — the
+difference between "observed" and "enumerated".
+
+[`marked_modules`](@ref) is the search this uses: the loaded modules that carry marks, found by
+walking rather than by a registry inside this package, because a table here would be written
+while the *marked* package is precompiled and so would be missing from its cache image.
+[`summary_text`](@ref) is what the exit hook prints, available as a string for a report of your
+own.
+
+## What it costs
+
+One short-circuit read in the body, and a write on the first call only.
+
+| emitted into the body | 1 thread | 8 threads | counts correctly? |
+|---|---|---|---|
+| nothing | 1.00× | 1.00× | — |
+| **the flag `@experimental` emits** | 1.03× | **0.985×** | yes |
+| a counter, plain shared `Ref` | 1.03× | 3.76× | **no** — 40% lost to races |
+| a counter, global atomic | 1.17× | 4.87× | yes |
+| a counter, per-thread atomic | 1.12× | 2.79× | yes |
+| `@warn`, guarded so it fires once | 5.65× | — | yes |
+| `@warn maxlog=1` | 59.57× | — | yes |
+
+10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2.
+
+Two of those rows decided the design. A flag written once and only read afterwards stops dirtying
+the cache line, which is why it is free at eight threads while every counting scheme is not. And
+the guarded `@warn` costs 5.65× *even though it fires once*: what stops the definition inlining is
+the call being in the body at all, not the warning being printed. That is why the notice is a
+summary at exit rather than a warning at the call.
+
+## What it does not answer
+
+  * **How often.** Presence only — [`Entry`](@ref)`.count` is always `nothing`. Counting is an
+    opt-in layer that is not built yet.
+  * **Which method.** Marks are name-keyed, so two methods of a marked name share one flag.
+  * **Which call site, or by what path.** Also the opt-in layer.
+  * **Anything about a declaration-only mark.** A name list, a `struct`, a `const`, a `module`:
+    recorded and audited, never observed. See [`@experimental`](@ref) for the table.