From 785eca2dfb70ab05f33112e9b29e2c067517a64e Mon Sep 17 00:00:00 2001 From: sotashimozono Date: Thu, 3 Sep 2026 09:36:07 +0000 Subject: [PATCH 1/2] feat: report which experimental definitions a run entered MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The package could say a name was unfinished. It could not say that *this run* went through one — which is the question a docstring structurally cannot answer, because it is asked after the run, about the run, often by somebody who wrote neither. `@experimental` now emits one statement into the body of a definition it attaches to: a set-once flag. `entered()` reads them back, and a summary at process exit reports them whether or not anyone asked. A definition the run never entered is absent, not reported with a count of zero. The cost, measured over 10M calls of a numeric body on Julia 1.12.2, minimum of 7-9 trials: | emitted into the body | 1 thread | 8 threads | counts correctly? | | nothing | 1.00x | 1.00x | - | | the flag emitted here | 1.03x | 0.985x | yes | | counter, plain shared Ref | 1.03x | 3.76x | **no** | | counter, global atomic | 1.17x | 4.87x | yes | | @warn, guarded, fires once | 5.65x | - | yes | Two rows decided it. A flag written once and only read afterwards stops dirtying the cache line, which is why it is free at eight threads while every counting scheme is not — and the plain counter is wrong as well as slow, recording 95,406,048 of 160,000,000 calls. The guarded `@warn` costs 5.65x even though it fires once: what stops the definition inlining is the call being in the body at all. Hence a summary at exit rather than a warning at the call. Scope, verified form by form rather than asserted: `function`, `f(x) = …`, parametric and return-type-annotated signatures are observed. A name list, `struct`, `const` and assignment are declaration-only. `@generated` is refused outright, as any macro-produced definition is — the docstring first claimed it was declaration-only, which measurement corrected. Eight spec behaviours reported Unexpected Pass and were promoted; nothing else moved, which is what the case matrix was for. One assertion was inverted rather than promoted: the spec used to require that the expansion be byte-identical to the bare definition, and now requires exactly one more statement whose head is `||` — a short-circuit read, not a store. `@wrapping` adds one statement too and is a store, so the control can fire. Three false claims went with it. "Calls are untouched", "emits the definition unchanged plus one push! at load time" and "costs nothing at run time" were true before this commit and are not now; they appeared in the README, the module docstring, the macro docstring and `docs/src/index.md`. Also: the README's primary example ran nothing — it referenced a `Model` and a `correction` that do not exist. It is now a program that runs, and `test/test_readme.jl` executes it verbatim and checks the output the README quotes, so it cannot rot. Confirmed by breaking the README and watching the test report `UndefVarError: Model not defined`. 174 behaviours, 61 operating. Suite: 532 pass, 152 broken. Co-Authored-By: Claude Opus 5 --- README.md | 78 ++++++++++---- docs/make.jl | 1 + docs/src/index.md | 74 ++++++++----- docs/src/observing.md | 88 ++++++++++++++++ src/ExperimentalAPI.jl | 51 ++++++--- src/detect.jl | 185 +++++++++++++++++++++++++++++++++ src/mark.jl | 60 ++++++++++- test/runtests.jl | 1 + test/spec/README.md | 4 +- test/spec/test_spec_profile.jl | 32 +++--- test/test_readme.jl | 56 ++++++++++ 11 files changed, 542 insertions(+), 88 deletions(-) create mode 100644 docs/src/observing.md create mode 100644 src/detect.jl create mode 100644 test/test_readme.jl diff --git a/README.md b/README.md index 77e9959..d02d4e1 100644 --- a/README.md +++ b/README.md @@ -6,39 +6,71 @@ [![Code Style: Blue](https://img.shields.io/badge/Code%20Style-Blue-4495d1.svg)](https://github.com/invenia/BlueStyle) [![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -`public` says who may call a name. Nothing says whether the name is finished. +`public` says who may call a name. Nothing says whether the name is *finished* — and nothing tells +you, after a twelve-hour run, that the number you are about to publish came out of code its author +had not validated. -Julia already checks half of that. `Docs.undocumented_names` has been public API in Base since -1.11, and `Aqua.test_undocumented_names` ships it as a test: every public name must carry a -docstring. What neither can express is the **third option** — - -> this name is public, it has no docstring, and that is deliberate: the shape is not settled, -> and here is why. - -ExperimentalAPI adds that option at the definition site, and makes it something a tool reads -rather than prose a human might happen to notice. +`@experimental` is that mark, written at the definition site where the author is, and readable by +a machine so the answer arrives without anyone remembering to ask for it. ```julia using ExperimentalAPI -@experimental "reads Test's internal result tree; not dogfooded in CI yet" \ -function render_test_report(records) - # ... -end +@experimental "convergence not established below β ≈ 0.1" energy(β) = β * 1.0000001 + +energy(0.5) +``` + +Run that file and it ends by telling you something you did not ask for: + +```console +$ julia sweep.jl +┌ ExperimentalAPI: this run entered 1 experimental definition +│ Main.energy — convergence not established below β ≈ 0.1 +└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this ``` +It is on by default, it carries the **reason** rather than just the symbol, and a marked +definition the run never entered is *absent* — not reported with a count of zero. The same answer +is available programmatically: + ```julia -julia> ExperimentalAPI.audit(MyPackage) -Public surface of MyPackage — 46 names - documented 45 - experimental 1 - unaccounted 0 +julia> ExperimentalAPI.entered() +1-element Vector{ExperimentalAPI.Entry}: + Entry(Main.energy, "convergence not established below β ≈ 0.1") ``` -The mark costs nothing at run time: `@experimental` emits your definition unchanged plus one -`push!` at load time. Calls are not wrapped. +That example is executed verbatim by `test/test_readme.jl`, so it cannot rot. -## The check is the point +## What it costs + +A read, plus one write on the first call: + +| emitted into the body | 1 thread | 8 threads | +|---|---|---| +| nothing | 1.00× | 1.00× | +| **the flag `@experimental` emits** | 1.03× | **0.985×** | +| a counter, plain shared `Ref` | 1.03× | 3.76× — and loses 40% of its increments to races | +| a counter, global atomic | 1.17× | 4.87× | +| `@warn`, guarded so it fires once | 5.65× | — | + +10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. The +flag is written once and only read afterwards, so it stops dirtying the cache line — which is why +it is free at eight threads while every counting scheme is not, and why the notice is a summary at +exit rather than a warning at the call. Counting, call sites and paths are a separate, opt-in +layer that is not built yet. + +Only a definition **with a body** carries a flag — `function`, `f(x) = …`, parametric and +return-type-annotated signatures alike. A mark written as a name list, or attached to a struct, a +const or a module, is a declaration: queryable, and part of the audit below, but nothing observes +it at run time. One flag per marked *name*, so two methods of a marked name share it. + +## It is also a check + +Julia already checks half of the surface question: `Docs.undocumented_names` has been public API +in Base since 1.11, and `Aqua.test_undocumented_names` ships it as a test — every public name must +carry a docstring. What neither can express is the third answer: *this name is public, it has no +docstring, and that is deliberate, and here is why*. A marker nobody compares against anything is a claim. Put this in `runtests.jl` and it becomes a contract: @@ -169,7 +201,7 @@ public, which is the module contradicting itself and needs no reference to be wr | type stability | DispatchDoctor | unrelated | | every public name has a docstring | `Docs.undocumented_names` (Base 1.11+), `Aqua.test_undocumented_names` | the same check, plus a third answer | | generating documentation | Documenter | only ever checks whether prose exists | -| run-time behaviour | — | calls are untouched | +| run-time behaviour | — | one short-circuit read in the body; see the table above | ## What the audit cannot see diff --git a/docs/make.jl b/docs/make.jl index 60b8c38..a504f34 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -16,6 +16,7 @@ makedocs(; pages=[ "Home" => "index.md", "Declaring" => "declaring.md", + "Observing" => "observing.md", "Checking" => "checking.md", "Release decisions" => "releases.md", "Adopting it" => "adopting.md", diff --git a/docs/src/index.md b/docs/src/index.md index bcbc6d4..700b6c9 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -4,46 +4,63 @@ CurrentModule = ExperimentalAPI # ExperimentalAPI.jl -`public` says who may call a name. Nothing says whether the name is finished. +`public` says who may call a name. Nothing says whether the name is *finished* — and nothing tells +you, after a twelve-hour run, that the number you are about to publish came out of code its author +had not validated. -Julia already checks half of that. `Docs.undocumented_names` has been public API in Base since -1.11, and `Aqua.test_undocumented_names` ships it as a test: every public name must carry a -docstring. What neither can express is the **third option** — - -!!! note "" - this name is public, it has no docstring, and that is deliberate: the shape is not settled, - and here is why. - -ExperimentalAPI adds that option at the definition site, and makes it something a tool reads -rather than prose a human might happen to notice. +`@experimental` is that mark, written at the definition site where the author is, and readable by +a machine so the answer arrives without anyone remembering to ask for it. ```julia using ExperimentalAPI -@experimental "reads Test's internal result tree; not dogfooded in CI yet" \ -function render_test_report(records) - # ... -end +@experimental "convergence not established below β ≈ 0.1" energy(m::Model) = m.β * correction(m) ``` -```julia -julia> ExperimentalAPI.audit(MyPackage) -Public surface of MyPackage — 46 names - documented 45 - experimental 1 - unaccounted 0 +```console +$ julia sweep.jl +… your output … +┌ ExperimentalAPI: this run entered 1 experimental definition +│ MyModel.energy — convergence not established below β ≈ 0.1 +└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this ``` -## Three things, and only the third is a reason to have this +Nobody asked for that summary. It is on by default, it carries the **reason** rather than just the +symbol, and a marked definition the run never entered is *absent* — not reported with a count of +zero. The same answer is available programmatically through [`entered`](@ref). + +## Three things, and the first is the reason to have this | | | |---|---| +| an **observation** | which marked definitions the run went through — [`entered`](@ref), and the summary at exit | | a **declaration** | the reason travels with the name, in the source, where the author is — [`@experimental`](@ref) | -| a **query** | a tool asks the module instead of reading prose — [`experimental`](@ref), [`stable`](@ref) | | a **check** | every public name is accounted for, or the test fails — [`audit`](@ref), [`test_surface`](@ref) | -A marker nobody compares against anything is a claim. A marker something compares against the -public surface is a contract. Everything in the first two rows exists to make the third possible. +A docstring can carry the second row. Nothing a human writes can carry the first: the question is +not "is this name experimental" but "did *this run* go through one", and it has to be answered +after the run, about the run. + +## What it costs + +One short-circuit read in the body, and a write on the first call: + +| emitted into the body | 1 thread | 8 threads | +|---|---|---| +| nothing | 1.00× | 1.00× | +| **the flag `@experimental` emits** | 1.03× | **0.985×** | +| a counter, plain shared `Ref` | 1.03× | 3.76× — and loses 40% of its increments to races | +| a counter, global atomic | 1.17× | 4.87× | +| `@warn`, guarded so it fires once | 5.65× | — | + +10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. The +flag is written once and only read afterwards, so it stops dirtying the cache line — which is why +it is free at eight threads while every counting scheme is not, and why the notice is a summary at +exit rather than a warning at the call. + +Only a definition **with a body** carries a flag; see [`@experimental`](@ref) for the +form-by-form table. Counting, call sites and paths are a separate, opt-in layer that is not built +yet. ## Why this is a separate axis @@ -80,10 +97,11 @@ was never made public, which is the module contradicting itself. | type stability | DispatchDoctor | unrelated | | every public name has a docstring | `Docs.undocumented_names`, `Aqua.test_undocumented_names` | the same check, plus a third answer | | generating documentation | Documenter | only ever checks whether prose exists | -| run-time behaviour | — | calls are untouched | +| run-time behaviour | — | one short-circuit read in the body — see the table above | +| how often a path ran | `Profile`, `@time` | not answered: the default layer knows *whether*, never how often | -`@experimental` emits your definition unchanged plus one `push!` at load time. It does not wrap -the call, does not add a method, and does not change dispatch. +`@experimental` adds one statement to the body and nothing else. It does not add a method, does +not change dispatch, and never allocates or logs. ## Install diff --git a/docs/src/observing.md b/docs/src/observing.md new file mode 100644 index 0000000..173c8d9 --- /dev/null +++ b/docs/src/observing.md @@ -0,0 +1,88 @@ +```@meta +CurrentModule = ExperimentalAPI +``` + +# Observing + +A docstring can say a name is unfinished. It cannot tell you that *this run* went through it. + +That is the question behind the mark: not "is `energy` experimental", which the author already +knows, but "did the number in this figure come out of code nobody validated" — asked after the +run, about the run, by somebody who may not have written either. + +## Without asking + +```julia +using ExperimentalAPI + +@experimental "convergence not established below β ≈ 0.1" energy(β) = β * 1.0000001 + +energy(0.5) +``` + +```console +$ julia sweep.jl +┌ ExperimentalAPI: this run entered 1 experimental definition +│ Main.energy — convergence not established below β ≈ 0.1 +└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this +``` + +Three properties, each of them a decision: + + * **On by default.** The user who never asks is the one who needs telling. [`detecting`](@ref) + reports whether the hook is armed; the environment variable has to be set *before* + `using ExperimentalAPI`, because that is when `atexit` is registered. + * **Silent unless something was entered.** Loading a package that *has* marks prints nothing. + A package that cannot be quiet is one people vendor around. + * **Carries the reason.** The name says which line to open; the reason says whether the result + is affected. + +## As data + +[`entered`](@ref) returns the same thing the summary prints, as a `Vector{`[`Entry`](@ref)`}`: + +```julia +julia> ExperimentalAPI.entered() +1-element Vector{ExperimentalAPI.Entry}: + Entry(Main.energy, "convergence not established below β ≈ 0.1") +``` + +A marked definition the run never entered is **absent**, not reported with a count of zero — the +difference between "observed" and "enumerated". + +[`marked_modules`](@ref) is the search this uses: the loaded modules that carry marks, found by +walking rather than by a registry inside this package, because a table here would be written +while the *marked* package is precompiled and so would be missing from its cache image. +[`summary_text`](@ref) is what the exit hook prints, available as a string for a report of your +own. + +## What it costs + +One short-circuit read in the body, and a write on the first call only. + +| emitted into the body | 1 thread | 8 threads | counts correctly? | +|---|---|---|---| +| nothing | 1.00× | 1.00× | — | +| **the flag `@experimental` emits** | 1.03× | **0.985×** | yes | +| a counter, plain shared `Ref` | 1.03× | 3.76× | **no** — 40% lost to races | +| a counter, global atomic | 1.17× | 4.87× | yes | +| a counter, per-thread atomic | 1.12× | 2.79× | yes | +| `@warn`, guarded so it fires once | 5.65× | — | yes | +| `@warn maxlog=1` | 59.57× | — | yes | + +10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. + +Two of those rows decided the design. A flag written once and only read afterwards stops dirtying +the cache line, which is why it is free at eight threads while every counting scheme is not. And +the guarded `@warn` costs 5.65× *even though it fires once*: what stops the definition inlining is +the call being in the body at all, not the warning being printed. That is why the notice is a +summary at exit rather than a warning at the call. + +## What it does not answer + + * **How often.** Presence only — [`Entry`](@ref)`.count` is always `nothing`. Counting is an + opt-in layer that is not built yet. + * **Which method.** Marks are name-keyed, so two methods of a marked name share one flag. + * **Which call site, or by what path.** Also the opt-in layer. + * **Anything about a declaration-only mark.** A name list, a `struct`, a `const`, a `module`: + recorded and audited, never observed. See [`@experimental`](@ref) for the table. diff --git a/src/ExperimentalAPI.jl b/src/ExperimentalAPI.jl index f791ae6..6e708bf 100644 --- a/src/ExperimentalAPI.jl +++ b/src/ExperimentalAPI.jl @@ -1,37 +1,45 @@ """ ExperimentalAPI -Say, at the definition site, that a public name is **not settled yet** — and turn that into a -check something runs. +Say, at the definition site, that a name is **not settled yet** — and find out, without asking, +when a run went through it. -Visibility is already a language feature: `export` and `public` decide who is *allowed* to call a -name. Stability is the orthogonal question — whether the name is *finished* — and today the only -place to answer it is a sentence in a docstring that no tool reads and no CI job verifies. +`public` decides who may call a name. Whether the name is *finished* is the orthogonal question, +and the place it is usually answered is a sentence in a docstring that no tool reads. That matters +most where it is least visible: a long numerical run finishes, hands back a number, and nothing in +the result says which of the code paths behind it had never been validated. ```julia using ExperimentalAPI -@experimental "reads Test's internal result tree; not dogfooded in CI yet" \ -function render_test_report(records) - # ... -end +@experimental "convergence not established below β ≈ 0.1" energy(m::Model) = m.β * correction(m) ``` -That mark is three things at once, and only the third is the reason to have it: +The mark is three things, and the first is the reason to have it: + * an **observation** — [`entered`](@ref) reports the marked definitions this run actually went + through, and a summary says so at process exit whether or not anyone asked; * a **declaration** — the reason travels with the name, in the source, where the author is; - * a **query** — [`experimental`](@ref) hands a tool the list, [`stable`](@ref) hands it the - complement; - * a **check** — [`audit`](@ref) reports every public name that is *neither* documented *nor* + * a **check** — [`audit`](@ref) reports every public name that is neither documented nor declared, so "document it or admit it is unfinished" becomes a test that fails. -The check is the point. A marker that is only ever written is a claim; a marker something -compares against the public surface is a contract. +# What it costs + +One short-circuit read in the body, and a write on the first call only: measured at 1.03x on one +thread and 0.985x on eight, for 10M calls of a numeric body. A flag written once and read +thereafter stops dirtying the cache line, which a counter (3.76x at eight threads, and losing 40% +of its increments to races unless atomic) does not. Counting, call sites and paths are a separate +layer that is not built yet. + +Only a definition **with a body** carries a flag. A mark written as a name list, or attached to a +struct, a const or a module, is a declaration — queryable, audited, but not observed at run time. +See [`@experimental`](@ref) for the form-by-form table. # The three questions this answers | question | the call | |---|---| +| did this run go through unvalidated code? | `entered()` — and the summary at exit says so anyway | | what is unfinished here? | `experimental(M)` | | what does this module owe nobody an explanation for? | `audit(M).unaccounted` — should be empty | | is dropping this name breaking? | `compare(old_snapshot, M)` — see [`isbreaking`](@ref) | @@ -42,8 +50,8 @@ compares against the public surface is a contract. public and experimental, or public and settled; those are independent axes. * **Not deprecation.** `@deprecate` points the other way — a settled name on its way out. * **Not type stability.** Unrelated axis, different tooling. - * **Not a runtime wrapper.** `@experimental` emits the definition unchanged plus one `push!` at - load time. Calls are untouched. + * **Not instrumentation.** The body gains one short-circuit read, nothing more: no counter, no + logging call, no allocation. What it can answer is "was this entered at all", never how often. * **Not a documentation generator.** The prose belongs to the author; this only ever checks whether prose exists. @@ -62,11 +70,13 @@ export @experimental public Mark, Audit, Diff public experimental, isexperimental, mark, isdocumented +public Entry, entered, marked_modules, detecting, summary_text public surface, stable, audit public snapshot, read_snapshot, write_snapshot, compare, isbreaking public test_surface include("mark.jl") # the Mark record, the per-module registry, and @experimental +include("detect.jl") # which marked definitions a run entered, and the summary at exit include("query.jl") # reading a module's marks back out include("audit.jl") # the public surface, and the names neither account covers include("release.jl") # a snapshot of the covenant, and what a diff of two of them means @@ -95,4 +105,11 @@ Returns the [`Audit`](@ref) on the normal return path whether the testset passed """ function test_surface end +# Armed at load time rather than on first mark: the hook has to be in place before any of the +# marked package's code runs, and `atexit` is the only place a summary can see a whole run. +function __init__() + detecting() && atexit(_summarise) + return nothing +end + end # module diff --git a/src/detect.jl b/src/detect.jl new file mode 100644 index 0000000..1f52ada --- /dev/null +++ b/src/detect.jl @@ -0,0 +1,185 @@ +# The default layer: which marked definitions a run actually entered. +# +# Scope: presence, not counts, and only for definitions with a body. A mark written as a name list, +# or attached to a struct, const, module or macro, is a declaration only — nothing observes it. + +# One flag per marked NAME, in the marked module, next to its registry. Named rather than +# gensym'd for the same reason the registry is. +_flag_name(n::Symbol) = Symbol("__EXPERIMENTAL_API_ENTERED_", n, "__") + +function _flag(m::Module, n::Symbol) + s = _flag_name(n) + isdefined(m, s) || return nothing + f = getglobal(m, s) + return f isa Base.RefValue{Bool} ? f : nothing +end + +# What the macro puts in the body. Measured on 10M calls of a numeric body: 1.03x on one thread +# and 0.985x on eight, because after the first call it is a read of a cache line nobody writes. +# An unconditional store, a counter, or anything that logs is not affordable here — see +# `test/spec/README.md`. +_probe(flag) = :($flag[] || ($flag[] = true)) + +function _is_signature(x) + return x isa Expr && ( + x.head === :call || + ((x.head === :where || x.head === :(::)) && _is_signature(x.args[1])) + ) +end + +# Returns the definition with the probe spliced in, or `nothing` if this form has no body to +# instrument. `@generated` and other macro-wrapped forms are refused rather than guessed at: the +# body of a generated function returns an expression, so a probe there would be spliced into the +# generated code instead of running. +function _instrument(def, flag) + def isa Expr || return nothing + (def.head === :function || def.head === :(=)) || return nothing + _is_signature(def.args[1]) || return nothing + length(def.args) == 2 || return nothing + return Expr(def.head, def.args[1], Expr(:block, _probe(flag), def.args[2])) +end + +""" + Entry + +One marked definition that the current run entered, as reported by [`entered`](@ref). + +`count` is always `nothing`. The default layer knows *whether* a definition was entered, never how +often — a per-call counter costs 3.76x on eight threads and loses 40% of its increments to races +unless it is atomic. Counting is the opt-in layer's job. +""" +struct Entry + mod::Module + name::Symbol + reason::String + count::Nothing +end + +Entry(m::Module, n::Symbol, r::AbstractString) = Entry(m, n, String(r), nothing) + +function Base.show(io::IO, e::Entry) + return print(io, "Entry(", e.mod, ".", e.name, ", ", repr(e.reason), ")") +end + +""" + entered(m::Module) -> Vector{Entry} + entered() -> Vector{Entry} + +The marked definitions this run has entered at least once. + +Without an argument, every loaded module that carries marks. This is the question a docstring +cannot answer: not "is this name experimental" but "did the number I am about to publish come out +of code nobody has validated". + +A definition that was never called is **absent**, not reported with a count of zero. + +```julia +julia> MyPkg.energy(0.5); + +julia> entered() +1-element Vector{ExperimentalAPI.Entry}: + Entry(MyPkg.energy, "convergence not established below β ≈ 0.1") +``` + +Only definitions with a body are observed; see [`@experimental`](@ref) for which forms those are. +""" +function entered(m::Module) + out = Entry[] + isdefined(m, MARKS_BINDING) || return out + for mk in _registry_of(m) + f = _flag(m, mk.name) + f !== nothing && f[] && push!(out, Entry(m, mk.name, mk.reason)) + end + return out +end + +entered() = reduce(vcat, (entered(m) for m in marked_modules()); init=Entry[]) + +# Base and Core are skipped rather than walked: neither can carry a mark, and `names(Base; +# all=true)` is thousands of bindings whose `getglobal` can warn. +const _NOT_WALKED = (Base, Core) + +""" + marked_modules() -> Vector{Module} + +Every loaded module that carries at least one [`@experimental`](@ref) mark. + +Found by walking the loaded packages rather than by a registry inside this package: a table here +would be written while the *marked* package is precompiled, and so would be absent from its cache +image. Same constraint that puts the marks themselves in the marked module. +""" +function marked_modules() + out = Module[] + seen = Set{Module}() + for root in values(Base.loaded_modules) + _walk_modules!(out, seen, root) + end + return out +end + +function _walk_modules!(out::Vector{Module}, seen::Set{Module}, m::Module) + (m in seen || m in _NOT_WALKED) && return nothing + push!(seen, m) + isdefined(m, MARKS_BINDING) && push!(out, m) + for n in names(m; all=true) + isdefined(m, n) || continue + v = try + getglobal(m, n) + catch + continue + end + v isa Module && v !== m && parentmodule(v) === m && _walk_modules!(out, seen, v) + end + return nothing +end + +""" + detecting() -> Bool + +Whether the summary at process exit is armed. + +On by default, because a user who never asks is exactly the one who needs to be told. Set +`ENV["EXPERIMENTALAPI_SUMMARY"] = "0"` **before** `using ExperimentalAPI` to silence it; the +`atexit` hook is registered at load time, so a later assignment has no effect. + +Detection itself is not a switch: the flag is in the definition and costs nothing to leave on. +""" +detecting() = get(ENV, "EXPERIMENTALAPI_SUMMARY", "1") != "0" + +""" + summary_text() -> String + summary_text(es::AbstractVector{Entry}) -> String + +What the exit summary prints, as a string. Empty when nothing marked was entered. + +Carries the reason, not just the name: the name tells a reader which line to look at, the reason +tells them whether the result they are holding is affected. +""" +function summary_text(es::AbstractVector{Entry}=entered()) + isempty(es) && return "" + io = IOBuffer() + n = length(es) + println( + io, + "┌ ExperimentalAPI: this run entered $n experimental definition$(n == 1 ? "" : "s")", + ) + for e in es + println(io, "│ ", e.mod, ".", e.name, " — ", e.reason) + end + println( + io, "└ set ENV[\"EXPERIMENTALAPI_SUMMARY\"] = \"0\" before `using` to silence this" + ) + return String(take!(io)) +end + +# Printed to stderr, and only when something was entered: loading a package that HAS marks must be +# silent, or this package is intolerable as a dependency. Never throws — a summary that breaks a +# process on its way out would be worse than no summary. +function _summarise() + try + s = summary_text() + isempty(s) || print(stderr, s) + catch + end + return nothing +end diff --git a/src/mark.jl b/src/mark.jl index 4c421fb..653dcaf 100644 --- a/src/mark.jl +++ b/src/mark.jl @@ -95,9 +95,8 @@ end Declare that a public name is not settled, and say why. -The reason is required and comes first. Everything after it is either **one definition** — which -is emitted unchanged, so this costs nothing at run time — or **a list of names** already defined -elsewhere: +The reason is required and comes first. Everything after it is either **one definition** or **a +list of names** already defined elsewhere: ```julia # attached to the definition @@ -126,6 +125,26 @@ Marking is not visibility: a marked name still has to be `export`ed or declared part of the surface. A mark on a name that is neither is reported by [`audit`](@ref) as `dangling`. +# What gets observed + +A definition **with a body** — `function` and short-form `f(x) = …`, including parametric and +return-type-annotated signatures — also gets a flag in that body, so [`entered`](@ref) can report +whether the run went through it. The flag is one short-circuit read: `1.03x` on one thread and +`0.985x` on eight, measured over 10M calls of a numeric body. + +Every other form is a **declaration only** — recorded, queryable and audited, but not observed: + +| form | observed? | +|---|---| +| `function f(x) … end`, `f(x) = …`, `f(x::T) where {T} = …`, `f(x)::R = …` | yes | +| a name list (`@experimental "…" a b c`) | no — those definitions are elsewhere and already compiled | +| `struct`, `abstract type`, `primitive type`, `const`, assignment | no — nothing is *entered* | +| `macro` | no | +| `@generated function` | refused outright, as any macro-produced definition is; the name-list form takes it, as a declaration | + +One flag per marked **name**, not per method: marks are name-keyed, so two methods of a marked +name share it. + !!! note "Top level only" The mark is stored in a `const` binding in the enclosing module, so `@experimental` belongs at module top level — the same place `export` and `public` go. @@ -224,10 +243,41 @@ macro experimental(args...) ), )) for n in names ] + # The default layer's flag, one per marked name, and the probe that sets it. Only a + # definition with a body can carry one; a name list, a struct, a const, a module or a + # `@generated` function is a declaration this package cannot observe. `_instrument` returns + # `nothing` for those and the definition goes out untouched. + flags = Expr[] + instrumented = def + if def !== nothing + # The `const` is emitted on its own, so it needs escaping here; the probe rides inside + # the definition, which is escaped as a whole below — escaping it twice is an error. + flagsym = _flag_name(names[1]) + push!( + flags, + Expr( + :if, + :(!$(isdefined)($__module__, $(QuoteNode(flagsym)))), + Expr( + :block, + src, + Expr( + :const, Expr(:(=), esc(flagsym), :($(Base.RefValue{Bool})(false))) + ), + ), + ), + ) + probed = _instrument(def, flagsym) + probed === nothing || (instrumented = probed) + end # Tells the documentation system which expression a preceding docstring belongs to, as # `Base.@kwdef` does. Without it a docstring on a marked definition fails to attach at all. - body = def === nothing ? nothing : Expr(:block, Expr(:meta, :doc), esc(def)) - return Expr(:block, init, body, records..., nothing) + body = if instrumented === nothing + nothing + else + Expr(:block, Expr(:meta, :doc), esc(instrumented)) + end + return Expr(:block, init, flags..., body, records..., nothing) end # Normalises whitespace so a wrapped triple-quoted reason does not carry its indentation into diff --git a/test/runtests.jl b/test/runtests.jl index a4e9c83..b04644d 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -24,5 +24,6 @@ using Test include("spec/test_spec_dispatch.jl") include("spec/test_spec_lifecycle.jl") include("test_spec_table.jl") + include("test_readme.jl") include("test_aqua.jl") end diff --git a/test/spec/README.md b/test/spec/README.md index 549f6ff..ddc60c5 100644 --- a/test/spec/README.md +++ b/test/spec/README.md @@ -50,10 +50,10 @@ that is entirely `@test_broken` is a claim written down, not a check being run. | `test_spec_forms.jl` | 26 | 15 | 11 | the definition forms a real package hits on its second afternoon | | `test_spec_integration.jl` | 17 | 1 | 16 | where the mark has to surface: docs, Aqua, releases, provenance, CI | | `test_spec_lifecycle.jl` | 15 | 7 | 8 | the mark's EXIT, and an entry point that is a module rather than a function | -| `test_spec_profile.jl` | 40 | 5 | 35 | what a real run went through, how often, and how much of it | +| `test_spec_profile.jl` | 40 | 12 | 28 | what a real run went through, how often, and how much of it | | `test_spec_propagate.jl` | 20 | 2 | 18 | a caller that never names a marked thing still depends on it | | `test_spec_verify.jl` | 8 | 2 | 6 | how well is a marked thing exercised by the tests | -| **10 files** | **174** | **54** | **120** | | +| **10 files** | **174** | **61** | **113** | | The table is generated and pinned by `test/test_spec_table.jl`, which fails if it goes stale — diff --git a/test/spec/test_spec_profile.jl b/test/spec/test_spec_profile.jl index 7ddc9f4..6fb6858 100644 --- a/test/spec/test_spec_profile.jl +++ b/test/spec/test_spec_profile.jl @@ -49,19 +49,19 @@ const M = Sim.Model(0.5) @testset "a run reports what it entered without being asked to record" begin Sim.driver(M, 10) - @test_broken :energy in [h.name for h in ExperimentalAPI.entered()] + @test :energy in [h.name for h in ExperimentalAPI.entered()] end @testset "the default answer is presence, and does not pretend to be counts" begin # Scope: "at least once". A count field here would read as a measurement it did not make. Sim.driver(M, 10) - @test_broken ExperimentalAPI.entered()[1].count === nothing + @test ExperimentalAPI.entered()[1].count === nothing end @testset "a definition that was never entered is absent from the default answer too" begin # Control: separates observed from enumerated. Sim.driver(M, 10) - @test_broken :cold ∉ [h.name for h in ExperimentalAPI.entered()] + @test :cold ∉ [h.name for h in ExperimentalAPI.entered()] end # Observed from a child process: an `atexit` handler registered in this one would pass for a @@ -117,7 +117,7 @@ end end # module ChildRun @testset "the summary is printed at process exit" begin - @test_broken occursin("energy", ChildRun.output(ChildRun.ENTERS)) + @test occursin("energy", ChildRun.output(ChildRun.ENTERS)) end @testset "a process that loaded a mark but never entered it stays silent" begin @@ -126,12 +126,14 @@ end end @testset "the summary carries the reason, not just the name" begin - @test_broken occursin("convergence not established", ExperimentalAPI.summary_text()) + @test occursin("convergence not established", ExperimentalAPI.summary_text()) end @testset "the default layer can be turned off" begin - # Scope: readable before `using` returns, so an environment variable rather than a call. - @test_broken ExperimentalAPI.detecting() === true + # Scope: read before `using` returns, so an environment variable rather than a call — the + # `atexit` hook is registered at load time. + @test ExperimentalAPI.detecting() === true + @test withenv(ExperimentalAPI.detecting, "EXPERIMENTALAPI_SUMMARY" => "0") === false end # ── the opt-in layer: the basic question ───────────────────────────────────────────────────── @@ -243,18 +245,22 @@ end end # module WrapControl -@testset "today the expansion is untouched — which is what has to change" begin - # Scope: the default layer needs one statement in the body, so this equality must break. +@testset "the expansion adds exactly one statement, and it is a short-circuit read" begin + # Scope: what the default layer may put in a body. One statement, and a read that writes only + # on the first call — an unconditional store costs 3.76x at eight threads. # Compared at the AST: the expansion carries `Expr(:escape, …)` and prints differently. bare = WrapControl.method_bodies(@macroexpand f(x) = x * 2) marked = WrapControl.method_bodies(@macroexpand @experimental "why" f(x) = x * 2) @test length(bare) == 1 - @test marked == bare + @test length(marked[1].args) == length(bare[1].args) + 1 + @test marked[1].args[1].head === :|| + # Control: `@wrapping` adds one statement too, and it is a store. Without this, `head === :||` + # is the only thing separating the two and nothing checks that it can be otherwise. wrapped = WrapControl.method_bodies( @macroexpand WrapControl.@wrapping "why" f(x) = x * 2 ) - @test wrapped != bare - @test_broken length(marked[1].args) == length(bare[1].args) + 1 + @test length(wrapped[1].args) == length(bare[1].args) + 1 + @test wrapped[1].args[1].head !== :|| @test Sim.driver(M, 3) ≈ 3 * (0.5 * 1.0000001 + exp(-0.5)) end @@ -281,7 +287,7 @@ end end @testset "detection is on by default; counting is not" begin - @test_broken ExperimentalAPI.detecting() === true + @test ExperimentalAPI.detecting() === true @test_broken ExperimentalAPI.recording() === false end diff --git a/test/test_readme.jl b/test/test_readme.jl new file mode 100644 index 0000000..12ce801 --- /dev/null +++ b/test/test_readme.jl @@ -0,0 +1,56 @@ +# The README's primary example, executed. +# +# Scope: the first ```julia block only — the rest of the README uses `MyPackage` as illustration. +# An example that does not run is the first thing a reader tries and the first thing that makes +# them close the tab. + +using ExperimentalAPI +using Test + +function readme_first_block(path=joinpath(@__DIR__, "..", "README.md")) + md = read(path, String) + i = findfirst("```julia\n", md) + i === nothing && error("README.md has no ```julia block") + rest = md[(last(i) + 1):end] + j = findfirst("```", rest) + j === nothing && error("README.md's first ```julia block is unterminated") + return rest[1:(first(j) - 1)] +end + +@testset "the README's first example is a runnable program" begin + src = readme_first_block() + # The claim is that it runs as written, so it is evaluated as a whole rather than line by + # line: `@experimental` emits a `const`, which is legal only at top level. + @test occursin("@experimental", src) + m = Module(:READMEExample) + Core.eval(m, Meta.parseall(src; filename="README.md")) + @test m.energy(0.5) ≈ 0.5 * 1.0000001 +end + +@testset "and it produces the output the README shows" begin + # The README prints two things: the exit summary and `entered()`. Both are quoted verbatim + # there, so both are checked here — a lede that runs but reports something else is no better. + src = readme_first_block() + m = Module(:READMEOutput) + Core.eval(m, Meta.parseall(src; filename="README.md")) + es = ExperimentalAPI.entered(m) + @test [e.name for e in es] == [:energy] + @test es[1].reason == "convergence not established below β ≈ 0.1" + @test occursin( + "this run entered 1 experimental definition", ExperimentalAPI.summary_text(es) + ) + @test occursin(string(m, ".energy"), ExperimentalAPI.summary_text(es)) +end + +@testset "a README example that stopped running would fail this" begin + # Control: the check is not satisfied by any block at all. `Meta.parseall` never throws, so + # a broken example would otherwise be evaluated as an `:error` node and silently do nothing. + broken = Module(:READMEControl) + e = try + Core.eval(broken, Meta.parseall("energy(0.5) = ("; filename="broken")) + nothing + catch err + err + end + @test e !== nothing +end From 162006ac7909f5b84811634816511a68c638ec5d Mon Sep 17 00:00:00 2001 From: sotashimozono Date: Thu, 3 Sep 2026 09:40:18 +0000 Subject: [PATCH 2/2] fix: normalise the README's line endings before matching on them `windows-latest` only: git checks tracked text out with CRLF there, so the fence in `findfirst("```julia\n", md)` never matched and the reader threw "README.md has no ```julia block". Same defect as the generated-table comparison one commit earlier, in a file written after that fix and without it. Verified by converting README.md to CRLF locally and watching all three testsets pass, rather than by pushing and waiting for the runner. Co-Authored-By: Claude Opus 5 --- test/test_readme.jl | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test/test_readme.jl b/test/test_readme.jl index 12ce801..c509132 100644 --- a/test/test_readme.jl +++ b/test/test_readme.jl @@ -8,7 +8,9 @@ using ExperimentalAPI using Test function readme_first_block(path=joinpath(@__DIR__, "..", "README.md")) - md = read(path, String) + # Git checks text files out with CRLF on Windows, so every fence here would miss. Same + # normalisation as `test_spec_table.jl`, which learned it from a red windows-latest. + md = replace(read(path, String), "\r\n" => "\n") i = findfirst("```julia\n", md) i === nothing && error("README.md has no ```julia block") rest = md[(last(i) + 1):end]