diff --git a/README.md b/README.md index 77e9959..d02d4e1 100644 --- a/README.md +++ b/README.md @@ -6,39 +6,71 @@ [![Code Style: Blue](https://img.shields.io/badge/Code%20Style-Blue-4495d1.svg)](https://github.com/invenia/BlueStyle) [![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -`public` says who may call a name. Nothing says whether the name is finished. +`public` says who may call a name. Nothing says whether the name is *finished* — and nothing tells +you, after a twelve-hour run, that the number you are about to publish came out of code its author +had not validated. -Julia already checks half of that. `Docs.undocumented_names` has been public API in Base since -1.11, and `Aqua.test_undocumented_names` ships it as a test: every public name must carry a -docstring. What neither can express is the **third option** — - -> this name is public, it has no docstring, and that is deliberate: the shape is not settled, -> and here is why. - -ExperimentalAPI adds that option at the definition site, and makes it something a tool reads -rather than prose a human might happen to notice. +`@experimental` is that mark, written at the definition site where the author is, and readable by +a machine so the answer arrives without anyone remembering to ask for it. ```julia using ExperimentalAPI -@experimental "reads Test's internal result tree; not dogfooded in CI yet" \ -function render_test_report(records) - # ... -end +@experimental "convergence not established below β ≈ 0.1" energy(β) = β * 1.0000001 + +energy(0.5) +``` + +Run that file and it ends by telling you something you did not ask for: + +```console +$ julia sweep.jl +┌ ExperimentalAPI: this run entered 1 experimental definition +│ Main.energy — convergence not established below β ≈ 0.1 +└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this ``` +It is on by default, it carries the **reason** rather than just the symbol, and a marked +definition the run never entered is *absent* — not reported with a count of zero. The same answer +is available programmatically: + ```julia -julia> ExperimentalAPI.audit(MyPackage) -Public surface of MyPackage — 46 names - documented 45 - experimental 1 - unaccounted 0 +julia> ExperimentalAPI.entered() +1-element Vector{ExperimentalAPI.Entry}: + Entry(Main.energy, "convergence not established below β ≈ 0.1") ``` -The mark costs nothing at run time: `@experimental` emits your definition unchanged plus one -`push!` at load time. Calls are not wrapped. +That example is executed verbatim by `test/test_readme.jl`, so it cannot rot. -## The check is the point +## What it costs + +A read, plus one write on the first call: + +| emitted into the body | 1 thread | 8 threads | +|---|---|---| +| nothing | 1.00× | 1.00× | +| **the flag `@experimental` emits** | 1.03× | **0.985×** | +| a counter, plain shared `Ref` | 1.03× | 3.76× — and loses 40% of its increments to races | +| a counter, global atomic | 1.17× | 4.87× | +| `@warn`, guarded so it fires once | 5.65× | — | + +10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. The +flag is written once and only read afterwards, so it stops dirtying the cache line — which is why +it is free at eight threads while every counting scheme is not, and why the notice is a summary at +exit rather than a warning at the call. Counting, call sites and paths are a separate, opt-in +layer that is not built yet. + +Only a definition **with a body** carries a flag — `function`, `f(x) = …`, parametric and +return-type-annotated signatures alike. A mark written as a name list, or attached to a struct, a +const or a module, is a declaration: queryable, and part of the audit below, but nothing observes +it at run time. One flag per marked *name*, so two methods of a marked name share it. + +## It is also a check + +Julia already checks half of the surface question: `Docs.undocumented_names` has been public API +in Base since 1.11, and `Aqua.test_undocumented_names` ships it as a test — every public name must +carry a docstring. What neither can express is the third answer: *this name is public, it has no +docstring, and that is deliberate, and here is why*. A marker nobody compares against anything is a claim. Put this in `runtests.jl` and it becomes a contract: @@ -169,7 +201,7 @@ public, which is the module contradicting itself and needs no reference to be wr | type stability | DispatchDoctor | unrelated | | every public name has a docstring | `Docs.undocumented_names` (Base 1.11+), `Aqua.test_undocumented_names` | the same check, plus a third answer | | generating documentation | Documenter | only ever checks whether prose exists | -| run-time behaviour | — | calls are untouched | +| run-time behaviour | — | one short-circuit read in the body; see the table above | ## What the audit cannot see diff --git a/docs/make.jl b/docs/make.jl index 60b8c38..a504f34 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -16,6 +16,7 @@ makedocs(; pages=[ "Home" => "index.md", "Declaring" => "declaring.md", + "Observing" => "observing.md", "Checking" => "checking.md", "Release decisions" => "releases.md", "Adopting it" => "adopting.md", diff --git a/docs/src/index.md b/docs/src/index.md index bcbc6d4..700b6c9 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -4,46 +4,63 @@ CurrentModule = ExperimentalAPI # ExperimentalAPI.jl -`public` says who may call a name. Nothing says whether the name is finished. +`public` says who may call a name. Nothing says whether the name is *finished* — and nothing tells +you, after a twelve-hour run, that the number you are about to publish came out of code its author +had not validated. -Julia already checks half of that. `Docs.undocumented_names` has been public API in Base since -1.11, and `Aqua.test_undocumented_names` ships it as a test: every public name must carry a -docstring. What neither can express is the **third option** — - -!!! note "" - this name is public, it has no docstring, and that is deliberate: the shape is not settled, - and here is why. - -ExperimentalAPI adds that option at the definition site, and makes it something a tool reads -rather than prose a human might happen to notice. +`@experimental` is that mark, written at the definition site where the author is, and readable by +a machine so the answer arrives without anyone remembering to ask for it. ```julia using ExperimentalAPI -@experimental "reads Test's internal result tree; not dogfooded in CI yet" \ -function render_test_report(records) - # ... -end +@experimental "convergence not established below β ≈ 0.1" energy(m::Model) = m.β * correction(m) ``` -```julia -julia> ExperimentalAPI.audit(MyPackage) -Public surface of MyPackage — 46 names - documented 45 - experimental 1 - unaccounted 0 +```console +$ julia sweep.jl +… your output … +┌ ExperimentalAPI: this run entered 1 experimental definition +│ MyModel.energy — convergence not established below β ≈ 0.1 +└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this ``` -## Three things, and only the third is a reason to have this +Nobody asked for that summary. It is on by default, it carries the **reason** rather than just the +symbol, and a marked definition the run never entered is *absent* — not reported with a count of +zero. The same answer is available programmatically through [`entered`](@ref). + +## Three things, and the first is the reason to have this | | | |---|---| +| an **observation** | which marked definitions the run went through — [`entered`](@ref), and the summary at exit | | a **declaration** | the reason travels with the name, in the source, where the author is — [`@experimental`](@ref) | -| a **query** | a tool asks the module instead of reading prose — [`experimental`](@ref), [`stable`](@ref) | | a **check** | every public name is accounted for, or the test fails — [`audit`](@ref), [`test_surface`](@ref) | -A marker nobody compares against anything is a claim. A marker something compares against the -public surface is a contract. Everything in the first two rows exists to make the third possible. +A docstring can carry the second row. Nothing a human writes can carry the first: the question is +not "is this name experimental" but "did *this run* go through one", and it has to be answered +after the run, about the run. + +## What it costs + +One short-circuit read in the body, and a write on the first call: + +| emitted into the body | 1 thread | 8 threads | +|---|---|---| +| nothing | 1.00× | 1.00× | +| **the flag `@experimental` emits** | 1.03× | **0.985×** | +| a counter, plain shared `Ref` | 1.03× | 3.76× — and loses 40% of its increments to races | +| a counter, global atomic | 1.17× | 4.87× | +| `@warn`, guarded so it fires once | 5.65× | — | + +10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. The +flag is written once and only read afterwards, so it stops dirtying the cache line — which is why +it is free at eight threads while every counting scheme is not, and why the notice is a summary at +exit rather than a warning at the call. + +Only a definition **with a body** carries a flag; see [`@experimental`](@ref) for the +form-by-form table. Counting, call sites and paths are a separate, opt-in layer that is not built +yet. ## Why this is a separate axis @@ -80,10 +97,11 @@ was never made public, which is the module contradicting itself. | type stability | DispatchDoctor | unrelated | | every public name has a docstring | `Docs.undocumented_names`, `Aqua.test_undocumented_names` | the same check, plus a third answer | | generating documentation | Documenter | only ever checks whether prose exists | -| run-time behaviour | — | calls are untouched | +| run-time behaviour | — | one short-circuit read in the body — see the table above | +| how often a path ran | `Profile`, `@time` | not answered: the default layer knows *whether*, never how often | -`@experimental` emits your definition unchanged plus one `push!` at load time. It does not wrap -the call, does not add a method, and does not change dispatch. +`@experimental` adds one statement to the body and nothing else. It does not add a method, does +not change dispatch, and never allocates or logs. ## Install diff --git a/docs/src/observing.md b/docs/src/observing.md new file mode 100644 index 0000000..173c8d9 --- /dev/null +++ b/docs/src/observing.md @@ -0,0 +1,88 @@ +```@meta +CurrentModule = ExperimentalAPI +``` + +# Observing + +A docstring can say a name is unfinished. It cannot tell you that *this run* went through it. + +That is the question behind the mark: not "is `energy` experimental", which the author already +knows, but "did the number in this figure come out of code nobody validated" — asked after the +run, about the run, by somebody who may not have written either. + +## Without asking + +```julia +using ExperimentalAPI + +@experimental "convergence not established below β ≈ 0.1" energy(β) = β * 1.0000001 + +energy(0.5) +``` + +```console +$ julia sweep.jl +┌ ExperimentalAPI: this run entered 1 experimental definition +│ Main.energy — convergence not established below β ≈ 0.1 +└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this +``` + +Three properties, each of them a decision: + + * **On by default.** The user who never asks is the one who needs telling. [`detecting`](@ref) + reports whether the hook is armed; the environment variable has to be set *before* + `using ExperimentalAPI`, because that is when `atexit` is registered. + * **Silent unless something was entered.** Loading a package that *has* marks prints nothing. + A package that cannot be quiet is one people vendor around. + * **Carries the reason.** The name says which line to open; the reason says whether the result + is affected. + +## As data + +[`entered`](@ref) returns the same thing the summary prints, as a `Vector{`[`Entry`](@ref)`}`: + +```julia +julia> ExperimentalAPI.entered() +1-element Vector{ExperimentalAPI.Entry}: + Entry(Main.energy, "convergence not established below β ≈ 0.1") +``` + +A marked definition the run never entered is **absent**, not reported with a count of zero — the +difference between "observed" and "enumerated". + +[`marked_modules`](@ref) is the search this uses: the loaded modules that carry marks, found by +walking rather than by a registry inside this package, because a table here would be written +while the *marked* package is precompiled and so would be missing from its cache image. +[`summary_text`](@ref) is what the exit hook prints, available as a string for a report of your +own. + +## What it costs + +One short-circuit read in the body, and a write on the first call only. + +| emitted into the body | 1 thread | 8 threads | counts correctly? | +|---|---|---|---| +| nothing | 1.00× | 1.00× | — | +| **the flag `@experimental` emits** | 1.03× | **0.985×** | yes | +| a counter, plain shared `Ref` | 1.03× | 3.76× | **no** — 40% lost to races | +| a counter, global atomic | 1.17× | 4.87× | yes | +| a counter, per-thread atomic | 1.12× | 2.79× | yes | +| `@warn`, guarded so it fires once | 5.65× | — | yes | +| `@warn maxlog=1` | 59.57× | — | yes | + +10M calls of `sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6)))`, minimum of 7–9 trials, Julia 1.12.2. + +Two of those rows decided the design. A flag written once and only read afterwards stops dirtying +the cache line, which is why it is free at eight threads while every counting scheme is not. And +the guarded `@warn` costs 5.65× *even though it fires once*: what stops the definition inlining is +the call being in the body at all, not the warning being printed. That is why the notice is a +summary at exit rather than a warning at the call. + +## What it does not answer + + * **How often.** Presence only — [`Entry`](@ref)`.count` is always `nothing`. Counting is an + opt-in layer that is not built yet. + * **Which method.** Marks are name-keyed, so two methods of a marked name share one flag. + * **Which call site, or by what path.** Also the opt-in layer. + * **Anything about a declaration-only mark.** A name list, a `struct`, a `const`, a `module`: + recorded and audited, never observed. See [`@experimental`](@ref) for the table. diff --git a/src/ExperimentalAPI.jl b/src/ExperimentalAPI.jl index f791ae6..6e708bf 100644 --- a/src/ExperimentalAPI.jl +++ b/src/ExperimentalAPI.jl @@ -1,37 +1,45 @@ """ ExperimentalAPI -Say, at the definition site, that a public name is **not settled yet** — and turn that into a -check something runs. +Say, at the definition site, that a name is **not settled yet** — and find out, without asking, +when a run went through it. -Visibility is already a language feature: `export` and `public` decide who is *allowed* to call a -name. Stability is the orthogonal question — whether the name is *finished* — and today the only -place to answer it is a sentence in a docstring that no tool reads and no CI job verifies. +`public` decides who may call a name. Whether the name is *finished* is the orthogonal question, +and the place it is usually answered is a sentence in a docstring that no tool reads. That matters +most where it is least visible: a long numerical run finishes, hands back a number, and nothing in +the result says which of the code paths behind it had never been validated. ```julia using ExperimentalAPI -@experimental "reads Test's internal result tree; not dogfooded in CI yet" \ -function render_test_report(records) - # ... -end +@experimental "convergence not established below β ≈ 0.1" energy(m::Model) = m.β * correction(m) ``` -That mark is three things at once, and only the third is the reason to have it: +The mark is three things, and the first is the reason to have it: + * an **observation** — [`entered`](@ref) reports the marked definitions this run actually went + through, and a summary says so at process exit whether or not anyone asked; * a **declaration** — the reason travels with the name, in the source, where the author is; - * a **query** — [`experimental`](@ref) hands a tool the list, [`stable`](@ref) hands it the - complement; - * a **check** — [`audit`](@ref) reports every public name that is *neither* documented *nor* + * a **check** — [`audit`](@ref) reports every public name that is neither documented nor declared, so "document it or admit it is unfinished" becomes a test that fails. -The check is the point. A marker that is only ever written is a claim; a marker something -compares against the public surface is a contract. +# What it costs + +One short-circuit read in the body, and a write on the first call only: measured at 1.03x on one +thread and 0.985x on eight, for 10M calls of a numeric body. A flag written once and read +thereafter stops dirtying the cache line, which a counter (3.76x at eight threads, and losing 40% +of its increments to races unless atomic) does not. Counting, call sites and paths are a separate +layer that is not built yet. + +Only a definition **with a body** carries a flag. A mark written as a name list, or attached to a +struct, a const or a module, is a declaration — queryable, audited, but not observed at run time. +See [`@experimental`](@ref) for the form-by-form table. # The three questions this answers | question | the call | |---|---| +| did this run go through unvalidated code? | `entered()` — and the summary at exit says so anyway | | what is unfinished here? | `experimental(M)` | | what does this module owe nobody an explanation for? | `audit(M).unaccounted` — should be empty | | is dropping this name breaking? | `compare(old_snapshot, M)` — see [`isbreaking`](@ref) | @@ -42,8 +50,8 @@ compares against the public surface is a contract. public and experimental, or public and settled; those are independent axes. * **Not deprecation.** `@deprecate` points the other way — a settled name on its way out. * **Not type stability.** Unrelated axis, different tooling. - * **Not a runtime wrapper.** `@experimental` emits the definition unchanged plus one `push!` at - load time. Calls are untouched. + * **Not instrumentation.** The body gains one short-circuit read, nothing more: no counter, no + logging call, no allocation. What it can answer is "was this entered at all", never how often. * **Not a documentation generator.** The prose belongs to the author; this only ever checks whether prose exists. @@ -62,11 +70,13 @@ export @experimental public Mark, Audit, Diff public experimental, isexperimental, mark, isdocumented +public Entry, entered, marked_modules, detecting, summary_text public surface, stable, audit public snapshot, read_snapshot, write_snapshot, compare, isbreaking public test_surface include("mark.jl") # the Mark record, the per-module registry, and @experimental +include("detect.jl") # which marked definitions a run entered, and the summary at exit include("query.jl") # reading a module's marks back out include("audit.jl") # the public surface, and the names neither account covers include("release.jl") # a snapshot of the covenant, and what a diff of two of them means @@ -95,4 +105,11 @@ Returns the [`Audit`](@ref) on the normal return path whether the testset passed """ function test_surface end +# Armed at load time rather than on first mark: the hook has to be in place before any of the +# marked package's code runs, and `atexit` is the only place a summary can see a whole run. +function __init__() + detecting() && atexit(_summarise) + return nothing +end + end # module diff --git a/src/detect.jl b/src/detect.jl new file mode 100644 index 0000000..1f52ada --- /dev/null +++ b/src/detect.jl @@ -0,0 +1,185 @@ +# The default layer: which marked definitions a run actually entered. +# +# Scope: presence, not counts, and only for definitions with a body. A mark written as a name list, +# or attached to a struct, const, module or macro, is a declaration only — nothing observes it. + +# One flag per marked NAME, in the marked module, next to its registry. Named rather than +# gensym'd for the same reason the registry is. +_flag_name(n::Symbol) = Symbol("__EXPERIMENTAL_API_ENTERED_", n, "__") + +function _flag(m::Module, n::Symbol) + s = _flag_name(n) + isdefined(m, s) || return nothing + f = getglobal(m, s) + return f isa Base.RefValue{Bool} ? f : nothing +end + +# What the macro puts in the body. Measured on 10M calls of a numeric body: 1.03x on one thread +# and 0.985x on eight, because after the first call it is a read of a cache line nobody writes. +# An unconditional store, a counter, or anything that logs is not affordable here — see +# `test/spec/README.md`. +_probe(flag) = :($flag[] || ($flag[] = true)) + +function _is_signature(x) + return x isa Expr && ( + x.head === :call || + ((x.head === :where || x.head === :(::)) && _is_signature(x.args[1])) + ) +end + +# Returns the definition with the probe spliced in, or `nothing` if this form has no body to +# instrument. `@generated` and other macro-wrapped forms are refused rather than guessed at: the +# body of a generated function returns an expression, so a probe there would be spliced into the +# generated code instead of running. +function _instrument(def, flag) + def isa Expr || return nothing + (def.head === :function || def.head === :(=)) || return nothing + _is_signature(def.args[1]) || return nothing + length(def.args) == 2 || return nothing + return Expr(def.head, def.args[1], Expr(:block, _probe(flag), def.args[2])) +end + +""" + Entry + +One marked definition that the current run entered, as reported by [`entered`](@ref). + +`count` is always `nothing`. The default layer knows *whether* a definition was entered, never how +often — a per-call counter costs 3.76x on eight threads and loses 40% of its increments to races +unless it is atomic. Counting is the opt-in layer's job. +""" +struct Entry + mod::Module + name::Symbol + reason::String + count::Nothing +end + +Entry(m::Module, n::Symbol, r::AbstractString) = Entry(m, n, String(r), nothing) + +function Base.show(io::IO, e::Entry) + return print(io, "Entry(", e.mod, ".", e.name, ", ", repr(e.reason), ")") +end + +""" + entered(m::Module) -> Vector{Entry} + entered() -> Vector{Entry} + +The marked definitions this run has entered at least once. + +Without an argument, every loaded module that carries marks. This is the question a docstring +cannot answer: not "is this name experimental" but "did the number I am about to publish come out +of code nobody has validated". + +A definition that was never called is **absent**, not reported with a count of zero. + +```julia +julia> MyPkg.energy(0.5); + +julia> entered() +1-element Vector{ExperimentalAPI.Entry}: + Entry(MyPkg.energy, "convergence not established below β ≈ 0.1") +``` + +Only definitions with a body are observed; see [`@experimental`](@ref) for which forms those are. +""" +function entered(m::Module) + out = Entry[] + isdefined(m, MARKS_BINDING) || return out + for mk in _registry_of(m) + f = _flag(m, mk.name) + f !== nothing && f[] && push!(out, Entry(m, mk.name, mk.reason)) + end + return out +end + +entered() = reduce(vcat, (entered(m) for m in marked_modules()); init=Entry[]) + +# Base and Core are skipped rather than walked: neither can carry a mark, and `names(Base; +# all=true)` is thousands of bindings whose `getglobal` can warn. +const _NOT_WALKED = (Base, Core) + +""" + marked_modules() -> Vector{Module} + +Every loaded module that carries at least one [`@experimental`](@ref) mark. + +Found by walking the loaded packages rather than by a registry inside this package: a table here +would be written while the *marked* package is precompiled, and so would be absent from its cache +image. Same constraint that puts the marks themselves in the marked module. +""" +function marked_modules() + out = Module[] + seen = Set{Module}() + for root in values(Base.loaded_modules) + _walk_modules!(out, seen, root) + end + return out +end + +function _walk_modules!(out::Vector{Module}, seen::Set{Module}, m::Module) + (m in seen || m in _NOT_WALKED) && return nothing + push!(seen, m) + isdefined(m, MARKS_BINDING) && push!(out, m) + for n in names(m; all=true) + isdefined(m, n) || continue + v = try + getglobal(m, n) + catch + continue + end + v isa Module && v !== m && parentmodule(v) === m && _walk_modules!(out, seen, v) + end + return nothing +end + +""" + detecting() -> Bool + +Whether the summary at process exit is armed. + +On by default, because a user who never asks is exactly the one who needs to be told. Set +`ENV["EXPERIMENTALAPI_SUMMARY"] = "0"` **before** `using ExperimentalAPI` to silence it; the +`atexit` hook is registered at load time, so a later assignment has no effect. + +Detection itself is not a switch: the flag is in the definition and costs nothing to leave on. +""" +detecting() = get(ENV, "EXPERIMENTALAPI_SUMMARY", "1") != "0" + +""" + summary_text() -> String + summary_text(es::AbstractVector{Entry}) -> String + +What the exit summary prints, as a string. Empty when nothing marked was entered. + +Carries the reason, not just the name: the name tells a reader which line to look at, the reason +tells them whether the result they are holding is affected. +""" +function summary_text(es::AbstractVector{Entry}=entered()) + isempty(es) && return "" + io = IOBuffer() + n = length(es) + println( + io, + "┌ ExperimentalAPI: this run entered $n experimental definition$(n == 1 ? "" : "s")", + ) + for e in es + println(io, "│ ", e.mod, ".", e.name, " — ", e.reason) + end + println( + io, "└ set ENV[\"EXPERIMENTALAPI_SUMMARY\"] = \"0\" before `using` to silence this" + ) + return String(take!(io)) +end + +# Printed to stderr, and only when something was entered: loading a package that HAS marks must be +# silent, or this package is intolerable as a dependency. Never throws — a summary that breaks a +# process on its way out would be worse than no summary. +function _summarise() + try + s = summary_text() + isempty(s) || print(stderr, s) + catch + end + return nothing +end diff --git a/src/mark.jl b/src/mark.jl index 4c421fb..653dcaf 100644 --- a/src/mark.jl +++ b/src/mark.jl @@ -95,9 +95,8 @@ end Declare that a public name is not settled, and say why. -The reason is required and comes first. Everything after it is either **one definition** — which -is emitted unchanged, so this costs nothing at run time — or **a list of names** already defined -elsewhere: +The reason is required and comes first. Everything after it is either **one definition** or **a +list of names** already defined elsewhere: ```julia # attached to the definition @@ -126,6 +125,26 @@ Marking is not visibility: a marked name still has to be `export`ed or declared part of the surface. A mark on a name that is neither is reported by [`audit`](@ref) as `dangling`. +# What gets observed + +A definition **with a body** — `function` and short-form `f(x) = …`, including parametric and +return-type-annotated signatures — also gets a flag in that body, so [`entered`](@ref) can report +whether the run went through it. The flag is one short-circuit read: `1.03x` on one thread and +`0.985x` on eight, measured over 10M calls of a numeric body. + +Every other form is a **declaration only** — recorded, queryable and audited, but not observed: + +| form | observed? | +|---|---| +| `function f(x) … end`, `f(x) = …`, `f(x::T) where {T} = …`, `f(x)::R = …` | yes | +| a name list (`@experimental "…" a b c`) | no — those definitions are elsewhere and already compiled | +| `struct`, `abstract type`, `primitive type`, `const`, assignment | no — nothing is *entered* | +| `macro` | no | +| `@generated function` | refused outright, as any macro-produced definition is; the name-list form takes it, as a declaration | + +One flag per marked **name**, not per method: marks are name-keyed, so two methods of a marked +name share it. + !!! note "Top level only" The mark is stored in a `const` binding in the enclosing module, so `@experimental` belongs at module top level — the same place `export` and `public` go. @@ -224,10 +243,41 @@ macro experimental(args...) ), )) for n in names ] + # The default layer's flag, one per marked name, and the probe that sets it. Only a + # definition with a body can carry one; a name list, a struct, a const, a module or a + # `@generated` function is a declaration this package cannot observe. `_instrument` returns + # `nothing` for those and the definition goes out untouched. + flags = Expr[] + instrumented = def + if def !== nothing + # The `const` is emitted on its own, so it needs escaping here; the probe rides inside + # the definition, which is escaped as a whole below — escaping it twice is an error. + flagsym = _flag_name(names[1]) + push!( + flags, + Expr( + :if, + :(!$(isdefined)($__module__, $(QuoteNode(flagsym)))), + Expr( + :block, + src, + Expr( + :const, Expr(:(=), esc(flagsym), :($(Base.RefValue{Bool})(false))) + ), + ), + ), + ) + probed = _instrument(def, flagsym) + probed === nothing || (instrumented = probed) + end # Tells the documentation system which expression a preceding docstring belongs to, as # `Base.@kwdef` does. Without it a docstring on a marked definition fails to attach at all. - body = def === nothing ? nothing : Expr(:block, Expr(:meta, :doc), esc(def)) - return Expr(:block, init, body, records..., nothing) + body = if instrumented === nothing + nothing + else + Expr(:block, Expr(:meta, :doc), esc(instrumented)) + end + return Expr(:block, init, flags..., body, records..., nothing) end # Normalises whitespace so a wrapped triple-quoted reason does not carry its indentation into diff --git a/test/runtests.jl b/test/runtests.jl index a4e9c83..b04644d 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -24,5 +24,6 @@ using Test include("spec/test_spec_dispatch.jl") include("spec/test_spec_lifecycle.jl") include("test_spec_table.jl") + include("test_readme.jl") include("test_aqua.jl") end diff --git a/test/spec/README.md b/test/spec/README.md index 549f6ff..ddc60c5 100644 --- a/test/spec/README.md +++ b/test/spec/README.md @@ -50,10 +50,10 @@ that is entirely `@test_broken` is a claim written down, not a check being run. | `test_spec_forms.jl` | 26 | 15 | 11 | the definition forms a real package hits on its second afternoon | | `test_spec_integration.jl` | 17 | 1 | 16 | where the mark has to surface: docs, Aqua, releases, provenance, CI | | `test_spec_lifecycle.jl` | 15 | 7 | 8 | the mark's EXIT, and an entry point that is a module rather than a function | -| `test_spec_profile.jl` | 40 | 5 | 35 | what a real run went through, how often, and how much of it | +| `test_spec_profile.jl` | 40 | 12 | 28 | what a real run went through, how often, and how much of it | | `test_spec_propagate.jl` | 20 | 2 | 18 | a caller that never names a marked thing still depends on it | | `test_spec_verify.jl` | 8 | 2 | 6 | how well is a marked thing exercised by the tests | -| **10 files** | **174** | **54** | **120** | | +| **10 files** | **174** | **61** | **113** | | The table is generated and pinned by `test/test_spec_table.jl`, which fails if it goes stale — diff --git a/test/spec/test_spec_profile.jl b/test/spec/test_spec_profile.jl index 7ddc9f4..6fb6858 100644 --- a/test/spec/test_spec_profile.jl +++ b/test/spec/test_spec_profile.jl @@ -49,19 +49,19 @@ const M = Sim.Model(0.5) @testset "a run reports what it entered without being asked to record" begin Sim.driver(M, 10) - @test_broken :energy in [h.name for h in ExperimentalAPI.entered()] + @test :energy in [h.name for h in ExperimentalAPI.entered()] end @testset "the default answer is presence, and does not pretend to be counts" begin # Scope: "at least once". A count field here would read as a measurement it did not make. Sim.driver(M, 10) - @test_broken ExperimentalAPI.entered()[1].count === nothing + @test ExperimentalAPI.entered()[1].count === nothing end @testset "a definition that was never entered is absent from the default answer too" begin # Control: separates observed from enumerated. Sim.driver(M, 10) - @test_broken :cold ∉ [h.name for h in ExperimentalAPI.entered()] + @test :cold ∉ [h.name for h in ExperimentalAPI.entered()] end # Observed from a child process: an `atexit` handler registered in this one would pass for a @@ -117,7 +117,7 @@ end end # module ChildRun @testset "the summary is printed at process exit" begin - @test_broken occursin("energy", ChildRun.output(ChildRun.ENTERS)) + @test occursin("energy", ChildRun.output(ChildRun.ENTERS)) end @testset "a process that loaded a mark but never entered it stays silent" begin @@ -126,12 +126,14 @@ end end @testset "the summary carries the reason, not just the name" begin - @test_broken occursin("convergence not established", ExperimentalAPI.summary_text()) + @test occursin("convergence not established", ExperimentalAPI.summary_text()) end @testset "the default layer can be turned off" begin - # Scope: readable before `using` returns, so an environment variable rather than a call. - @test_broken ExperimentalAPI.detecting() === true + # Scope: read before `using` returns, so an environment variable rather than a call — the + # `atexit` hook is registered at load time. + @test ExperimentalAPI.detecting() === true + @test withenv(ExperimentalAPI.detecting, "EXPERIMENTALAPI_SUMMARY" => "0") === false end # ── the opt-in layer: the basic question ───────────────────────────────────────────────────── @@ -243,18 +245,22 @@ end end # module WrapControl -@testset "today the expansion is untouched — which is what has to change" begin - # Scope: the default layer needs one statement in the body, so this equality must break. +@testset "the expansion adds exactly one statement, and it is a short-circuit read" begin + # Scope: what the default layer may put in a body. One statement, and a read that writes only + # on the first call — an unconditional store costs 3.76x at eight threads. # Compared at the AST: the expansion carries `Expr(:escape, …)` and prints differently. bare = WrapControl.method_bodies(@macroexpand f(x) = x * 2) marked = WrapControl.method_bodies(@macroexpand @experimental "why" f(x) = x * 2) @test length(bare) == 1 - @test marked == bare + @test length(marked[1].args) == length(bare[1].args) + 1 + @test marked[1].args[1].head === :|| + # Control: `@wrapping` adds one statement too, and it is a store. Without this, `head === :||` + # is the only thing separating the two and nothing checks that it can be otherwise. wrapped = WrapControl.method_bodies( @macroexpand WrapControl.@wrapping "why" f(x) = x * 2 ) - @test wrapped != bare - @test_broken length(marked[1].args) == length(bare[1].args) + 1 + @test length(wrapped[1].args) == length(bare[1].args) + 1 + @test wrapped[1].args[1].head !== :|| @test Sim.driver(M, 3) ≈ 3 * (0.5 * 1.0000001 + exp(-0.5)) end @@ -281,7 +287,7 @@ end end @testset "detection is on by default; counting is not" begin - @test_broken ExperimentalAPI.detecting() === true + @test ExperimentalAPI.detecting() === true @test_broken ExperimentalAPI.recording() === false end diff --git a/test/test_readme.jl b/test/test_readme.jl new file mode 100644 index 0000000..c509132 --- /dev/null +++ b/test/test_readme.jl @@ -0,0 +1,58 @@ +# The README's primary example, executed. +# +# Scope: the first ```julia block only — the rest of the README uses `MyPackage` as illustration. +# An example that does not run is the first thing a reader tries and the first thing that makes +# them close the tab. + +using ExperimentalAPI +using Test + +function readme_first_block(path=joinpath(@__DIR__, "..", "README.md")) + # Git checks text files out with CRLF on Windows, so every fence here would miss. Same + # normalisation as `test_spec_table.jl`, which learned it from a red windows-latest. + md = replace(read(path, String), "\r\n" => "\n") + i = findfirst("```julia\n", md) + i === nothing && error("README.md has no ```julia block") + rest = md[(last(i) + 1):end] + j = findfirst("```", rest) + j === nothing && error("README.md's first ```julia block is unterminated") + return rest[1:(first(j) - 1)] +end + +@testset "the README's first example is a runnable program" begin + src = readme_first_block() + # The claim is that it runs as written, so it is evaluated as a whole rather than line by + # line: `@experimental` emits a `const`, which is legal only at top level. + @test occursin("@experimental", src) + m = Module(:READMEExample) + Core.eval(m, Meta.parseall(src; filename="README.md")) + @test m.energy(0.5) ≈ 0.5 * 1.0000001 +end + +@testset "and it produces the output the README shows" begin + # The README prints two things: the exit summary and `entered()`. Both are quoted verbatim + # there, so both are checked here — a lede that runs but reports something else is no better. + src = readme_first_block() + m = Module(:READMEOutput) + Core.eval(m, Meta.parseall(src; filename="README.md")) + es = ExperimentalAPI.entered(m) + @test [e.name for e in es] == [:energy] + @test es[1].reason == "convergence not established below β ≈ 0.1" + @test occursin( + "this run entered 1 experimental definition", ExperimentalAPI.summary_text(es) + ) + @test occursin(string(m, ".energy"), ExperimentalAPI.summary_text(es)) +end + +@testset "a README example that stopped running would fail this" begin + # Control: the check is not satisfied by any block at all. `Meta.parseall` never throws, so + # a broken example would otherwise be evaluated as an `:error` node and silently do nothing. + broken = Module(:READMEControl) + e = try + Core.eval(broken, Meta.parseall("energy(0.5) = ("; filename="broken")) + nothing + catch err + err + end + @test e !== nothing +end