test: the case matrix, written before the implementation

.github/workflows/CI.yml

@@ -49,6 +49,12 @@ jobs:
       - uses: julia-actions/cache@v3
       - uses: julia-actions/julia-buildpkg@v1
       - uses: julia-actions/julia-runtest@v1
+        env:
+          # `test/spec/test_spec_profile.jl` asserts `Threads.nthreads() > 1`. A
+          # `Threads.@threads` loop runs its body the same number of times whatever the thread
+          # count is, so a concurrency test on a single-threaded runner cannot fail for a
+          # recorder that is not thread safe — it would be untestable rather than passing.
+          JULIA_NUM_THREADS: '4'
       - uses: julia-actions/julia-processcoverage@v1
         if: matrix.julia == '1.12' && matrix.os == 'ubuntu-latest'
       - uses: codecov/codecov-action@v7

Project.toml

@@ -14,13 +14,15 @@ ExperimentalAPITestExt = "Test"
 
 [compat]
 Aqua = "0.8"
+Profile = "1"
 TOML = "1"
 Test = "1"
 julia = "1.11"
 
 [extras]
 Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
+Profile = "9abbd945-dff8-562f-b5e8-e1ebf5ef1b79"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Test", "Aqua"]
+test = ["Test", "Aqua", "Profile"]

README.md

@@ -183,6 +183,14 @@ reads as if it had none:
 - **Whether a name appears in your guide, README or docs site.** Docstring presence is not
   documentation-page presence, and those two gaps are usually different sets.
 
+## Where this is going
+
+`test/spec/` is the specification for the rest: the propagation, profiling and lifecycle work is
+written there as tests before it is implemented, so it cannot drift from the code. Most of it is
+`@test_broken` today, and [`test/spec/README.md`](test/spec/README.md) explains why that register
+was chosen, which of the negative controls are actually running, and the two defects the exercise
+already found in the shipped code.
+
 ## License
 
 MIT

src/mark.jl

@@ -1,11 +1,9 @@
-# The mark itself: what one `@experimental` declaration records, where it is stored, and the
-# macro that writes it.
+# What one `@experimental` declaration records, where it is stored, and the macro that writes it.
 #
-# Storage is a `const` vector inside the MARKED module, not a table inside this one. That is not
-# a style choice — a registry living here would be populated while the marked package is being
-# precompiled, and nothing written into a third module at that moment is part of the cache image
-# that gets loaded later. Base's own `Docs.META` is a per-module binding for exactly this reason,
-# and `test/test_precompile.jl` is the test that would catch it if this stopped being true.
+# Storage is a `const` vector inside the MARKED module: a registry here would be populated during
+# the marked package's precompilation, and nothing written into a third module then survives into
+# its cache image. `Docs.META` is per-module for the same reason. Pinned by
+# `test/test_precompile.jl`.
 
 """
     Mark
@@ -35,6 +33,15 @@ struct Mark
     tracking::Union{String,Nothing}
     file::Symbol
     line::Int
+
+    # The reason is the payload, so the invariant belongs in the type: `Mark` is `public`, and
+    # every construction route that is not the macro gets it for free here.
+    function Mark(mod, name, reason, since, tracking, file, line)
+        r = String(strip(reason))
+        isempty(r) &&
+            throw(ArgumentError("a Mark's reason may not be empty — it is the payload"))
+        return new(mod, name, r, since, tracking, file, line)
+    end
 end
 
 function Base.show(io::IO, m::Mark)
@@ -52,18 +59,13 @@ function Base.show(io::IO, ::MIME"text/plain", m::Mark)
     return print(io, "  declared: ", m.file, ":", m.line)
 end
 
-# The binding every marked module gets. Named, not gensym'd, so `M.__EXPERIMENTAL_API_MARKS__`
-# is greppable and inspectable when a query result surprises someone.
+# Named rather than gensym'd, so it is greppable when a query result surprises someone.
 const MARKS_BINDING = :__EXPERIMENTAL_API_MARKS__
 
-# The registry is created by code the macro emits INTO the marked module — not by `Core.eval`
-# from here. The difference is not stylistic. Julia 1.12 rejects reading a binding that was
-# created earlier in the same top-level statement ("define the const at top-level before running
-# the function that uses it"), and a `Core.eval`-then-`getglobal` helper does exactly that. The
-# emitted form puts the `const` in one top-level statement and the `push!` in the next, so the
-# world age has advanced in between and the read is legal.
-#
-# `@macroexpand` therefore shows the whole mechanism, which is the second reason to prefer it.
+# Created by code the macro emits into the marked module, not by `Core.eval` from here: Julia
+# 1.12 rejects reading a binding created earlier in the same top-level statement, which is what a
+# `Core.eval`-then-`getglobal` helper does. The emitted form puts the `const` and the `push!` in
+# separate statements, so the world age has advanced in between.
 function _registry_of(m::Module)
     v = getglobal(m, MARKS_BINDING)
     v isa Vector{Mark} || throw(
@@ -75,8 +77,7 @@ function _registry_of(m::Module)
     return v
 end
 
-# Re-marking a name replaces its entry instead of appending, so re-including a file (Revise, an
-# `include` reached twice) cannot make one name appear in `experimental(M)` several times.
+# Replaces rather than appends, so re-including a file cannot duplicate a name.
 function _mark!(reg::Vector{Mark}, mk::Mark)
     i = findfirst(x -> x.name === mk.name, reg)
     if i === nothing
@@ -155,9 +156,8 @@ macro experimental(args...)
     isempty(rest) &&
         throw(ArgumentError("@experimental: nothing to mark — give a definition or a name"))
 
-    # `k = v` pairs bind tighter than the subject only when they are NOT the last argument: the
-    # last argument is always the thing being marked, which keeps `@experimental "…" x = 3`
-    # unambiguous against `@experimental "…" since=v"1" x = 3`.
+    # The last argument is always the subject, which keeps `@experimental "…" x = 3` unambiguous
+    # against `@experimental "…" since=v"1" x = 3`.
     since, tracking, i = nothing, nothing, 1
     while i < length(rest)
         a = rest[i]
@@ -198,16 +198,18 @@ macro experimental(args...)
         names = [_defname(def)]
     end
 
-    # Statement order carries a constraint: the `const` must land in its own top-level statement,
-    # because the `_mark!` calls below READ that binding and Julia 1.12 forbids reading a binding
-    # created in the same world age.
+    # The `const` must land in its own top-level statement: the `_mark!` calls below read that
+    # binding, and Julia 1.12 forbids reading one created in the same world age.
     marks = esc(MARKS_BINDING)
-    init = :(
-        if !$(isdefined)($__module__, $(QuoteNode(MARKS_BINDING)))
-            const $marks = $(Mark)[]
-        end
-    )
     src = __source__
+    # Built with `Expr` so no `LineNumberNode` from this file reaches the expansion. `const` in
+    # local scope is a lowering error this macro cannot catch, so the only lever is where the
+    # error points, and it must point at the caller. Pinned in `test_spec_forms.jl`.
+    init = Expr(
+        :if,
+        :(!$(isdefined)($__module__, $(QuoteNode(MARKS_BINDING)))),
+        Expr(:block, src, Expr(:const, Expr(:(=), marks, :($(Mark)[])))),
+    )
     records = [
         :($(_mark!)(
             $marks,
@@ -222,16 +224,14 @@ macro experimental(args...)
             ),
         )) for n in names
     ]
-    # `Expr(:meta, :doc)` is how a macro tells the documentation system which expression inside
-    # its expansion a preceding docstring belongs to — the mechanism `Base.@kwdef` uses. Without
-    # it, `"""docs""" @experimental "why" f(x) = x` fails with "cannot document the following
-    # expression", which would make the two accounts this package asks for mutually exclusive.
+    # Tells the documentation system which expression a preceding docstring belongs to, as
+    # `Base.@kwdef` does. Without it a docstring on a marked definition fails to attach at all.
     body = def === nothing ? nothing : Expr(:block, Expr(:meta, :doc), esc(def))
     return Expr(:block, init, body, records..., nothing)
 end
 
-# Whitespace is normalised so a reason written as a wrapped triple-quoted string does not carry
-# its indentation into every message that prints it. Nothing else about the text is touched.
+# Normalises whitespace so a wrapped triple-quoted reason does not carry its indentation into
+# every message. Nothing else about the text is touched.
 function _reason(s)
     r = String(strip(s))
     isempty(r) && throw(
@@ -251,9 +251,8 @@ function _subject_names(subject)
             a.args[1] isa Symbol &&
             length(a.args) == 2 &&
             a.args[2] isa LineNumberNode
-            # A BARE `@foo` — the name a macro is public under. A macrocall carrying arguments
-            # is a definition this macro cannot read, not a name, and must fall through to be
-            # refused rather than silently recorded as `Symbol("@doc")`.
+            # A bare `@foo` is a name; a macrocall carrying arguments is a definition this macro
+            # cannot read, and must fall through to be refused rather than recorded.
             push!(names, a.args[1])
         elseif a isa QuoteNode && a.value isa Symbol
             push!(names, a.value)              # `:foo`, for a name a reader prefers to quote
@@ -276,8 +275,8 @@ function _defname(ex::Expr)
     h === :abstract && return _typename(ex.args[1])
     h === :primitive && return _typename(ex.args[1])
     h === :const && return _defname(ex.args[1])
-    # A `module` cannot be flattened out of the block this macro emits — Julia requires it as a
-    # direct top-level statement — so it takes the name-list form rather than being wrapped.
+    # Julia requires `module` as a direct top-level statement, so it cannot be wrapped and takes
+    # the name-list form.
     h === :module && throw(
         ArgumentError(
             "@experimental cannot attach to a `module`, which Julia requires at top level. " *

test/runtests.jl

@@ -11,5 +11,18 @@ using Test
     include("test_ext.jl")
     include("test_precompile.jl")
     include("test_dogfood.jl")
+    # The case matrix. Written before the implementation, so most of it is @test_broken;
+    # see test/spec/README.md for why that is the right register.
+    include("spec/test_spec_declare.jl")
+    include("spec/test_spec_propagate.jl")
+    include("spec/test_spec_docstring.jl")
+    include("spec/test_spec_verify.jl")
+    include("spec/test_spec_profile.jl")
+    include("spec/test_spec_foreign.jl")
+    include("spec/test_spec_forms.jl")
+    include("spec/test_spec_integration.jl")
+    include("spec/test_spec_dispatch.jl")
+    include("spec/test_spec_lifecycle.jl")
+    include("test_spec_table.jl")
     include("test_aqua.jl")
 end

test/spec/README.md

@@ -0,0 +1,141 @@
+# The case matrix
+
+These files are the specification. They are written before the implementation, so most of them
+are `@test_broken`.
+
+`@test_broken` was chosen deliberately over comments or a TODO list:
+
+  * an expression that throws (because the function does not exist yet) registers as **Broken**,
+    not as an error, so the suite stays green while the spec is incomplete;
+  * an expression that starts **passing** registers as `Error: Unexpected Pass`, which fails the
+    suite until someone promotes it to `@test`.
+
+So the spec cannot silently rot in either direction: it neither blocks work that has not been
+done, nor lets finished work go unnoticed.
+
+One caveat, measured rather than assumed: an expression whose value is not a `Bool` — an
+`@eval module … end`, whose value is a `Module` — reports `Error: Expression evaluated to
+non-Boolean` instead of `Unexpected Pass`. Both fail the suite loudly, so nothing rots either
+way, but every `@eval module` block in this directory now ends in a `Bool` and checks *what* got
+marked, so the message is the expected one and the assertion has content beyond "it parsed".
+
+## Overlap with `test/test_*.jl`
+
+`test_spec_declare.jl` re-covers definition forms that `test/test_mark.jl` already pins, and
+`test_spec_docstring.jl` re-covers part of `test/test_audit.jl`'s bucket contract, through
+separate fixtures. That is deliberate while the spec is the design document — but two fixtures
+pinning one contract have to be kept in sync by hand, so the older files should be folded in or
+retired once the spec stops moving.
+
+Anything already implemented is a plain `@test`.
+
+## What is covered
+
+The measure is **distinct behaviours** — one per leaf `@testset` — not assertions. An assertion
+count moves without any implementation progress: `test_spec_declare.jl` has 36 assertion lines
+but 91 runtime assertions, because several run inside `for mk in experimental(Declared)`, so a
+thirteenth fixture mark would buy four more passing assertions and cover nothing new. A leaf
+testset is one claim, and adding one means writing one.
+
+*operating today* is the column that matters when reading a claim about this directory: a leaf
+that is entirely `@test_broken` is a claim written down, not a check being run.
+
+<!-- BEGIN GENERATED: julia --project=test test/spec/summary.jl -->
+| file | behaviours | operating today | specified only | concern |
+|---|---|---|---|---|
+| `test_spec_declare.jl` | 11 | 7 | 4 | what can carry a mark: function, method, struct, const, module, macro, extension |
+| `test_spec_dispatch.jl` | 14 | 4 | 10 | one call site, several methods, only some marked — the branch |
+| `test_spec_docstring.jl` | 9 | 6 | 3 | a mark and a docstring are different accounts and must coexist |
+| `test_spec_foreign.jl` | 14 | 5 | 9 | marking a method on somebody else's generic — the `QAtlas.fetch` case |
+| `test_spec_forms.jl` | 26 | 15 | 11 | the definition forms a real package hits on its second afternoon |
+| `test_spec_integration.jl` | 17 | 1 | 16 | where the mark has to surface: docs, Aqua, releases, provenance, CI |
+| `test_spec_lifecycle.jl` | 15 | 7 | 8 | the mark's EXIT, and an entry point that is a module rather than a function |
+| `test_spec_profile.jl` | 40 | 5 | 35 | what a real run went through, how often, and how much of it |
+| `test_spec_propagate.jl` | 20 | 2 | 18 | a caller that never names a marked thing still depends on it |
+| `test_spec_verify.jl` | 8 | 2 | 6 | how well is a marked thing exercised by the tests |
+| **10 files** | **174** | **54** | **120** | |
+<!-- END GENERATED -->
+
+The table is generated and pinned by `test/test_spec_table.jl`, which fails if it goes stale —
+the hand-written version drifted inside the change that introduced it.
+
+## Two layers, and why the split is where it is
+
+The goal is a tool that says **where** experimental code was used, and a user who learns they
+used it **without opting in**. Those are different jobs with different budgets, and the boundary
+between them was measured rather than chosen. 10M calls of a realistic numeric body, Julia
+1.12.2, minimum of 7–9 trials:
+
+| emitted into the body | 1 thread | 8 threads | counts correctly? |
+|---|---|---|---|
+| nothing | 1.00× | 1.00× | — |
+| set-once flag, read-mostly | 1.03× | **0.985×** | yes |
+| counter, plain shared `Ref` | 1.03× | 3.76× | **no** |
+| counter, global atomic | 1.17× | 4.87× | yes |
+| counter, per-thread atomic | 1.12× | 2.79× | yes |
+| `@warn`, guarded so it fires once | 5.65× | — | yes |
+| `@warn maxlog=1` | 59.57× | — | yes |
+
+Two results decided it. The plain counter is not merely slow in parallel — it recorded
+95,406,048 of 160,000,000 calls, **losing 40% to races**, so it is wrong as well as expensive.
+And a flag written once and only read afterwards never dirties the cache line again, which is
+why it is free at eight threads while every counting scheme is not.
+
+So **presence is detected by default and costs nothing; counts, call sites and paths are
+opt-in.** The `@warn` rows are why the default notice is a summary at process exit rather than a
+warning at first entry: the cost is the logging call sitting in the body, not the warning being
+printed, and guarding it so that it fires once does not recover it.
+
+### One requirement was withdrawn
+
+This directory used to require that `@experimental` **never wrap the call**, and
+`test_spec_profile.jl` pinned it structurally. That is gone: presence cannot be detected without
+emitting something into the body. What replaced it is narrower and measured — the emitted
+statement must be read-mostly, must add exactly one statement, and must not bring the logging
+machinery with it. The last of those is checked today, with a macro that *does* log as the
+positive control.
+
+## Negative controls
+
+Each group has a negative control **specified**; most are not yet operating, because the control
+is `@test_broken` alongside the claim it controls. They are listed here as design, not as
+evidence:
+
+| group | the control | operating? |
+|---|---|---|
+| propagate   | `top_good` is *exactly as deep* as `top_bad` and must come back `:clean` | no |
+| profile     | `cold` is marked and never called, and must be **absent**, not reported with count zero | no |
+| verify      | the fixture is exercised only *partly*, so a checker that always reports 100% cannot pass | no |
+| integration | a settled name must get **no** docs note | no |
+| dispatch    | `which()` really does throw for the branching signatures the file rests on | **yes** |
+| forms       | the misuse refusals name the missing half, rather than one generic message | **yes** |
+| lifecycle   | deleting a settled name is still breaking, so `isbreaking` cannot answer `false` always | **yes** |
+
+"no" means the assertion about the *implementation* does not run, because the implementation is
+not there. It does not mean the row is unchecked: the fixture premise each control rests on is
+pinned live where it could be got wrong — `test_spec_verify.jl:38` asserts the fixture really is
+only partly exercised, and `test_spec_dispatch.jl:93` asserts the specificity relation the whole
+file assumes. A control resting on a false premise is the failure mode those guard.
+
+The one that matters most is not in the table because it is a rule rather than a fixture:
+`:unknown` must never be reported as `:clean`. Two fixtures (`Holder.f::Function`, `TABLE[i](x)`)
+really can reach the marked function while being statically invisible. Answering "no experimental
+dependency" there is not a weaker claim, it is a false one.
+
+## What the spec already found
+
+Two defects, both live in the shipped code, both of the kind the spec was written to catch —
+a mark that silently records the wrong thing rather than refusing:
+
+1. `@experimental "…" (c::C)(x) = c.k * x` marks **`:c`**, the argument name. Not the type, not
+   a function — a local that is not a binding anywhere. It produces **two** wrong signals, not
+   one: the audit reports `:c` as *dangling* (declared, no such binding) **and** `:C` as
+   *unaccounted* (public, never declared), so it tells the author to go declare the very thing
+   that line declares. Both halves have to move together; `test_spec_forms.jl` pins each.
+2. A mark inside a function body is refused by *Julia*, not by this package:
+   `syntax: unsupported const declaration on local variable`. Half fixed — the expansion now
+   carries the caller's `LineNumberNode`, so the message names the line the author wrote instead
+   of `ExperimentalAPI/src/mark.jl`, which read as a bug in the package. The message still never
+   says `@experimental`, and may not be able to: `const` in local scope fails during lowering,
+   before any emitted code runs, and the one alternative that avoids `const` (`global`) fails
+   *silently* in local scope, which is worse.