test: the case matrix, written before the implementation

[sotashimozono]

Ten files under test/spec/. 174 behaviours: 54 operating today, 120 specified only (157 @test_broken assertions). The suite stays green.

This is the specification for what the package has to become, written as tests rather than as a
document, so it cannot drift away from the code.

Why @test_broken and not a TODO list

Measured before committing to the structure:

@test_broken false        ->  Broken            suite stays green
@test_broken notyet()     ->  Broken            throwing counts as broken, not as an error
@test_broken 1 == 1       ->  Error: Unexpected Pass

So the spec cannot rot in either direction. It does not block work that has not been done, and
it fails the suite the moment finished work goes unpromoted.

What is covered

The measure is distinct behaviours — one per leaf @testset. It used to be the
@test:@test_broken ratio, which moves without any implementation progress:
test_spec_declare.jl has 36 assertion lines but runs 91 assertions, because several sit inside
for mk in experimental(Declared), so a thirteenth fixture mark would buy four more passing
assertions and cover nothing new.

operating today is the column to read when judging a claim about this directory: a leaf that is
entirely @test_broken is a claim written down, not a check being run.

file	behaviours	operating today	specified only	concern
test_spec_declare.jl	11	7	4	what can carry a mark: function, method, struct, const, module, macro, extension
test_spec_dispatch.jl	14	4	10	one call site, several methods, only some marked — the branch
test_spec_docstring.jl	9	6	3	a mark and a docstring are different accounts and must coexist
test_spec_foreign.jl	14	5	9	marking a method on somebody else's generic — the QAtlas.fetch case
test_spec_forms.jl	26	15	11	the definition forms a real package hits on its second afternoon
test_spec_integration.jl	17	1	16	where the mark has to surface: docs, Aqua, releases, provenance, CI
test_spec_lifecycle.jl	15	7	8	the mark's EXIT, and an entry point that is a module rather than a function
test_spec_profile.jl	40	5	35	what a real run went through, how often, and how much of it
test_spec_propagate.jl	20	2	18	a caller that never names a marked thing still depends on it
test_spec_verify.jl	8	2	6	how well is a marked thing exercised by the tests
10 files	174	54	120

The table is generated — julia --project=test test/spec/summary.jl — and pinned by
test/test_spec_table.jl, which fails if it goes stale. The first version of this table was
hand-written and drifted inside this very pull request: two files landed after it was typed, so it
said "nine files" when there were eleven and every count in it was wrong. A file missing from the
generator's CONCERNS, or from runtests.jl, is now an error rather than a silently missing row.

Two layers, and the measurement that put the boundary there

The goal is a tool that says where experimental code was used, and a user who learns they used
it without opting in. Those are different jobs with different budgets. 10M calls of a
realistic numeric body, Julia 1.12.2, minimum of 7–9 trials:

emitted into the body	1 thread	8 threads	counts correctly?
nothing	1.00×	1.00×	—
set-once flag, read-mostly	1.03×	0.985×	yes
counter, plain shared Ref	1.03×	3.76×	no
counter, global atomic	1.17×	4.87×	yes
counter, per-thread atomic	1.12×	2.79×	yes
@warn, guarded so it fires once	5.65×	—	yes
@warn maxlog=1	59.57×	—	yes

Two results decided it. The plain counter is not merely slow in parallel — it recorded
95,406,048 of 160,000,000 calls, losing 40% to races, so it is wrong as well as expensive.
And a flag written once and only read afterwards never dirties the cache line again, which is why
it is free at eight threads while every counting scheme is not.

So presence is detected by default and costs nothing; counts, call sites and paths are opt-in.
The @warn rows are why the default notice is a summary at process exit rather than a warning at
first entry: the cost is the logging call sitting in the body, not the warning being printed, and
guarding it so that it fires once does not recover it.

The summary is observed from a child process. Asserting that an atexit handler is registered
in this one would pass for a handler that prints nothing, and reading stdout alone would pass
trivially if the notice went to stderr. The two child scripts differ only in the final call, so
a summary keyed on "this module has marks" rather than on "this run entered one" fails the
control and passes the claim.

One requirement was withdrawn

test/spec/ used to require that @experimental never wrap the call, pinned structurally.
That is gone: presence cannot be detected without emitting something into the body. What replaced
it is narrower and measured — the emitted statement must be read-mostly, must add exactly one
statement, and must not bring the logging machinery with it. The last of those is checked today,
with a macro that does log as the positive control.

Negative controls: specified, and mostly not yet operating

A spec that only asserts the positive is satisfied by a tool that always answers yes. Each group
has a control specified; most are not running, because the control is @test_broken next to
the claim it controls.

group	the control	operating?
propagate	top_good is exactly as deep as top_bad and must come back :clean	no
profile	cold is marked and never called, and must be absent, not reported with count zero	no
verify	the fixture is exercised only partly, so a checker that always reports 100% cannot pass	no
integration	a settled name must get no docs note	no
dispatch	which() really does throw for the branching signatures the file rests on	yes
forms	the misuse refusals name the missing half, rather than one generic message	yes
lifecycle	deleting a settled name is still breaking, so isbreaking cannot answer false always	yes

"no" means the assertion about the implementation does not run, because the implementation is
not there — not that the row is unchecked. The fixture premise each control rests on is pinned
live where it could be got wrong (test_spec_verify.jl:38, test_spec_dispatch.jl:93).

And the one that matters most, a rule rather than a fixture: :unknown must never be reported as
:clean. Two fixtures (Holder.f::Function, TABLE[i](x)) really can reach the marked function
while being statically invisible. Answering "no experimental dependency" there is not a weaker
claim, it is a false one.

It found two defects while being written

Both shipped, both silent, both of exactly the kind it was written to catch — a mark that records
the wrong thing rather than refusing.

1. @experimental "…" (c::C)(x) = c.k * x marks :c, and the audit compounds it.

_signame walks the :: and returns the binding on the left, which is the argument name. The
mark lands on a local that is not a binding anywhere — and because it does, the audit produces a
second wrong signal:

marks recorded    : [:c]
is :c a binding?  : false
is :C a binding?  : true
audit dangling    : [:c]      # declared, no such binding
audit unaccounted : [:C]      # public, and — per the audit — never declared

So the tool tells the author to go declare the very thing that line declares. Both halves are
pinned as current behaviour, with a broken test asserting that fixing _signame clears both.

2. A mark inside a function body was refused by Julia, pointing into this package.

syntax: unsupported `const` declaration on local variable around …/ExperimentalAPI/src/mark.jl:219

which reads as a bug in the package rather than a misuse of it — the natural next step is to file
an issue here. const in local scope fails during lowering, before any code this macro emits
can run, so no check of ours can intercept it. The one lever left is where the error points, and
that is now fixed: the expansion is built with Expr and carries the caller's LineNumberNode.

syntax: unsupported `const` declaration on local variable around …/caller_side.jl:4

The remaining half — the message still never says @experimental — stays @test_broken, with the
reason it may be unreachable recorded next to it: the only expansion that avoids const is
global, which in local scope fails silently, which is worse.

The location test goes through a real file on purpose. The same misuse written through @eval or
Core.eval, which is how every other refusal in the suite is probed, produces the message with no
location at all — so a location assertion made that way is vacuous and passes against the old
expansion too. Confirmed by reverting src/mark.jl and watching it fail.

Also in this branch

test_spec_profile.jl had a wall-clock assertion. Four routes were measured and each failed for
its own reason, so the file now carries no timing assertion — as a finding, not a gap:

route	what it actually measured
a < 5b + 1e-3	the arms did different work — Sim.energy(Sim.Model(x)) against a bare call. Failed at 11.8 ms vs 2.1 ms
the same ratio, arms made identical	ratio ran 0.79–3.03 over eight trials on an idle machine; a 5× threshold sits inside that
@allocated equality	0 on Julia 1.12 and 16 on 1.11 for the same code — and a counter wrapper push!ing into a warmed vector allocates zero, so it cannot see the regression it was guarding
a threshold on the figures above, now that a flag will be emitted	ruled out in advance: those come from an idle machine, and the same thresholds on a shared CI runner across three OSes would be a flake generator

The third was caught by writing the positive control and watching it fail to fire.

So the claim is checked where it is exact — at the expansion, compared at the AST rather than as
strings — and what only a benchmark can establish is left to a benchmark.

Two more findings from the same benchmarking: threadid() returned 9 under -t 8, because
the interactive pool is counted separately from the default one, so per-thread storage must be
sized by maxthreadid() or the first hit from a REPL task throws BoundsError. And putting
@warn in a body costs 5.65× even when guarded to fire once — what stops the definition
inlining is the call being there at all, not the warning being printed.

[github-actions]

📚 Docs preview: https://codes.sota-shimozono.com/ExperimentalAPI.jl/previews/PR7/

(updates on each push to this PR)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[sotashimozono]

Checked out d526200, ran the suite, and reproduced both defects the description claims. The
central idea holds up under measurement, so the notes below are about the edges.

Suite as it stands: 460 Pass, 136 Broken, 596 total, green, Pkg.test() exit 0.

The load-bearing idea works, and it is checkable

The @test_broken register was the right call, and the reason given for it is the one that
matters: @test_broken 1 == 1 is an Error: Unexpected Pass, so the spec fails the suite when
finished work goes unpromoted. That is not a claim about intent — it is enforced, and the green
suite is the evidence. Nothing in these 136 broken assertions is already implemented.

The negative controls are well conceived. test_spec_profile.jl:66-67 states the distinction
precisely:

A recorder that lists every mark in the module passes the two tests above. This is the control
that separates "observed" from "enumerated".

That is exactly the failure a profiler of this kind falls into, and naming it in the fixture is
worth more than the assertion itself.

---

1. The description has drifted from its own branch

The thesis of this PR is:

written as tests rather than as a document, so it cannot drift away from the code

The description is a document, and on this branch it has drifted. Measured per file, at runtime:

file	description	actual
declare	10 / 10	82 / 9
forms	15 / 11	43 / 14
verify	5 / 6	11 / 6
propagate	5 / 18	11 / 19
docstring	21 / 4	25 / 4
foreign	8 / 11	8 / 10
runtime	3 / 8	4 / 8
profile	0 / 29	1 / 28
integration	0 / 14	0 / 17
dispatch	absent	8 / 10
lifecycle	absent	9 / 11
total	67 / 111	202 / 136

"Nine files under test/spec/" is eleven. dispatch and lifecycle (337 lines) landed after the
description was written and are wired into runtests.jl, so the suite runs strictly more than the
description accounts for.

Not a correctness problem, and PR descriptions go stale routinely. It is worth a comment only
because this PR is the argument that prose drifts and tests do not, and the prose drifted inside
the PR making the argument.

Suggestion: if the ratio is the published progress measure, generate the table. A few lines in
the test/spec/README.md build, or a recipe that prints it from a verbose=true testset, removes
the whole category.

2. "Every group has a negative control" is present tense; the controls are not active yet

test_spec_profile.jl:68:

@test_broken :cold ∉ [h.name for h in ExperimentalAPI.record(() -> Sim.driver(M, 10))]

The control is itself @test_broken, so today it controls nothing — it is a specified control,
not an operating one. It starts protecting the moment record exists and the assertion is
promoted, which is the whole design, but until then a reader of the description believes a
guarantee is in force that is not.

profile is 29 broken to 1 passing and integration is 17 broken to 0 passing, so those two
groups currently assert nothing at all about the implementation. That is a fine state for a spec
written first; it is just a different state from what "every group has a negative control" reads
as. Suggest the future tense, or a column in the table marking which controls are live.

3. The pass:broken ratio moves without implementation progress

test_spec_declare.jl has 36 assertion lines and 91 runtime assertions. The gap is loops:

for mk in experimental(Declared)        # :107 -- 12 marks
    @test !isempty(strip(mk.reason))
    @test mk.mod === Declared
    @test mk.line > 0
    @test String(mk.file) == @__FILE__
end

Those are real property checks and worth having. But they scale with fixture size, while the
broken assertions are hand-written per behaviour. Adding a thirteenth mark to Declared adds four
passing assertions and covers nothing new, so the ratio improves without progress.

If the ratio is meant as the measure, counting distinct behaviours — one per @testset leaf,
say — is harder to inflate than counting assertions.

4. Defect 1 is worse than the description says

The description says @experimental "..." (c::C)(x) = c.k * x marks :c. Reproduced, and the
consequence goes further than a stray mark — it produces two wrong signals from one mistake:

marks recorded    : [:c]
is :c a binding?  : false
is :C a binding?  : true
audit dangling    : [:c]
audit unaccounted : [:C]

So the author marks the callable, and the audit then reports that C is unaccounted for --
the exact opposite of what they declared, on the name they were declaring it about. A user who
trusts the audit is told to go document a thing they just marked.

Worth stating in the broken test so whoever implements it knows both halves have to move, not
only _signame.

5. Defect 2 points into this package, not at the caller

The description says the error "points at a line the author did not write". Reproduced, and it is
sharper than that — it points at a line inside ExperimentalAPI:

syntax: unsupported `const` declaration on local variable
  around .../ExperimentalAPI/src/mark.jl:207

The string experimental does not appear anywhere in the message. So the user experience of
misusing @experimental is a syntax error attributed to this package source, which reads as a bug
in the package rather than a misuse of it — the most expensive form this could take, since the
natural next step is to file an issue here.

Given that the package elsewhere spends real effort on messages that name the fix
(Name it instead: @experimental "why" Inner), this one is out of character rather than a missing
nicety.

---

Smaller

• test/spec/README.md is 56 lines and is not linked from the top-level README.md. Since it
  carries the rationale for the register, a one-line pointer would help a reader who lands on the
  repo root and finds 136 broken tests without context.
• test_spec_integration.jl asserts about docs, Aqua, releases, provenance and CI. Those have
  external dependencies (a docs build, a registry) that the other files do not. Worth confirming
  they can be made to run in CI at all before promoting them, so they do not become the group that
  stays broken because it cannot be otherwise.