Skip to content

feat: report which experimental definitions a run entered - #9

Merged
sotashimozono merged 2 commits into
mainfrom
feat/observe-what-a-run-entered
Sep 3, 2026
Merged

feat: report which experimental definitions a run entered#9
sotashimozono merged 2 commits into
mainfrom
feat/observe-what-a-run-entered

Conversation

@sotashimozono

Copy link
Copy Markdown
Member

The package could say a name was unfinished. It could not say that this run went through one — which is the question a docstring structurally cannot answer, because it is asked after the run, about the run, often by somebody who wrote neither.

using ExperimentalAPI

@experimental "convergence not established below β ≈ 0.1" energy(β) = β * 1.0000001

energy(0.5)
$ julia sweep.jl
┌ ExperimentalAPI: this run entered 1 experimental definition
│   Main.energy — convergence not established below β ≈ 0.1
└ set ENV["EXPERIMENTALAPI_SUMMARY"] = "0" before `using` to silence this

On by default. Silent when nothing marked was entered — loading a package that has marks prints nothing. Carries the reason, not just the symbol. entered() returns the same thing as data.

What it costs, and why the design is what it is

10M calls of sqrt(abs(sin(x)cos(x) + exp(-|x|/1e6))), Julia 1.12.2, minimum of 7–9 trials:

emitted into the body 1 thread 8 threads counts correctly?
nothing 1.00× 1.00×
the flag emitted here 1.03× 0.985× yes
counter, plain shared Ref 1.03× 3.76× no
counter, global atomic 1.17× 4.87× yes
counter, per-thread atomic 1.12× 2.79× yes
@warn, guarded so it fires once 5.65× yes
@warn maxlog=1 59.57× yes

Two rows decided it:

  • A flag written once and only read afterwards stops dirtying the cache line, which is why it is free at eight threads while every counting scheme is not. The plain counter is also wrong: it recorded 95,406,048 of 160,000,000 calls, losing 40% to races.
  • The guarded @warn costs 5.65× even though it fires once — what stops the definition inlining is the call being in the body at all, not the warning being printed. That is why the notice is a summary at exit rather than a warning at the call.

Counting, call sites and paths stay opt-in and are not built.

Scope, measured form by form

form observed?
function f(x) … end, f(x) = …, f(x::T) where {T} = …, f(x)::R = … yes
a name list, struct, const, assignment no — declaration only
@generated function refused outright, as any macro-produced definition is

The @generated row is a correction: the docstring first said "declaration only", and running it showed the macro rejects it and points at the name-list form.

What the case matrix did

Eight behaviours reported Unexpected Pass and were promoted. Nothing else moved — no collateral failures, which is the whole reason the spec was written first.

One assertion was inverted rather than promoted. The spec used to require the expansion be identical to the bare definition; it now requires exactly one more statement whose head is || — a short-circuit read, not a store. WrapControl.@wrapping adds one statement too and it is a store, so the control can fire.

Three claims that stopped being true

"Calls are untouched", "emits the definition unchanged plus one push! at load time", and "costs nothing at run time" were accurate before this change and are not now. They appeared in the README, the module docstring, the macro docstring and docs/src/index.md; all four are corrected.

The README example now runs

It referenced a Model and a correction that do not exist. It is now a program that runs, and test/test_readme.jl executes the first ```julia block verbatim and checks the output the README quotes — so it cannot rot. Confirmed by breaking the README and watching the test report UndefVarError: Model` not defined`.

New docs page Observing, and entered / Entry / marked_modules / detecting / summary_text are public and documented.

174 behaviours, 61 operating (was 54). Suite: 532 pass, 152 broken, 0 failing.

🤖 Generated with Claude Code

The package could say a name was unfinished. It could not say that *this run* went through one —
which is the question a docstring structurally cannot answer, because it is asked after the run,
about the run, often by somebody who wrote neither.

`@experimental` now emits one statement into the body of a definition it attaches to: a
set-once flag. `entered()` reads them back, and a summary at process exit reports them whether or
not anyone asked. A definition the run never entered is absent, not reported with a count of zero.

The cost, measured over 10M calls of a numeric body on Julia 1.12.2, minimum of 7-9 trials:

  | emitted into the body       | 1 thread | 8 threads | counts correctly? |
  | nothing                     |  1.00x   |  1.00x    |         -         |
  | the flag emitted here       |  1.03x   |  0.985x   |        yes        |
  | counter, plain shared Ref   |  1.03x   |  3.76x    |      **no**       |
  | counter, global atomic      |  1.17x   |  4.87x    |        yes        |
  | @warn, guarded, fires once  |  5.65x   |     -     |        yes        |

Two rows decided it. A flag written once and only read afterwards stops dirtying the cache line,
which is why it is free at eight threads while every counting scheme is not — and the plain
counter is wrong as well as slow, recording 95,406,048 of 160,000,000 calls. The guarded `@warn`
costs 5.65x even though it fires once: what stops the definition inlining is the call being in the
body at all. Hence a summary at exit rather than a warning at the call.

Scope, verified form by form rather than asserted: `function`, `f(x) = …`, parametric and
return-type-annotated signatures are observed. A name list, `struct`, `const` and assignment are
declaration-only. `@generated` is refused outright, as any macro-produced definition is — the
docstring first claimed it was declaration-only, which measurement corrected.

Eight spec behaviours reported Unexpected Pass and were promoted; nothing else moved, which is
what the case matrix was for. One assertion was inverted rather than promoted: the spec used to
require that the expansion be byte-identical to the bare definition, and now requires exactly one
more statement whose head is `||` — a short-circuit read, not a store. `@wrapping` adds one
statement too and is a store, so the control can fire.

Three false claims went with it. "Calls are untouched", "emits the definition unchanged plus one
push! at load time" and "costs nothing at run time" were true before this commit and are not now;
they appeared in the README, the module docstring, the macro docstring and `docs/src/index.md`.

Also: the README's primary example ran nothing — it referenced a `Model` and a `correction` that
do not exist. It is now a program that runs, and `test/test_readme.jl` executes it verbatim and
checks the output the README quotes, so it cannot rot. Confirmed by breaking the README and
watching the test report `UndefVarError: Model not defined`.

174 behaviours, 61 operating. Suite: 532 pass, 152 broken.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sotashimozono sotashimozono self-assigned this Sep 3, 2026
@github-actions github-actions Bot added the enhancement New feature or request label Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

📚 Docs preview: https://codes.sota-shimozono.com/ExperimentalAPI.jl/previews/PR9/

(updates on each push to this PR)

@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.80519% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/detect.jl 93.65% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

`windows-latest` only: git checks tracked text out with CRLF there, so the fence in
`findfirst("```julia\n", md)` never matched and the reader threw "README.md has no ```julia
block".

Same defect as the generated-table comparison one commit earlier, in a file written after that
fix and without it. Verified by converting README.md to CRLF locally and watching all three
testsets pass, rather than by pushing and waiting for the runner.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sotashimozono
sotashimozono merged commit 10a80f5 into main Sep 3, 2026
13 checks passed
@sotashimozono
sotashimozono deleted the feat/observe-what-a-run-entered branch September 3, 2026 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant