fix: a mark is not a substitute for a docstring, and the docs examples now run

README.md

@@ -67,10 +67,10 @@ it at run time. One flag per marked *name*, so two methods of a marked name shar
 
 ## It is also a check
 
-Julia already checks half of the surface question: `Docs.undocumented_names` has been public API
-in Base since 1.11, and `Aqua.test_undocumented_names` ships it as a test — every public name must
-carry a docstring. What neither can express is the third answer: *this name is public, it has no
-docstring, and that is deliberate, and here is why*.
+**A mark is not a substitute for a docstring.** Every public name should have one; the mark is a
+second, independent account — the docstring says what the name does, the mark says whether its
+shape is settled. A marked name with no prose fails the check exactly as an unmarked one does,
+and there is no switch that turns that off.
 
 A marker nobody compares against anything is a claim. Put this in `runtests.jl` and it becomes a
 contract:
@@ -81,9 +81,8 @@ using MyPackage, ExperimentalAPI, Test
 ExperimentalAPI.test_surface(MyPackage)
 ```
 
-It fails, naming the symbol, when a public name has neither a docstring nor a mark — and also
-when a mark points at a name that was never made public, which is the module contradicting
-itself.
+It fails, naming the symbol, when a public name has no docstring — and also when a mark points at
+a name that was never made public, which is the module contradicting itself.
 
 Adopting it on a package that already has a backlog:
 
@@ -116,14 +115,20 @@ Attached to a definition, or as a list of names defined elsewhere:
 
 ```julia
 # the definition site
-@experimental "signature will be wrapped once the write-back refactor settles" \
-function ingest(config; doc, kwargs...)
-    # ...
-end
+@experimental(
+    "signature will be wrapped once the write-back refactor settles",
+    function ingest(config; doc, kwargs...)
+        # ...
+    end,
+)
 
 # names an included file defines
-@experimental "reads Test's internal result tree; not dogfooded in CI" \
-    render_test_report dump_test_report load_test_dump
+@experimental(
+    "reads Test's internal result tree; not dogfooded in CI",
+    render_test_report,
+    dump_test_report,
+    load_test_dump,
+)
 
 # with the issue where the shape is being decided
 @experimental("export format is a guess until someone consumes it",
@@ -199,7 +204,7 @@ public, which is the module contradicting itself and needs no reference to be wr
 | who may call a name | `export`, `public` (1.11) | orthogonal — a name can be public and unfinished |
 | a name on its way out | `@deprecate` | opposite direction |
 | type stability | DispatchDoctor | unrelated |
-| every public name has a docstring | `Docs.undocumented_names` (Base 1.11+), `Aqua.test_undocumented_names` | the same check, plus a third answer |
+| every public name has a docstring | `Docs.undocumented_names` (Base 1.11+), `Aqua.test_undocumented_names` | the same requirement, not a looser one — plus `foreign` and `dangling` |
 | generating documentation | Documenter | only ever checks whether prose exists |
 | run-time behaviour | — | one short-circuit read in the body; see the table above |
 

docs/src/adopting.md

@@ -15,7 +15,7 @@ julia> using MyPackage, ExperimentalAPI
 julia> audit(MyPackage)
 ```
 
-Two numbers matter. `unaccounted` is the backlog. `dangling` should be zero on day one, because
+Two numbers matter. `undocumented` is the backlog. `dangling` should be zero on day one, because
 there are no marks yet.
 
 ## Turn the test on with the backlog listed
@@ -44,19 +44,25 @@ means the only way the list changes is deliberately, and only downwards.
 
 ## Then, for each name, one decision
 
-For every name in the list, exactly one of two things is true, and both are cheap:
+For every name in the list, **write the docstring** — that part is not optional, and it is what
+deletes the `skip` entry. Then ask a second question: is the shape settled?
 
-- **it is settled** → write the docstring, delete the `skip` entry;
-- **it is not settled** → say so, with the reason, and delete the `skip` entry:
+- **settled** → nothing more to do;
+- **not settled** → say so, with the reason, *in addition* to the docstring:
 
 ```julia
-@experimental "reads Test's internal result tree; not dogfooded in CI" \
-    render_test_report dump_test_report load_test_dump
+@experimental(
+    "reads Test's internal result tree; not dogfooded in CI",
+    render_test_report,
+    dump_test_report,
+    load_test_dump,
+)
 ```
 
-The second is not a lesser outcome. A name that is genuinely unfinished is *better* described by
-a mark with a reason than by a docstring that has to pretend the shape is final — and it buys the
-right to change it, which the docstring does not.
+The mark is not a substitute for the prose and never was: the docstring says what the name does,
+which is owed either way. What the mark adds is the thing a docstring cannot carry — a machine
+can read it, so a run that goes through the name says so, and dropping the name later is not a
+breaking change.
 
 ## What "declare it" is worth later
 
@@ -71,4 +77,5 @@ Once the marks exist, three things follow that did not before:
 ## A note on where this ends
 
 The goal is not zero experimental names. A package with none is either finished or lying. The
-goal is zero **unaccounted** names — no public name about which nothing at all has been said.
+goal is zero **undocumented** names — every public name described, whether or not its shape has
+settled.

docs/src/checking.md

@@ -14,17 +14,21 @@ This is the part the rest of the package exists for.
 | bucket | meaning |
 |---|---|
 | `documented` | has a docstring |
-| `declared` | has an [`@experimental`](@ref) mark (may also be documented) |
+| `declared` | has an [`@experimental`](@ref) mark — an *additional* account, never a substitute |
 | `foreign` | public here, but bound in another package — not this module's to account for |
-| `unaccounted` | **neither documented nor declared** — the finding |
+| `undocumented` | **no docstring, marked or not** — the finding [`test_surface`](@ref) asserts empty |
+| `unaccounted` | neither account at all — a subset of `undocumented`, and the worst case |
 | `dangling` | marked, but not public — the module contradicting itself |
 
+A mark says the shape is unsettled. That is never a reason to say nothing about what the name
+does, so a marked name with no prose is a finding exactly as an unmarked one is.
+
 ```julia
 julia> audit(Archeion)
 Public surface of Archeion — 36 names
   documented      33
   experimental     0
-  unaccounted      3   ← neither documented nor @experimental
+  undocumented     3   ← no docstring
      FTPSTransport, pull_file, push_dir
 ```
 
@@ -86,10 +90,10 @@ The two checks are complementary and neither subsumes the other:
 makedocs(; modules = [MyPackage], checkdocs = :public)   # every public name has a docstring
 ```
 
-Documenter's `checkdocs = :public` fails a build when a public name has no docstring — but it has
-no notion of "declared unfinished instead", so a package that wants that third option needs both:
-`checkdocs` for the names that must be documented, `test_surface` for the rule that lets a mark
-stand in for a docstring.
+Documenter's `checkdocs = :public` fails a build when a public name has no docstring, and
+[`test_surface`](@ref) requires the same thing — the two agree, and running both is not a
+contradiction. What `test_surface` adds is `foreign` (a re-exported name whose prose is somebody
+else's job) and `dangling` (a mark on a name that was never made public), neither of which
+Documenter has a notion of.
 
-If you would rather run only one, run `test_surface`: it accepts everything `checkdocs = :public`
-accepts, plus marks.
+Run both. Neither is a looser version of the other.

docs/src/declaring.md

@@ -11,14 +11,20 @@ definition** or **a list of names**.
 
 ```julia
 # attached to the definition: the mark and the thing it describes cannot drift apart
-@experimental "signature will be wrapped once the write-back refactor settles" \
-function ingest(config; doc, kwargs...)
-    # ...
-end
+@experimental(
+    "signature will be wrapped once the write-back refactor settles",
+    function ingest(config; doc, kwargs...)
+        # ...
+    end,
+)
 
 # a list, for names an included file defines
-@experimental "reads Test's internal result tree; not dogfooded in CI" \
-    render_test_report dump_test_report load_test_dump
+@experimental(
+    "reads Test's internal result tree; not dogfooded in CI",
+    render_test_report,
+    dump_test_report,
+    load_test_dump,
+)
 ```
 
 The attached form is preferred where it fits, for the same reason a docstring goes above its
@@ -89,10 +95,12 @@ questions, and nothing forces a choice between them:
 
 Ingest `doc` into the registry described by `config`.
 """
-@experimental "signature will be wrapped once the write-back refactor settles" \
-function ingest(config; doc, kwargs...)
-    # ...
-end
+@experimental(
+    "signature will be wrapped once the write-back refactor settles",
+    function ingest(config; doc, kwargs...)
+        # ...
+    end,
+)
 ```
 
 ## Marking is not making public

docs/src/index.md

@@ -95,7 +95,7 @@ was never made public, which is the module contradicting itself.
 | who may call a name | `export`, `public` (1.11) | orthogonal |
 | a name on its way out | `@deprecate` | opposite direction |
 | type stability | DispatchDoctor | unrelated |
-| every public name has a docstring | `Docs.undocumented_names`, `Aqua.test_undocumented_names` | the same check, plus a third answer |
+| every public name has a docstring | `Docs.undocumented_names`, `Aqua.test_undocumented_names` | the same requirement, not a looser one — plus `foreign` and `dangling` |
 | generating documentation | Documenter | only ever checks whether prose exists |
 | run-time behaviour | — | one short-circuit read in the body — see the table above |
 | how often a path ran | `Profile`, `@time` | not answered: the default layer knows *whether*, never how often |

ext/ExperimentalAPITestExt.jl

@@ -16,21 +16,25 @@ function ExperimentalAPI.test_surface(
         # of findings, so on a clean module all of it collapses to nothing and the testset would
         # report `0 tests passed` — a green indistinguishable from the extension having failed to
         # load, or from `m` having no public names at all. These make the pass mean something.
-        @testset "nothing unaccounted for" begin
-            @test isempty(setdiff(a.unaccounted, skip))
+        @testset "every public name has a docstring" begin
+            @test isempty(setdiff(a.undocumented, skip))
             @test isempty(a.dangling)
         end
         # One testset per name, so a failing CI log names the symbol in its header rather than
         # printing a set difference the reader has to diff by eye.
-        @testset "$n is documented or @experimental" for n in setdiff(a.unaccounted, skip)
-            @test isdocumented(m, n) || isexperimental(m, n)
+        @testset "$n has a docstring" for n in setdiff(a.undocumented, skip)
+            @test isdocumented(m, n)
         end
-        # The allowlist can only shrink. An entry that has since been documented, declared or
-        # deleted fails here — otherwise adopting this on a package with a backlog would leave a
-        # list that silently stops describing anything, and a green suite would mean less every
-        # release.
+        # The allowlist can only shrink. An entry that has since been documented or deleted fails
+        # here — otherwise adopting this on a package with a backlog would leave a list that
+        # silently stops describing anything, and a green suite would mean less every release.
+        #
+        # `skip` is the only way to pass an undocumented name, and it is per-name and visible.
+        # There is deliberately no switch that turns the docstring requirement off wholesale: a
+        # mark records that a shape is unsettled, and that is never a reason to say nothing about
+        # what the name does.
         @testset "skip entry $n is still needed" for n in skip
-            @test n in a.unaccounted
+            @test n in a.undocumented
         end
         # Needs no oracle: the module marked a name it never made public.
         @testset "@experimental $n is public" for n in a.dangling

src/ExperimentalAPI.jl

@@ -20,8 +20,8 @@ The mark is three things, and the first is the reason to have it:
   * an **observation** — [`entered`](@ref) reports the marked definitions this run actually went
     through, and a summary says so at process exit whether or not anyone asked;
   * a **declaration** — the reason travels with the name, in the source, where the author is;
-  * a **check** — [`audit`](@ref) reports every public name that is neither documented nor
-    declared, so "document it or admit it is unfinished" becomes a test that fails.
+  * a **check** — [`audit`](@ref) reports every public name with no docstring, marked or not, so
+    "every public name is described" becomes a test that fails.
 
 # What it costs
 
@@ -41,7 +41,7 @@ See [`@experimental`](@ref) for the form-by-form table.
 |---|---|
 | did this run go through unvalidated code? | `entered()` — and the summary at exit says so anyway |
 | what is unfinished here? | `experimental(M)` |
-| what does this module owe nobody an explanation for? | `audit(M).unaccounted` — should be empty |
+| which public names are undescribed? | `audit(M).undocumented` — should be empty |
 | is dropping this name breaking? | `compare(old_snapshot, M)` — see [`isbreaking`](@ref) |
 
 # What this is not
@@ -57,10 +57,11 @@ See [`@experimental`](@ref) for the form-by-form table.
 
 # Scope of the check
 
-[`audit`](@ref) compares `names(M)` — exported *and* `public` names — against two accounts:
-a docstring, or a mark. It sees **names**, not signatures and not prose quality. A public name
-with a docstring reading "TODO" is accounted for; a settled name whose method signature changed
-under it is invisible here. See [`compare`](@ref) for the same limit on the release side.
+[`audit`](@ref) compares `names(M)` — exported *and* `public` names — against two independent
+accounts: a docstring, and a mark. They are not alternatives; the docstring is owed either way.
+It sees **names**, not signatures and not prose quality. A public name with a docstring reading
+"TODO" is accounted for; a settled name whose method signature changed under it is invisible here.
+See [`compare`](@ref) for the same limit on the release side.
 """
 module ExperimentalAPI
 
@@ -84,8 +85,13 @@ include("release.jl")   # a snapshot of the covenant, and what a diff of two of
 """
     test_surface(m::Module; skip = Symbol[], outputlevel::Int = 0) -> Audit
 
-Assert, as a `@testset`, that every public name of `m` is either documented or declared
-[`@experimental`](@ref) — and that every mark applies to a name that is actually public.
+Assert, as a `@testset`, that every public name of `m` has a **docstring** — and that every mark
+applies to a name that is actually public.
+
+A mark is not an alternative to prose. `@experimental` records that a shape is unsettled, which is
+never a reason to say nothing about what the name does, so a marked-but-undocumented name fails
+this test exactly as an unmarked one does. There is no switch to turn that off; `skip` is the only
+escape, and it is per-name, visible, and can only shrink.
 
 Available once `Test` is loaded (it lives in a package extension, so `ExperimentalAPI` itself
 never pulls `Test` into a runtime dependency). Put it in `runtests.jl`:
@@ -96,9 +102,9 @@ using MyPackage, ExperimentalAPI, Test
 ExperimentalAPI.test_surface(MyPackage)
 ```
 
-`skip` is for adopting this on a package that already has a backlog: the listed names are
-allowed to be unaccounted for. **A stale entry fails the test** — a name in `skip` that has since
-been documented, declared, or removed is reported, so the list can only shrink.
+`skip` is for adopting this on a package that already has a backlog: the listed names are allowed
+to have no docstring. **A stale entry fails the test** — a name in `skip` that has since been
+documented or removed is reported, so the list can only shrink.
 
 Returns the [`Audit`](@ref) on the normal return path whether the testset passed or not.
 `outputlevel ≥ 1` also prints it.