release: 0.3.0 — beta, with five verifier-accuracy fixes

CHANGELOG.md

@@ -7,14 +7,66 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-Accuracy fixes from the 2026-08-10 library audit: three numeric-claim
-false-positive classes stopped, two silent false-negative classes closed
-(short count labels, single-span/fenced CLI flags), markdown link titles
-handled, and the semantic layer now judges against real source passages
-instead of the content itself. Every fix carries a regression corpus case.
+## [0.3.0] - 2026-08-11
+
+**Beta.** The deterministic core is stable and the public API is now covered
+by a compatibility promise (see the README status section). Two releases'
+worth of accuracy work lands here: the 2026-08-10 audit fixes plus a
+beta-review pass that closed two more silent false-negative classes in the
+fence extractor and two link false positives.
+
+### Changed
+
+- **`Development Status :: 4 - Beta`** (was `2 - Pre-Alpha`). `verify`,
+  `VerifyContext`, `VerifyResult`, `Finding`, `FindingKind`,
+  `raise_if_failed` and the `Judge` protocol will not change shape without a
+  deprecation in a minor release.
+
+### Added
+
+- **`py.typed`.** The package is fully annotated but shipped no PEP 561
+  marker, so type checkers ignored it in downstream projects.
+- **Packaging guards.** `__version__` and the `pyproject.toml` version are
+  pinned equal by a test — the release flow bumps both by hand, and drift
+  would ship a wheel whose metadata disagrees with the runtime value.
+- **README:** a per-checker table of what each checker settles and which
+  truth source it needs, plus an explicit known-limitations list.
 
 ### Fixed
 
+- **Indented code fences are no longer invisible.** A fence nested under a
+  list item — how LLMs routinely write install steps — matched nothing, so
+  every import and shell flag inside one passed unchecked. The extractor is
+  now a line scanner: it accepts a fence at any indentation and strips that
+  indent from the body (uniformly indented code otherwise fails `ast.parse`,
+  which was the silent skip). This was the largest remaining hole.
+- **Tilde fences (`~~~`) are extracted.** CommonMark-legal and previously
+  unrecognized — another silent pass.
+- **Percent-encoded link targets resolve.** A link to a file whose name
+  contains a space is written `docs/my%20file.md`; that was checked
+  literally and flagged a file that exists. The raw form is still tried
+  first, so a file genuinely named `a%20b.md` resolves, and a decoded
+  target that escapes `project_root` is still caught.
+- **Balanced parentheses in link targets are not truncated.**
+  `[doc](docs/a(1).md)` was cut to `docs/a(1)` and flagged.
+- **Link syntax shown as an example is no longer checked as a link.** A doc
+  documenting its own conventions ("write it as `` `[text](target.md)` ``")
+  was flagged for a target it never claimed existed — no renderer resolves a
+  link inside a code span or fence. Links are now read from prose only —
+  spans of any delimiter width, so a doubled delimiter around a span that
+  itself contains backticks is masked whole rather than at its edges. Line
+  numbers are unaffected, and a real link sharing a line with an example is
+  still checked. Found by running verify over its own README.
+- The fence scanner also enforces the rules the old regex ignored: an
+  unclosed fence is not a fence (its body was the rest of the document), a
+  closing run must match the opening character and length, and an inline
+  ``` ```code``` ``` span no longer opens one.
+- Stripping fences before the flag scan now blanks the lines in place
+  rather than deleting them, so prose either side of a code block never
+  becomes adjacent when the checker looks backwards for a command name.
+
+### Fixed (2026-08-10 audit)
+
 - **Comma-grouped numbers are one claim.** "1,234 tests" previously
   extracted the "234" fragment and flagged an error-severity count
   mismatch against `tests=1234`; `1,234` is now extracted as the single
@@ -55,7 +107,7 @@ instead of the content itself. Every fix carries a regression corpus case.
   judge was provided"; the message now names the failing object and the
   protocol mismatch.
 
-### Changed
+### Changed (2026-08-10 audit)
 
 - Import resolution passes the module name to the child interpreter via
   `argv` instead of f-string interpolation into the `-c` program —

README.md

@@ -41,12 +41,39 @@ if not result.ok:
 Together they bracket generation: rag verifies *"is this claim supported?"*;
 verify checks *"does this named thing exist?"*
 
+## What each checker verifies
+
+| Checker | Claim it settles | Truth source you declare |
+|---|---|---|
+| imports | every import in a Python code fence resolves, by full dotted path | `env_python` |
+| flags | every `--flag` in an inline span or shell fence appears in that command's `--help` | `help_commands` / `allowed_help_cmds` |
+| links | every local markdown link target exists under the project | `project_root` |
+| counts | every numeric claim matches the number it names | `count_sources` |
+
+Findings are `error` when a claim is refuted and `warning` when it cannot be
+checked — an unverifiable claim is never a silent pass. `result.ok` is False
+only on errors; `raise_if_failed(result)` turns it into a hard gate.
+
 ## Status
 
-Alpha — the deterministic core (imports, flags, links, counts) is shipped
-and guarded by a labeled precision/recall corpus (gated ≥ 0.95 each) and
-mutation testing (gated ≥ 0.75). The LLM semantic layer is optional via
-the `[rag]` extra.
+Beta — the deterministic core (imports, flags, links, counts) is stable and
+guarded by a labeled precision/recall corpus (gated ≥ 0.95 each) and mutation
+testing (gated ≥ 0.75). The public API above (`verify`, `VerifyContext`,
+`VerifyResult`, `Finding`, `FindingKind`, `raise_if_failed`, and the `Judge`
+protocol) is what beta covers: it will not change shape without a deprecation
+in a minor release. The LLM semantic layer is optional via the `[rag]` extra
+and is the least settled part of the surface.
+
+### Known limitations
+
+- Reference-style links (`[text][ref]`) are not resolved — only inline
+  `[text](target)` links are checked.
+- Short flags (`-v`) are not checked; only `--long` forms are.
+- A flag written as bare `` `--flag` `` in prose is attributed to the nearest
+  preceding word, so it may degrade to a warning rather than resolve.
+- Counts are matched to a source by keyword overlap with the claim's
+  surrounding text; a numeric claim whose context names no source is skipped
+  rather than guessed at.
 
 ## License
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "attune-verify"
-version = "0.2.2"
+version = "0.3.0"
 description = "Generation fact-checker for the attune-* family. Verifies named entities in LLM output actually exist — imports import, CLI flags are real, links resolve, counts match source."
 readme = {file = "README.md", content-type = "text/markdown"}
 requires-python = ">=3.10"
@@ -17,7 +17,7 @@ keywords = [
     "llm", "attune", "grounding", "faithfulness",
 ]
 classifiers = [
-    "Development Status :: 2 - Pre-Alpha",
+    "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
     "License :: OSI Approved :: Apache Software License",
     "Programming Language :: Python :: 3",
@@ -50,6 +50,11 @@ Repository = "https://github.com/Smart-AI-Memory/attune-verify"
 [tool.setuptools.packages.find]
 where = ["src"]
 
+[tool.setuptools.package-data]
+# PEP 561: without this the marker is not installed and type checkers ignore
+# the package's annotations entirely.
+attune_verify = ["py.typed"]
+
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 asyncio_mode = "auto"

src/attune_verify/__init__.py

@@ -23,7 +23,7 @@
     raise_if_failed,
 )
 
-__version__ = "0.2.2"
+__version__ = "0.3.0"
 __all__ = [
     "verify",
     "VerifyContext",

src/attune_verify/_extract.py

@@ -34,14 +34,19 @@ class NumericClaim:
     line: Optional[int] = None
 
 
-# The opening fence may carry an info string after the language word
-# (```python title="ex.py") — [^\n]* consumes it so those fences are still
-# extracted; only the leading word is the language.
-_FENCE_RE = re.compile(
-    r"^```(\w*)[^\n]*\n(.*?)^```",
-    re.MULTILINE | re.DOTALL,
-)
-_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)")
+# An opening fence: optional indent, a run of 3+ backticks or tildes, then an
+# info string. The info string may carry more than the language word
+# (```python title="ex.py") — only the leading word is the language.
+_FENCE_OPEN_RE = re.compile(r"^([ \t]*)(`{3,}|~{3,})(.*)$")
+# A link target may contain one level of balanced parentheses — 'docs/a(1).md'
+# is a legal CommonMark target, and [^)]+ truncated it to 'docs/a(1', flagging
+# a file that exists.
+_LINK_RE = re.compile(r"\[([^\]]+)\]\(((?:[^()]|\([^()]*\))*)\)")
+# One inline code span — its contents are shown, not claimed. The delimiter is
+# a run of backticks closed by a run of the same length, so a span that itself
+# contains backticks (``` ``a `b` c`` ```) is masked whole rather than leaving
+# its middle exposed as prose.
+_INLINE_CODE_RE = re.compile(r"(?<!`)(`{1,3})(?!`)[^\n]*?(?<!`)\1(?!`)")
 # 2+ digit numbers (skip single digits). Comma-grouped values ("1,234") are one
 # claim — the first alternative captures the whole group before the bare \d{2,}
 # can grab a fragment. Digit runs touching a decimal point ("94.53", the "10"
@@ -53,33 +58,142 @@ class NumericClaim:
 _LINK_TITLE_RE = re.compile(r"""^(\S+)\s+("[^"]*"|'[^']*'|\([^)]*\))$""")
 
 
-def extract_code_fences(content: str) -> List[CodeFence]:
-    """Extract all fenced code blocks from markdown content."""
-    fences = []
-    for match in _FENCE_RE.finditer(content):
-        line = content[: match.start()].count("\n") + 1
-        # A bare fence keeps language "" — downstream checkers decide how to
-        # treat untagged blocks (the import checker parses them speculatively).
-        fences.append(
-            CodeFence(
-                language=match.group(1),
-                content=match.group(2),
-                line=line,
+@dataclass
+class _FenceSpan:
+    """One fence located in the content, by 0-based line index."""
+
+    open_index: int
+    close_index: int
+    language: str
+    body: List[str]
+
+
+def _iter_fence_spans(content: str) -> List[_FenceSpan]:
+    """Locate every closed code fence, line by line.
+
+    A line scan rather than one regex, because a fence is defined by
+    properties a single pattern reads poorly: the closing run must use the
+    same character and be at least as long as the opening one, and an indented
+    fence (a code block nested under a list item — routine in LLM-written
+    docs) carries that indent into every body line.
+
+    An unclosed fence is not a fence: its "body" is the rest of the document,
+    so treating it as code would drag ordinary prose into the checkers.
+    """
+    lines = [line.rstrip("\r") for line in content.split("\n")]
+    spans: List[_FenceSpan] = []
+    index = 0
+    while index < len(lines):
+        opening = _FENCE_OPEN_RE.match(lines[index])
+        if opening is None:
+            index += 1
+            continue
+        indent, marker, info = opening.groups()
+        # A tilde fence's info string is unrestricted; a backtick fence's must
+        # not contain a backtick, else ``` `code` in prose ``` opens a fence.
+        if marker[0] == "`" and "`" in info:
+            index += 1
+            continue
+        close_re = re.compile(rf"^[ \t]*{re.escape(marker[0])}{{{len(marker)},}}[ \t]*$")
+        close_index = next(
+            (j for j in range(index + 1, len(lines)) if close_re.match(lines[j])),
+            None,
+        )
+        if close_index is None:
+            index += 1
+            continue
+        spans.append(
+            _FenceSpan(
+                open_index=index,
+                close_index=close_index,
+                # A bare fence keeps language "" — downstream checkers decide
+                # how to treat untagged blocks (the import checker parses them
+                # speculatively).
+                language=_language_of(info),
+                body=[_strip_indent(line, len(indent)) for line in lines[index + 1 : close_index]],
             )
         )
-    return fences
+        index = close_index + 1
+    return spans
+
+
+def _language_of(info: str) -> str:
+    """Return the leading language word of a fence info string.
+
+    An info string may carry more than the language (```python title="ex.py"),
+    and the language itself may be followed by punctuation.
+    """
+    first = info.strip().split(maxsplit=1)
+    return re.match(r"\w*", first[0]).group(0) if first else ""
+
+
+def _strip_indent(line: str, width: int) -> str:
+    """Remove up to ``width`` leading spaces/tabs — the fence's own indent.
+
+    Without this, a fence nested under a list item yields uniformly indented
+    code that fails ``ast.parse``, so every import inside it went unchecked.
+    """
+    removed = 0
+    while removed < width and line[:1] in (" ", "\t"):
+        line = line[1:]
+        removed += 1
+    return line
+
+
+def extract_code_fences(content: str) -> List[CodeFence]:
+    """Extract all fenced code blocks from markdown content.
+
+    Backtick and tilde fences are both recognized, at any indentation; a
+    fence's own indent is stripped from its body so nested blocks parse.
+    """
+    return [
+        CodeFence(
+            language=span.language,
+            content="".join(f"{line}\n" for line in span.body),
+            line=span.open_index + 1,
+        )
+        for span in _iter_fence_spans(content)
+    ]
+
+
+def strip_code_fences(content: str) -> str:
+    """Blank out every fence, keeping line count and prose offsets intact.
+
+    Fence lines become empty rather than disappearing, so prose either side of
+    a block never becomes adjacent — a checker looking backwards for context
+    must not read across a code block it was told to ignore.
+    """
+    lines = content.split("\n")
+    for span in _iter_fence_spans(content):
+        for index in range(span.open_index, span.close_index + 1):
+            lines[index] = ""
+    return "\n".join(lines)
+
+
+def _mask_code(content: str) -> str:
+    """Blank code fences and inline spans, preserving every line break.
+
+    Link syntax shown as an example — ``Write it as `[text](target.md)` `` —
+    is not a link: no renderer resolves it, so checking it flags a target that
+    was never claimed to exist. Masking keeps line offsets intact, so a link's
+    reported line number is still its line in the original content.
+    """
+    masked = strip_code_fences(content)
+    return _INLINE_CODE_RE.sub(lambda m: " " * len(m.group(0)), masked)
 
 
 def extract_links(content: str) -> List[MarkdownLink]:
-    """Extract all markdown links from content.
+    """Extract all markdown links from prose.
 
-    Targets are normalized: an optional markdown title
-    (``docs/a.md "Read me"``) is stripped and ``<angle-bracket>`` wrapping is
-    removed, so checkers see only the path.
+    Links inside code fences or inline code spans are example syntax, not
+    claims, and are skipped. Targets are normalized: an optional markdown
+    title (``docs/a.md "Read me"``) is stripped and ``<angle-bracket>``
+    wrapping is removed, so checkers see only the path.
     """
     links = []
-    for match in _LINK_RE.finditer(content):
-        line = content[: match.start()].count("\n") + 1
+    prose = _mask_code(content)
+    for match in _LINK_RE.finditer(prose):
+        line = prose[: match.start()].count("\n") + 1
         links.append(
             MarkdownLink(
                 text=match.group(1),

src/attune_verify/checkers/flags.py

@@ -6,7 +6,7 @@
 import subprocess
 from typing import Dict, FrozenSet, List, Optional
 
-from attune_verify._extract import _FENCE_RE, extract_code_fences
+from attune_verify._extract import extract_code_fences, strip_code_fences
 from attune_verify.result import Finding, FindingKind
 
 # One inline code span (`mytool --flag`); fences are handled separately.
@@ -40,7 +40,7 @@ def check_flags(
     # Inline spans: `--flag` alone or a whole command in one span
     # (`mytool --flag`). Fence bodies are stripped first so they are never
     # double-scanned as inline code.
-    prose = _FENCE_RE.sub("", content)
+    prose = strip_code_fences(content)
     for match in _INLINE_CODE_RE.finditer(prose):
         span = match.group(1)
         for flag_match in _FLAG_TOKEN_RE.finditer(span):