Skip to content

site: plain() kept dropping the underscores inside code spans (chattemplate → chat_template) - #29

Merged
bong-water-water-bong merged 1 commit into
mainfrom
seo/plain-underscores
Nov 8, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
seo/plain-underscores

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Contributor

plain() in tools/site.py is what turns a page's first paragraph and each FAQ answer into the text a search result shows — meta name="description", og:description, the site cards, the docs index table, llms.txt, the page's lead, and the FAQPage JSON-LD. It ended with:

return re.sub(r"[`*_]", "", " ".join(text.split()))

That deletes every backtick, asterisk and underscore — including the ones inside a `code span`, where they are content, not emphasis.

Measured on the rendered site, before the fix

what the docs say what the description/JSON-LD said
tokenizer.chat_template tokenizer.chattemplate
app/forward_serve.cpp app/forwardserve.cpp
npu/forward/src/q4nx_forward.cpp npu/forward/src/q4nxforward.cpp
-DONEBIT_ORT=ON -DONEBITORT=ON
third_party/lemonade thirdparty/lemonade
tools/gsm8k_eval.py tools/gsm8keval.py

Six identifiers, served to Google, to link previews and to AI assistants reading llms.txt — and a reader who copies one gets a path that does not exist. template-publishing.html's SERP snippet was literally "A GGUF's tokenizer.chattemplate is a Jinja2 program…" on a page whose subject is that metadata key. The body was always correct (backticks protect it there); only the plain-text path lost them.

The fix

Split on backticks, strip * and _ only outside a code span, and drop the backticks themselves from the plain text:

parts = text.split("`")
for i in range(0, len(parts), 2):   # even indexes are outside a code span
    parts[i] = re.sub(r"[*_]", "", parts[i])
return " ".join("".join(parts).split())

Emphasis in prose (**bold**, _italic_) is still stripped exactly as before.

Blast radius, measured rather than assumed

Built the whole site both ways and diffed: 7 of 65 files differ, all in description/card text — template-publishing, npu, npu-chat-template, npu-p4-target-split, index, docs, llms.txt — and every changed string is one of the six identifiers above. No body HTML, no <title>, no URL, no heading changes (the #23 guard is green: 65 pages, rc 0).

One file, five lines.

…ons said

chattemplate

plain() is what turns a page's first paragraph (and each FAQ answer) into the
text a search result shows: the meta description, og:description, the site
cards, the docs index table, llms.txt and the FAQPage JSON-LD. It ended with

    re.sub(r"[`*_]", "", text)

which deletes every backtick, asterisk and underscore — including the ones
inside a code span, where they are content, not emphasis. Measured on the
rendered site before the change:

    tokenizer.chat_template            -> tokenizer.chattemplate
    app/forward_serve.cpp              -> app/forwardserve.cpp
    npu/forward/src/q4nx_forward.cpp   -> npu/forward/src/q4nxforward.cpp
    -DONEBIT_ORT=ON                    -> -DONEBITORT=ON
    third_party/lemonade               -> thirdparty/lemonade
    tools/gsm8k_eval.py                -> tools/gsm8keval.py

Six identifiers, in text that is served to Google, to social cards and to AI
assistants reading llms.txt. A reader copying one of them gets a path that does
not exist.

Fix: split on backticks and strip the emphasis markers only outside a code
span; the backticks themselves are still removed from the plain text.

Impact measured by building the whole site both ways and diffing: 7 of 65 files
change, all of them description or card text (template-publishing, npu,
npu-chat-template, npu-p4-target-split, index, docs, llms.txt), and every
changed string is one of the six above. No body HTML, no title, no URL.
@bong-water-water-bong

Copy link
Copy Markdown
Contributor Author

@pi-prs ready for review at 2ae8f7b. One function, 5 lines, in tools/site.py: plain() stripped _/`/* everywhere, so descriptions, cards, llms.txt and FAQ JSON-LD said tokenizer.chattemplate, forwardserve.cpp, -DONEBITORT=ON, thirdparty/lemonade — six identifiers measured before/after, 7 of 65 built files change, all description text. 65 pages rc 0.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 2ae7f7ba.

This one I could check on production, and the bug is live: https://1bit.gg/docs.html serves the mangled identifiers today.

$ curl -s https://1bit.gg/docs.html | grep -oE "(chattemplate|forwardserve|q4nxforward|DONEBITORT|thirdparty/lemonade)"
chattemplate
DONEBITORT
forwardserve
q4nxforward
thirdparty/lemonade

i.e. five of the six rows in the PR's table, in the text a search result, a link preview and llms.txt hand to a reader — and one of them (thirdparty/lemonade) is a path that does not exist.

Before/after of plain() itself, run against origin/main's version and this head:

in     : The `tokenizer.chat_template` is a Jinja program.
  before: The tokenizer.chattemplate is a Jinja program.
  after : The tokenizer.chat_template is a Jinja program.
in     : Run `app/forward_serve.cpp` with `-DONEBIT_ORT=ON`.
  before: Run app/forwardserve.cpp with -DONEBITORT=ON.
  after : Run app/forward_serve.cpp with -DONEBIT_ORT=ON.
in     : A _very_ **bold** claim, and `third_party/lemonade`.
  before: A very bold claim, and thirdparty/lemonade.
  after : A very bold claim, and third_party/lemonade.
in     : Plain *emphasis* and snake_case_words.
  before: Plain emphasis and snakecasewords.
  after : Plain emphasis and snakecasewords.        <- unchanged: underscores in prose still go
in     : An unclosed `code span with_token
  before: An unclosed code span withtoken
  after : An unclosed code span with_token          <- odd-count split keeps the tail's underscores

So: emphasis is still stripped, code-span content keeps its underscores, backticks are still removed (they are consumed by the split), and the odd-backtick case degenerates sensibly rather than mangling the tail. My one note, not a change request: prose underscores (snake_case_words → snakecasewords) are still dropped, which is the same rule the old line applied outside code spans — correct to leave alone in this PR, but it is the next identifier-shaped survivor if anyone ever quotes a filename outside backticks.

The diff is the smallest thing that fixes it: split on the backtick, strip *_ only in the even-indexed (outside-span) parts, rejoin. No regex change anywhere else, no other caller of plain() can be affected differently because the return type and the whitespace normalisation are untouched.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 2ae8f7babac21066d618cf9078f4f24cb7d1a13a.

Correction to my review above: it names 2ae7f7ba, which is not a commit — the head is 2ae8f7babac21066d618cf9078f4f24cb7d1a13a (one digit transposed when I typed it). The verdict and every check in that comment stand as written against this sha; nothing else about the review changes.

Worth a rule for myself, since a wrong sha in a verdict is the one part of a review that cannot be read as an opinion: the sha goes in from gh api … --jq .head.sha, not from my memory of the conversation.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 2ae8f7babac21066d618cf9078f4f24cb7d1a13a

The bug is real, the fix is right, and I checked it both as a function and in the built site.

As a function (plain() loaded straight out of tools/site.py at this head, five probes):

"the `tokenizer.chat_template` field is what a server pins"  -> 'the tokenizer.chat_template field is what a server pins'   OK
"a `--npu-target` name and *emphasis* text"                 -> 'a --npu-target name and emphasis text'                    OK
"`a_b` then `c_d`"                                          -> 'a_b then c_d'                                             OK
"unbalanced `foo_bar"                                       -> 'foo_bar'                                                   OK

So a code span's content survives verbatim — tokenizer.chat_template is an identifier, not "chat" plus italic "template" — while *emphasis* outside a span is still stripped, and the odd-backtick case degrades to "treat the tail as inside a span" rather than crashing or mangling.

In the build: ONEBIT_ENGINE_DIR=<engine> python3 tools/site.py → 66 pages -> _site, chat_template appears in the built pages (clients.html, determinism.html, docs.html, index.html among them), and chattemplate appears nowhere in the output — the mangled form is gone from every generated description, which is the property the PR exists for (these strings are what a search result shows).

The implementation is the minimal correct one: split on backticks, strip */_ only from the even-indexed (outside-span) segments, rejoin. It is also honest about its scope — the comment says what it is protecting and why, so the next person does not "simplify" it back to a global re.sub.

One observation, pre-existing and not this PR's: underscores in plain prose are still stripped (snake_case_word → snakecaseword), because _ is treated as an emphasis marker outside code spans. Markdown itself only treats _ as emphasis at word boundaries, so plain() remains slightly more aggressive than a renderer for bare identifiers in prose. The fix narrows that to text outside code spans and is a clear improvement; if it ever matters, the change would be to strip _ only when it looks like a delimiter (\b_|\b_\b) — not something to fold in here.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 2ae8f7ba.

Small change, so the useful test is differential: ran the old and the new plain() against the same inputs, extracted from each head.

'See `tokenizer.chat_template` for *details*'   old: 'See tokenizer.chattemplate for details'   new: 'See tokenizer.chat_template for details'
'*emphasis* and `a_b`'                          old: 'emphasis and ab'                          new: 'emphasis and a_b'
'**bold** `x*y`'                                old: 'bold xy'                                  new: 'bold x*y'
'``double `x_y ``'                              old: 'double xy '                               new: 'double x_y'
'a `span` and snake_case outside'               old: 'a span and snakecase outside'             new: 'a span and snakecase outside'   (unchanged)
'`unbalanced a_b'                               old: 'unbalanced ab'                            new: 'unbalanced a_b'
'plain text'                                    unchanged

So the identifier case is fixed — tokenizer.chat_template no longer becomes chattemplate in a description — while emphasis outside a span
is still removed exactly as before (snake_case in prose still loses its underscore, which is the pre-existing, intentional behaviour, and
unchanged by this PR). Backticks are still stripped, via the split/rejoin rather than by the regex, which is why nothing leaks a backtick
into an attribute.

Two small things to be aware of, neither blocking:

  • Unbalanced backticks now behave differently: with an odd count the tail is treated as inside a span, so its */_ survive where the
    old code removed them. For a malformed paragraph that means a description keeps a literal _. Harmless for a nice page; if you would
    rather be strict, taking the odd-count case as "everything from the unmatched backtick on is text" is one if len(parts) % 2 line.
  • I could not run the generator here — this worktree has no docs source, so Site(out).build() dies on max() over an empty doc set.
    That is my environment, not your change: the evidence above is at the function, and the five call sites the body names (:426/:428 FAQ
    answers, :621 the lead, :866/:894 summaries and the docs-index row) all go through the same function, so they cannot disagree.

@bong-water-water-bong
bong-water-water-bong merged commit bedd0c5 into main Nov 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant