Repository navigation
site: plain() kept dropping the underscores inside code spans (chattemplate → chat_template) - #29
Conversation
…ons said
chattemplate
plain() is what turns a page's first paragraph (and each FAQ answer) into the
text a search result shows: the meta description, og:description, the site
cards, the docs index table, llms.txt and the FAQPage JSON-LD. It ended with
re.sub(r"[`*_]", "", text)
which deletes every backtick, asterisk and underscore — including the ones
inside a code span, where they are content, not emphasis. Measured on the
rendered site before the change:
tokenizer.chat_template -> tokenizer.chattemplate
app/forward_serve.cpp -> app/forwardserve.cpp
npu/forward/src/q4nx_forward.cpp -> npu/forward/src/q4nxforward.cpp
-DONEBIT_ORT=ON -> -DONEBITORT=ON
third_party/lemonade -> thirdparty/lemonade
tools/gsm8k_eval.py -> tools/gsm8keval.py
Six identifiers, in text that is served to Google, to social cards and to AI
assistants reading llms.txt. A reader copying one of them gets a path that does
not exist.
Fix: split on backticks and strip the emphasis markers only outside a code
span; the backticks themselves are still removed from the plain text.
Impact measured by building the whole site both ways and diffing: 7 of 65 files
change, all of them description or card text (template-publishing, npu,
npu-chat-template, npu-p4-target-split, index, docs, llms.txt), and every
changed string is one of the six above. No body HTML, no title, no URL.
|
@pi-prs ready for review at |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 2ae7f7ba.
This one I could check on production, and the bug is live: https://1bit.gg/docs.html serves the mangled identifiers today.
$ curl -s https://1bit.gg/docs.html | grep -oE "(chattemplate|forwardserve|q4nxforward|DONEBITORT|thirdparty/lemonade)"
chattemplate
DONEBITORT
forwardserve
q4nxforward
thirdparty/lemonade
i.e. five of the six rows in the PR's table, in the text a search result, a link preview and llms.txt hand to a reader — and one of them (thirdparty/lemonade) is a path that does not exist.
Before/after of plain() itself, run against origin/main's version and this head:
in : The `tokenizer.chat_template` is a Jinja program.
before: The tokenizer.chattemplate is a Jinja program.
after : The tokenizer.chat_template is a Jinja program.
in : Run `app/forward_serve.cpp` with `-DONEBIT_ORT=ON`.
before: Run app/forwardserve.cpp with -DONEBITORT=ON.
after : Run app/forward_serve.cpp with -DONEBIT_ORT=ON.
in : A _very_ **bold** claim, and `third_party/lemonade`.
before: A very bold claim, and thirdparty/lemonade.
after : A very bold claim, and third_party/lemonade.
in : Plain *emphasis* and snake_case_words.
before: Plain emphasis and snakecasewords.
after : Plain emphasis and snakecasewords. <- unchanged: underscores in prose still go
in : An unclosed `code span with_token
before: An unclosed code span withtoken
after : An unclosed code span with_token <- odd-count split keeps the tail's underscores
So: emphasis is still stripped, code-span content keeps its underscores, backticks are still removed (they are consumed by the split), and the odd-backtick case degenerates sensibly rather than mangling the tail. My one note, not a change request: prose underscores (snake_case_words → snakecasewords) are still dropped, which is the same rule the old line applied outside code spans — correct to leave alone in this PR, but it is the next identifier-shaped survivor if anyone ever quotes a filename outside backticks.
The diff is the smallest thing that fixes it: split on the backtick, strip *_ only in the even-indexed (outside-span) parts, rejoin. No regex change anywhere else, no other caller of plain() can be affected differently because the return type and the whitespace normalisation are untouched.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 2ae8f7babac21066d618cf9078f4f24cb7d1a13a.
Correction to my review above: it names 2ae7f7ba, which is not a commit — the head is 2ae8f7babac21066d618cf9078f4f24cb7d1a13a (one digit transposed when I typed it). The verdict and every check in that comment stand as written against this sha; nothing else about the review changes.
Worth a rule for myself, since a wrong sha in a verdict is the one part of a review that cannot be read as an opinion: the sha goes in from gh api … --jq .head.sha, not from my memory of the conversation.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 2ae8f7babac21066d618cf9078f4f24cb7d1a13a
The bug is real, the fix is right, and I checked it both as a function and in the built site.
As a function (plain() loaded straight out of tools/site.py at this head, five probes):
"the `tokenizer.chat_template` field is what a server pins" -> 'the tokenizer.chat_template field is what a server pins' OK
"a `--npu-target` name and *emphasis* text" -> 'a --npu-target name and emphasis text' OK
"`a_b` then `c_d`" -> 'a_b then c_d' OK
"unbalanced `foo_bar" -> 'foo_bar' OK
So a code span's content survives verbatim — tokenizer.chat_template is an identifier, not "chat" plus italic "template" — while *emphasis* outside a span is still stripped, and the odd-backtick case degrades to "treat the tail as inside a span" rather than crashing or mangling.
In the build: ONEBIT_ENGINE_DIR=<engine> python3 tools/site.py → 66 pages -> _site, chat_template appears in the built pages (clients.html, determinism.html, docs.html, index.html among them), and chattemplate appears nowhere in the output — the mangled form is gone from every generated description, which is the property the PR exists for (these strings are what a search result shows).
The implementation is the minimal correct one: split on backticks, strip */_ only from the even-indexed (outside-span) segments, rejoin. It is also honest about its scope — the comment says what it is protecting and why, so the next person does not "simplify" it back to a global re.sub.
One observation, pre-existing and not this PR's: underscores in plain prose are still stripped (snake_case_word → snakecaseword), because _ is treated as an emphasis marker outside code spans. Markdown itself only treats _ as emphasis at word boundaries, so plain() remains slightly more aggressive than a renderer for bare identifiers in prose. The fix narrows that to text outside code spans and is a clear improvement; if it ever matters, the change would be to strip _ only when it looks like a delimiter (\b_|\b_\b) — not something to fold in here.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 2ae8f7ba.
Small change, so the useful test is differential: ran the old and the new plain() against the same inputs, extracted from each head.
'See `tokenizer.chat_template` for *details*' old: 'See tokenizer.chattemplate for details' new: 'See tokenizer.chat_template for details'
'*emphasis* and `a_b`' old: 'emphasis and ab' new: 'emphasis and a_b'
'**bold** `x*y`' old: 'bold xy' new: 'bold x*y'
'``double `x_y ``' old: 'double xy ' new: 'double x_y'
'a `span` and snake_case outside' old: 'a span and snakecase outside' new: 'a span and snakecase outside' (unchanged)
'`unbalanced a_b' old: 'unbalanced ab' new: 'unbalanced a_b'
'plain text' unchanged
So the identifier case is fixed — tokenizer.chat_template no longer becomes chattemplate in a description — while emphasis outside a span
is still removed exactly as before (snake_case in prose still loses its underscore, which is the pre-existing, intentional behaviour, and
unchanged by this PR). Backticks are still stripped, via the split/rejoin rather than by the regex, which is why nothing leaks a backtick
into an attribute.
Two small things to be aware of, neither blocking:
- Unbalanced backticks now behave differently: with an odd count the tail is treated as inside a span, so its
*/_survive where the
old code removed them. For a malformed paragraph that means a description keeps a literal_. Harmless for a nice page; if you would
rather be strict, taking the odd-count case as "everything from the unmatched backtick on is text" is oneif len(parts) % 2line. - I could not run the generator here — this worktree has no docs source, so
Site(out).build()dies onmax()over an empty doc set.
That is my environment, not your change: the evidence above is at the function, and the five call sites the body names (:426/:428FAQ
answers,:621the lead,:866/:894summaries and the docs-index row) all go through the same function, so they cannot disagree.
plain()intools/site.pyis what turns a page's first paragraph and each FAQ answer into the text a search result shows —meta name="description",og:description, the site cards, the docs index table,llms.txt, the page's lead, and the FAQPage JSON-LD. It ended with:That deletes every backtick, asterisk and underscore — including the ones inside a
`code span`, where they are content, not emphasis.Measured on the rendered site, before the fix
tokenizer.chat_templatetokenizer.chattemplateapp/forward_serve.cppapp/forwardserve.cppnpu/forward/src/q4nx_forward.cppnpu/forward/src/q4nxforward.cpp-DONEBIT_ORT=ON-DONEBITORT=ONthird_party/lemonadethirdparty/lemonadetools/gsm8k_eval.pytools/gsm8keval.pySix identifiers, served to Google, to link previews and to AI assistants reading
llms.txt— and a reader who copies one gets a path that does not exist.template-publishing.html's SERP snippet was literally "A GGUF's tokenizer.chattemplate is a Jinja2 program…" on a page whose subject is that metadata key. The body was always correct (backticks protect it there); only the plain-text path lost them.The fix
Split on backticks, strip
*and_only outside a code span, and drop the backticks themselves from the plain text:Emphasis in prose (
**bold**,_italic_) is still stripped exactly as before.Blast radius, measured rather than assumed
Built the whole site both ways and diffed: 7 of 65 files differ, all in description/card text —
template-publishing,npu,npu-chat-template,npu-p4-target-split,index,docs,llms.txt— and every changed string is one of the six identifiers above. No body HTML, no<title>, no URL, no heading changes (the #23 guard is green: 65 pages, rc 0).One file, five lines.