From df6d16587c9fe075440f2192372ebd713e3e8a57 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Fri, 12 Jun 2026 09:04:56 +0200 Subject: [PATCH 01/11] Fix noweb quoting of bracket-notation examples The prose examples like [[<>=]] made noweave parse the inner <<...>> as a real chunk reference: the raw underscore in the chunk name reached LaTeX unescaped (breaking compilation with "Missing $ inserted") and the reference linked to a never-defined chunk. Examples quoting the [[...]] notation itself, like [[[[...]]]], truncated at the first ]] and left stray bracket text in the PDF. Literal chunk syntax is now written with noweb's @<<...@>> escape inside [[...]], and literal double brackets with \texttt and split brackets, so the document compiles again and the examples render as intended. Co-Authored-By: Claude Fable 5 --- noweb.mk.nw | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/noweb.mk.nw b/noweb.mk.nw index 266a40e..bc599c9 100644 --- a/noweb.mk.nw +++ b/noweb.mk.nw @@ -123,17 +123,19 @@ Makefile rules. Many programming languages (particularly Python) use underscores in filenames, such as [[module_name.py]] or [[attachment_cache.py]]. -When these filenames appear in chunk definitions like [[<>=]], +When these filenames appear in chunk definitions like +[[@<>=]], LaTeX interprets the underscores as subscript commands during documentation generation (weaving), causing compilation errors. The error manifests as \enquote{Missing \$ inserted} because LaTeX expects math mode for subscripts. -\paragraph{The [[[[...]]]] notation solution} +\paragraph{The \texttt{[{}[\ldots]{}]} notation solution} -Noweb provides the [[[[...]]]] notation specifically to handle special +Noweb provides the \texttt{[{}[\ldots]{}]} notation specifically to handle special characters in code references. -When we write [[<<[[module_name.py]]>>=]] in a [[.nw]] file, noweb +When we write \texttt{<{}<[{}[module\_name.py]{}]>{}>=} in a [[.nw]] file, +noweb automatically escapes all LaTeX special characters (\_, \&, \%, etc.) when weaving documentation. The brackets tell noweb: \enquote{treat this as code, not LaTeX}. @@ -141,15 +143,16 @@ The brackets tell noweb: \enquote{treat this as code, not LaTeX}. \paragraph{Why we use it in Makefile rules} Since our Makefile rules must match the chunk names used in [[.nw]] files, and -we want all Python files to use [[[[...]]]] notation (to handle underscores), -we must specify [[-R"[[filename]]"]] in the notangle command. -The double quotes protect the brackets from shell interpretation, and notangle -then looks for a chunk named [[[[filename]]]]. +we want all Python files to use \texttt{[{}[\ldots]{}]} notation (to handle underscores), +we must specify \texttt{-R"[{}[filename]{}]"} in the notangle command. +The double quotes protect the brackets from shell interpretation, and notangle +then looks for a chunk named \texttt{[{}[filename]{}]}. This standardization means: \begin{enumerate} -\item All Python chunk definitions use [[<<[[filename.py]]>>=]] -\item All Makefile rules use [[-R"[[$(notdir $@)]]"]] +\item All Python chunk definitions use +\texttt{<{}<[{}[filename.py]{}]>{}>=} +\item All Makefile rules use \texttt{-R"[{}[\$(notdir \$@)]{}]"} \item Underscores work without escaping \item Consistent pattern across the project \end{enumerate} @@ -167,17 +170,17 @@ complexity. Python conventions where [[module_name]] is idiomatic. \end{description} -The [[[[...]]]] notation approach handles all special characters uniformly and +The \texttt{[{}[\ldots]{}]} notation approach handles all special characters uniformly and keeps [[.nw]] files readable. We will use notangle(1). -Note that we use the noweb [[[[...]]]] notation to quote the chunk name. +Note that we use the noweb \texttt{[{}[\ldots]{}]} notation to quote the chunk name. This is critical for handling filenames with underscores (common in Python) or other LaTeX special characters. -Without the brackets, chunk names like [[<>]] would cause +Without the brackets, chunk names like [[@<>]] would cause LaTeX to interpret the underscore as a subscript command, breaking documentation generation. -The [[[[...]]]] notation tells noweb to escape all special characters properly. +The \texttt{[{}[\ldots]{}]} notation tells noweb to escape all special characters properly. <>= NOTANGLEFLAGS?= From f9557b7cc912a357028fad57a253721bd250a645 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Fri, 12 Jun 2026 09:05:15 +0200 Subject: [PATCH 02/11] Weave highlighted documentation by default NOWEAVE.tex now weaves through noweb's autolang and tominted filters (dbosk noweb fork): each chunk is syntax-highlighted with minted using a per-chunk lexer, and -autodefs python3 fills the identifier index with -autolang gating the autodefs filters so chunks in other languages stay out of the index. The chunk-name-equals-filename convention our tangling rules already enforce is what drives the language inference. The bundled Pygments lexer keeps chunk references hyperlinked inside Python strings; noweb.mk provides a rule copying it from noweb's library directory (read from the LIB= line of the noweave script), and documents declare a dependency on it, as makefiles.pdf now does. Documents must load minted and compile with -shell-escape, which our own build already did. NOWEAVE.pdf keeps the classic rendering: its preamble is generated by noweave and cannot load minted. Everything is set with ?=, so projects without the fork override NOWEAVEFLAGS.tex. Co-Authored-By: Claude Fable 5 --- .gitignore | 1 + Makefile | 2 ++ Makefile.nw | 7 +++++ noweb.mk | 9 +++++- noweb.mk.nw | 89 +++++++++++++++++++++++++++++++++++++++++++++++++---- 5 files changed, 101 insertions(+), 7 deletions(-) diff --git a/.gitignore b/.gitignore index 191cadc..ddf0af8 100644 --- a/.gitignore +++ b/.gitignore @@ -17,3 +17,4 @@ transform.tex Dockerfile.tex intro.tex Makefile.tex +noweb_lexer.py diff --git a/Makefile b/Makefile index db0c4c1..5815d05 100644 --- a/Makefile +++ b/Makefile @@ -25,6 +25,7 @@ makefiles.pdf: exam.bib makefiles.pdf: transform.bib makefiles.pdf: tex.bib makefiles.pdf: Dockerfile.tex +makefiles.pdf: noweb_lexer.py define makefiles_depends makefiles.pdf: $(1:.mk=.tex) $(1) $(1:.mk=.tex): $(1).nw @@ -60,6 +61,7 @@ all: makefiles.tar.gz clean: ${RM} makefiles.pdf ${RM} Dockerfile.tex + ${RM} noweb_lexer.py ${RM} ${MKFILES:.mk=.tex} ${RM} gitattributes ${RM} makefiles.tar.gz diff --git a/Makefile.nw b/Makefile.nw index cfab450..980725b 100644 --- a/Makefile.nw +++ b/Makefile.nw @@ -82,10 +82,17 @@ makefiles.pdf: exam.bib makefiles.pdf: transform.bib makefiles.pdf: tex.bib makefiles.pdf: Dockerfile.tex +@ The default weave of [[noweb.mk]] highlights chunks with minted through +a custom Pygments lexer, which minted loads from the directory where +LaTeX runs; [[noweb.mk]] provides the rule that copies the lexer here, +we only declare the dependency. +<>= +makefiles.pdf: noweb_lexer.py @ We must add the generated files to the clean recipe. <>= ${RM} makefiles.pdf ${RM} Dockerfile.tex +${RM} noweb_lexer.py @ We want to use the PythonTeX and Minted packages. diff --git a/noweb.mk b/noweb.mk index d40e56f..7e1c03d 100644 --- a/noweb.mk +++ b/noweb.mk @@ -2,7 +2,12 @@ ifndef NOWEB_MK NOWEB_MK = true NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ -NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -x -n -delay -t2 +NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ + -autodefs python3 -index \ + -filter 'tominted -lexer ${NOWEB_LEXER}' +NOWEB_LEXER?= noweb_lexer.py +NOWEB_LIB?= $(shell sed -n 's/^LIB=//p' \ + $(shell command -v noweave) | head -1) NOWEAVE.pdf?= \ noweave ${NOWEAVEFLAGS.pdf} $< > ${@:.pdf=.tex} && \ latexmk -pdf ${@:.pdf=.tex} @@ -81,6 +86,8 @@ define def_weave_to_tex endef $(foreach suf,${NOWEB_SUFFIXES},$(eval $(call def_weave_to_tex,${suf}))) +${NOWEB_LEXER}: + cp ${NOWEB_LIB}/noweb_lexer.py $@ define with_suffix_target %$(1): %$(1).nw $${NOTANGLE$$(suffix $$@)} diff --git a/noweb.mk.nw b/noweb.mk.nw index bc599c9..47021b8 100644 --- a/noweb.mk.nw +++ b/noweb.mk.nw @@ -14,6 +14,16 @@ NOWEB file ([[.nw]]). There is also a [[NOWEAVE.pdf]] to weave directly to PDF. This assumes that the file is independent, \ie no special LaTeX preamble. +By default, [[NOWEAVE.tex]] weaves syntax-highlighted chunks through +noweb's [[tominted]] filter with a language-aware identifier index. +This requires noweb with the [[autolang]] and [[tominted]] filters (the +dbosk fork), a preamble that loads the minted package, LaTeX run with +[[-shell-escape]], and the document depending on [[noweb_lexer.py]] +(a rule for which this include provides). +The implementation section motivates each flag; everything is set with +[[?=]], so a project that lacks the fork falls back by setting +[[NOWEAVEFLAGS.tex]] itself. + \section{Implementation} @@ -40,24 +50,63 @@ endif We will use the [[noweave]] command to weave the documentation. We are interested in two cases: \begin{enumerate} -\item when a source program should be converted to TeX to be included in a +\item when a source program should be converted to TeX to be included in a larger document, and \item when a source program is independent and should be converted to PDF. \end{enumerate} +The two cases weave differently. +In the first case the including document controls the preamble, so it can +load the minted package and we weave syntax-highlighted chunks through the +[[tominted]] filter. +In the second case [[noweave]] generates the preamble itself, which loads +only the noweb package, so the highlighted rendering cannot compile and we +keep noweb's classic rendering. The order of the rules are important. -To ensure make takes the \enquote{shortcut} of the second case, we must specify +To ensure make takes the \enquote{shortcut} of the second case, we must specify that rule first. <>= <> <> +<> @ Now, for the first case, we let <>= NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ -NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -x -n -delay -t2 -@ Now we need to specify all the suffixes to use and then construct suffix rules +NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ + -autodefs python3 -index \ + -filter 'tominted -lexer ${NOWEB_LEXER}' +@ These flags weave syntax-highlighted documentation with a language-aware +identifier index; they require noweb with the [[autolang]] and [[tominted]] +filters (the dbosk fork) and the Icon autodefs filters. + +The [[-autolang]] option annotates every chunk with its language, inferred +from filename-like chunk names. +Our tangling rules below already force that naming---[[NOTANGLE]] extracts +the chunk named exactly after the target file---so the convention pays +twice: it lets make find the chunk and it lets noweb identify the language. +The annotations make the autodefs filters skip chunks in languages they do +not understand, so [[-autodefs python3]] fills the identifier index without +collecting make or shell assignments from a project's embedded [[Makefile]] +chunks (their assignments match the Python assignment pattern). +We use [[-index]] rather than the previous [[-x]] because the autodefs +output feeds the identifier index, which [[-x]] does not build. +Projects in other languages override the autodefs choice, \eg +[[NOWEAVEFLAGS.tex]] with [[-autodefs c]]; the [[?=]] operator makes that a +one-line change in the project Makefile. + +The [[tominted]] filter typesets each chunk with the minted package, +choosing the lexer per chunk from the same language annotations. +This places two demands on the including document: its preamble must load +minted and LaTeX must run with [[-shell-escape]] (minted runs Pygments). +The [[-lexer]] argument loads noweb's bundled Pygments lexer, which keeps +chunk references hyperlinked even inside Python string literals, where +stock Pygments refuses to escape to LaTeX; minted resolves the path +relative to where LaTeX runs, which is why we provide a rule for copying +the lexer file below. + +Now we need to specify all the suffixes to use and then construct suffix rules for all of them. Fortunately we can use the same recipe for all, so we only need to write one recipe for multiple targets. @@ -77,7 +126,29 @@ endef $(foreach suf,${NOWEB_SUFFIXES},$(eval $(call def_weave_to_tex,${suf}))) @ -To differentiate the second case from the first (in terms of suffix rules), we +\paragraph{Providing the minted lexer} + +The woven TeX asks minted to load [[noweb_lexer.py]] from the directory +where LaTeX runs, but the file ships with noweb, in its library directory. +That directory is not on any standard search path; however, the +[[noweave]] script carries it in its [[LIB=]] line, so we read it from +there instead of hard-coding an installation prefix. +<>= +${NOWEB_LEXER}: + cp ${NOWEB_LIB}/noweb_lexer.py $@ +<>= +NOWEB_LEXER?= noweb_lexer.py +NOWEB_LIB?= $(shell sed -n 's/^LIB=//p' \ + $(shell command -v noweave) | head -1) +@ This include cannot know the name of the document that needs the lexer, +so it only provides the rule; the including Makefile declares the +dependency, as in [[doc.pdf: noweb_lexer.py]]. +Remember to add [[noweb_lexer.py]] to the project's [[.gitignore]] and +clean recipe, and that minted refuses custom lexers that are not +whitelisted by SHA-256 hash in [[~/.config/latexminted/.latexminted_config]] +(a one-time setup per machine). + +To differentiate the second case from the first (in terms of suffix rules), we go from [[.nw]] directly to [[.pdf]]\footnote{% Note, however, that these pattern rules will never be used by make. The make algorithm performs a depth-first search, thus make will take the @@ -95,8 +166,14 @@ define def_weave_to_pdf endef $(foreach suf,${NOWEB_SUFFIXES},$(eval $(call def_weave_to_pdf,${suf}))) -@ What differs [[NOWEAVE.pdf]] from [[NOWEAVE.tex]] is the options to +@ What differs [[NOWEAVE.pdf]] from [[NOWEAVE.tex]] is the options to [[noweave]] and the compilation step (instead of having that separately). +Here the preamble is generated by [[noweave]] and loads only the noweb +package, never minted, so the [[tominted]] output would not compile; +this weave keeps noweb's classic rendering. +That is not only a restriction: the classic rendering hyperlinks +identifier \emph{uses} inside code, which the minted rendering gives up +(minted owns the code body), so a standalone weave still has its place. <>= NOWEAVE.pdf?= \ noweave ${NOWEAVEFLAGS.pdf} $< > ${@:.pdf=.tex} && \ From 17ed4d31e8dc29e87473d9e92828acd2b515a849 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Fri, 12 Jun 2026 09:30:26 +0200 Subject: [PATCH 03/11] Keep identifier links out of section headings The autodefs filters now index make targets such as upload and autocommit, which turns every [[...]] quote of those names into a hyperlinked identifier use --- including the ones in section headings, where hyperref cannot survive a link inside the moving argument (the PDF bookmark breaks with "Undefined control sequence \hyper@@link"). Headings now write \texttt where body prose keeps the quoted, linked form. Co-Authored-By: Claude Fable 5 --- Makefile.nw | 7 ++++++- pub.mk.nw | 8 ++++---- tex.mk.nw | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/Makefile.nw b/Makefile.nw index 980725b..5e9c2e7 100644 --- a/Makefile.nw +++ b/Makefile.nw @@ -62,9 +62,14 @@ MAKEFILES_INCLUDE=${INCLUDE_MAKEFILES} <> @ -\subsection{This [[Makefile]]'s target} +\subsection{This \texttt{Makefile}'s target} We also must add a target for this [[Makefile]] itself. +(The heading says \verb|\texttt{Makefile}| where the prose says +\texttt{[{}[Makefile]{}]}: the autodefs filters index the +[[Makefile]] target below, which turns every quoted [[Makefile]] +into a hyperlinked identifier use, and hyperref cannot survive a +link inside a sectioning command's moving argument.) <>= Makefile: Makefile.nw ${NOTANGLE.mk} diff --git a/pub.mk.nw b/pub.mk.nw index 973309c..bc773b2 100644 --- a/pub.mk.nw +++ b/pub.mk.nw @@ -69,7 +69,7 @@ PUB_COMMIT_OPTS?= -av @ -\section{Configuration for publishing files on a server, [[upload]]} +\section{Configuration for publishing files on a server, \texttt{upload}} Publication means that we upload the files somewhere. This is controlled by the following variable. @@ -218,7 +218,7 @@ We will now cover the different parts below. The [[<>]] block has been covered in the usage section, but the remaining are discussed below. -\subsection{The upload publication mechanism, [[upload]]} +\subsection{The upload publication mechanism, \texttt{upload}} The upload target consists of two parts. <>= @@ -426,7 +426,7 @@ endef @ -\subsection{GitHub releases, [[gh-release]]} +\subsection{GitHub releases, \texttt{gh-release}} Now let's turn our attention to the [[gh-release]] target. It's important that we push any changes, since the tag and release are created @@ -441,7 +441,7 @@ gh-release: ${PUB_FILES} \subsection{Automatically committing and tagging, - [[autotag]] and [[autocommit]]} + \texttt{autotag} and \texttt{autocommit}} The last feature allows us to automatically commit and make a tag when we publish. diff --git a/tex.mk.nw b/tex.mk.nw index 78e9ecb..4449a49 100644 --- a/tex.mk.nw +++ b/tex.mk.nw @@ -608,7 +608,7 @@ of just [[lncs]]. llncs: lncs @ -\subsection{LNCS style for [[biblatex]]} +\subsection{LNCS style for \texttt{biblatex}} There is also \iac{LNCS} style for the [[biblatex]] package available on GitHub. From b402a4344db2ad3f05e8a1e9feaea09559f87845 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Fri, 12 Jun 2026 09:30:41 +0200 Subject: [PATCH 04/11] Stack sh and make autodefs in the default weave noweb now ships autodefs filters for shell and make (and Haskell, Rust and Java), gated by the same @language annotations -autolang provides, so stacking them indexes every chunk by exactly the filter that understands it. The default stack --- Python, shell, make --- covers what our projects mix into one document: program modules, helper scripts and their build files; our own woven documentation now indexes its make variables and targets. The stack stays at three because noweave has seven filter slots and -autolang, each -autodefs and tominted occupy one each; the default leaves two slots free for project filters, and other languages remain a one-line NOWEAVEFLAGS.tex override. Co-Authored-By: Claude Fable 5 --- noweb.mk | 4 ++-- noweb.mk.nw | 25 ++++++++++++++++--------- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/noweb.mk b/noweb.mk index 7e1c03d..be10998 100644 --- a/noweb.mk +++ b/noweb.mk @@ -3,8 +3,8 @@ NOWEB_MK = true NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ - -autodefs python3 -index \ - -filter 'tominted -lexer ${NOWEB_LEXER}' + -autodefs python3 -autodefs sh -autodefs make \ + -index -filter 'tominted -lexer ${NOWEB_LEXER}' NOWEB_LEXER?= noweb_lexer.py NOWEB_LIB?= $(shell sed -n 's/^LIB=//p' \ $(shell command -v noweave) | head -1) diff --git a/noweb.mk.nw b/noweb.mk.nw index 47021b8..4f526ac 100644 --- a/noweb.mk.nw +++ b/noweb.mk.nw @@ -75,8 +75,8 @@ Now, for the first case, we let <>= NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ - -autodefs python3 -index \ - -filter 'tominted -lexer ${NOWEB_LEXER}' + -autodefs python3 -autodefs sh -autodefs make \ + -index -filter 'tominted -lexer ${NOWEB_LEXER}' @ These flags weave syntax-highlighted documentation with a language-aware identifier index; they require noweb with the [[autolang]] and [[tominted]] filters (the dbosk fork) and the Icon autodefs filters. @@ -86,15 +86,22 @@ from filename-like chunk names. Our tangling rules below already force that naming---[[NOTANGLE]] extracts the chunk named exactly after the target file---so the convention pays twice: it lets make find the chunk and it lets noweb identify the language. -The annotations make the autodefs filters skip chunks in languages they do -not understand, so [[-autodefs python3]] fills the identifier index without -collecting make or shell assignments from a project's embedded [[Makefile]] -chunks (their assignments match the Python assignment pattern). +The annotations make each autodefs filter skip chunks in languages it does +not understand, which is what allows \emph{stacking} the filters: a make +variable, a shell variable and a Python constant all match each other's +assignment patterns, but with the annotations each chunk is indexed by +exactly the filter that understands it. +The default stack---Python, shell, make---covers what our projects mix +into one document: program modules, helper scripts and their build files. We use [[-index]] rather than the previous [[-x]] because the autodefs output feeds the identifier index, which [[-x]] does not build. -Projects in other languages override the autodefs choice, \eg -[[NOWEAVEFLAGS.tex]] with [[-autodefs c]]; the [[?=]] operator makes that a -one-line change in the project Makefile. +Projects in other languages override the choice, \eg +[[NOWEAVEFLAGS.tex]] with [[-autodefs c]] or [[-autodefs rust]]; the +[[?=]] operator makes that a one-line change in the project Makefile. +One caution when extending the stack instead: [[noweave]] has seven +filter slots, and [[-autolang]], every [[-autodefs]] and the +[[tominted]] filter each occupy one, so the default leaves two slots +free for the project's own filters. The [[tominted]] filter typesets each chunk with the minted package, choosing the lexer per chunk from the same language annotations. From 65edd028bfc01d26cf12db7a915f8f9a85f1eb90 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Sun, 14 Jun 2026 14:06:42 +0200 Subject: [PATCH 05/11] Correct the filter-slot budget note in noweb.mk The caution claimed the default stack leaves two of noweave's seven filter slots free. It missed that -index inserts two filters (finduses + noidx); with -autolang and tominted also taking one each, the three-autodefs default fills all seven exactly. Adding a fourth -autodefs or a project -filter overflows unless something is dropped. Prose-only change in a documentation chunk; the tangled noweb.mk is unchanged. Co-Authored-By: Claude Fable 5 --- noweb.mk.nw | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/noweb.mk.nw b/noweb.mk.nw index 4f526ac..22da067 100644 --- a/noweb.mk.nw +++ b/noweb.mk.nw @@ -99,9 +99,12 @@ Projects in other languages override the choice, \eg [[NOWEAVEFLAGS.tex]] with [[-autodefs c]] or [[-autodefs rust]]; the [[?=]] operator makes that a one-line change in the project Makefile. One caution when extending the stack instead: [[noweave]] has seven -filter slots, and [[-autolang]], every [[-autodefs]] and the -[[tominted]] filter each occupy one, so the default leaves two slots -free for the project's own filters. +filter slots, and they are scarcer than they look. [[-index]] alone +inserts two filters ([[finduses]] and [[noidx]]); [[-autolang]] adds +one and [[tominted]] one. The default---three [[-autodefs]] on top of +those---therefore fills all seven, so adding a fourth [[-autodefs]] or +a project [[-filter]] overflows the pipeline unless something else is +dropped. The [[tominted]] filter typesets each chunk with the minted package, choosing the lexer per chunk from the same language annotations. From 5c9ce498c99a83089f7dc0212d5245843187c0a8 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Mon, 15 Jun 2026 12:06:08 +0200 Subject: [PATCH 06/11] Sync noweb.mk docs with the fork's minted loading and unbounded pipeline - Replace the seven-filter-slot caution with the dbosk fork's unbounded pipeline; keep the old ceiling only as a note for pre-fork noweave. - Note that the preamble loads minted through the noweb package (\usepackage[minted]{noweb}) rather than a hand-added \usepackage{minted}. - Record that noweave -minted can load minted into the generated preamble, so the standalone .pdf weave stays classic by choice, not necessity. Documentation only; the tangled noweb.mk is unchanged. Co-Authored-By: Claude Opus 4.8 --- noweb.mk.nw | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/noweb.mk.nw b/noweb.mk.nw index 22da067..52ec909 100644 --- a/noweb.mk.nw +++ b/noweb.mk.nw @@ -98,18 +98,24 @@ output feeds the identifier index, which [[-x]] does not build. Projects in other languages override the choice, \eg [[NOWEAVEFLAGS.tex]] with [[-autodefs c]] or [[-autodefs rust]]; the [[?=]] operator makes that a one-line change in the project Makefile. -One caution when extending the stack instead: [[noweave]] has seven -filter slots, and they are scarcer than they look. [[-index]] alone -inserts two filters ([[finduses]] and [[noidx]]); [[-autolang]] adds -one and [[tominted]] one. The default---three [[-autodefs]] on top of -those---therefore fills all seven, so adding a fourth [[-autodefs]] or -a project [[-filter]] overflows the pipeline unless something else is -dropped. +Extending the stack is unconstrained: the dbosk fork's [[noweave]] +accumulates filters in a single variable and joins them with [[|]], so +any number of [[-autodefs]] filters stack alongside [[-autolang]], +[[-index]] (which inserts two filters, [[finduses]] and [[noidx]]) and +[[tominted]] in one weave. Older, pre-fork [[noweave]] enumerated only +seven filter slots---a portability artifact of the array-less +[[/bin/sh]] pipeline builder---so against such a [[noweave]] the +default three [[-autodefs]] plus [[-autolang]] and [[tominted]] fill all +seven, and a fourth [[-autodefs]] or a project [[-filter]] would +overflow; the fork removes that ceiling. The [[tominted]] filter typesets each chunk with the minted package, choosing the lexer per chunk from the same language annotations. This places two demands on the including document: its preamble must load minted and LaTeX must run with [[-shell-escape]] (minted runs Pygments). +The preamble loads minted through the noweb package, with +[[\usepackage[minted]{noweb}]], rather than a separate +[[\usepackage{minted}]]; see [[preamble.tex]]. The [[-lexer]] argument loads noweb's bundled Pygments lexer, which keeps chunk references hyperlinked even inside Python string literals, where stock Pygments refuses to escape to LaTeX; minted resolves the path @@ -178,12 +184,15 @@ endef $(foreach suf,${NOWEB_SUFFIXES},$(eval $(call def_weave_to_pdf,${suf}))) @ What differs [[NOWEAVE.pdf]] from [[NOWEAVE.tex]] is the options to [[noweave]] and the compilation step (instead of having that separately). -Here the preamble is generated by [[noweave]] and loads only the noweb -package, never minted, so the [[tominted]] output would not compile; -this weave keeps noweb's classic rendering. -That is not only a restriction: the classic rendering hyperlinks -identifier \emph{uses} inside code, which the minted rendering gives up -(minted owns the code body), so a standalone weave still has its place. +Here the preamble is generated by [[noweave]], and this weave keeps +noweb's classic rendering rather than the [[tominted]] highlighting. +That is a deliberate choice, not a limitation of the generated preamble: +[[noweave -minted]] would make the generated preamble load minted (it +adds [[\noweboptions{minted}]] before [[\begin{document}]]) and switch +on [[tominted]], so a highlighted standalone weave is available if +wanted. We keep the classic rendering because it hyperlinks identifier +\emph{uses} inside code, which the minted rendering gives up (minted +owns the code body), so a standalone weave still has its place. <>= NOWEAVE.pdf?= \ noweave ${NOWEAVEFLAGS.pdf} $< > ${@:.pdf=.tex} && \ From 94fdc15d1d2c725b0cb10df178d4cf02c3576496 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Mon, 15 Jun 2026 12:08:12 +0200 Subject: [PATCH 07/11] Load minted through the noweb package option in preamble.tex Replace the hand-added \usepackage{minted} (plus a separate \usepackage{noweb}) with \usepackage[minted]{noweb}, so the minted dependency rides the noweb package the way tominted weaves expect. \setminted{autogobble} still applies, as minted is loaded (via \AtEndOfPackage) before it runs. Requires the noweb fork whose noweb.sty declares the minted option; to pass minted options such as outputdir, load minted directly before \usepackage{noweb} instead. Co-Authored-By: Claude Opus 4.8 --- preamble.tex | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/preamble.tex b/preamble.tex index 009918a..5bbb1c8 100644 --- a/preamble.tex +++ b/preamble.tex @@ -23,11 +23,11 @@ basicstyle=\small } -%\usepackage[outputdir=ltxobj]{minted} -\usepackage{minted} +% noweb loads minted for us via its package option; to pass minted +% options (e.g. outputdir) load it directly with \usepackage[...]{minted} +% before \usepackage{noweb} instead. +\usepackage[minted]{noweb} \setminted{autogobble} - -\usepackage{noweb} % Needed to relax penalty for breaking code chunks across pages, otherwise % there might be a lot of space following a code chunk. \def\nwendcode{\endtrivlist \endgroup} From e870778cb3de923ce65488ebb2b2cee0d8dcc524 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Tue, 23 Jun 2026 07:41:37 +0200 Subject: [PATCH 08/11] noweb.mk: drop -autodefs flags; -autolang now discovers them The dbosk fork's noweave, with -index, runs -autolang's inference as a pre-pass to discover which languages a document contains and stacks the matching autodefs filter for each. The default weave therefore no longer lists -autodefs python3/sh/make: discovery covers them, and an explicit -autodefs is still merged without duplication for chunks whose language the name does not reveal. Co-Authored-By: Claude Opus 4.8 --- noweb.mk | 1 - noweb.mk.nw | 39 ++++++++++++++++++++++----------------- 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/noweb.mk b/noweb.mk index be10998..7d3f398 100644 --- a/noweb.mk +++ b/noweb.mk @@ -3,7 +3,6 @@ NOWEB_MK = true NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ - -autodefs python3 -autodefs sh -autodefs make \ -index -filter 'tominted -lexer ${NOWEB_LEXER}' NOWEB_LEXER?= noweb_lexer.py NOWEB_LIB?= $(shell sed -n 's/^LIB=//p' \ diff --git a/noweb.mk.nw b/noweb.mk.nw index 52ec909..c8c6057 100644 --- a/noweb.mk.nw +++ b/noweb.mk.nw @@ -75,7 +75,6 @@ Now, for the first case, we let <>= NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ - -autodefs python3 -autodefs sh -autodefs make \ -index -filter 'tominted -lexer ${NOWEB_LEXER}' @ These flags weave syntax-highlighted documentation with a language-aware identifier index; they require noweb with the [[autolang]] and [[tominted]] @@ -91,23 +90,29 @@ not understand, which is what allows \emph{stacking} the filters: a make variable, a shell variable and a Python constant all match each other's assignment patterns, but with the annotations each chunk is indexed by exactly the filter that understands it. -The default stack---Python, shell, make---covers what our projects mix -into one document: program modules, helper scripts and their build files. + +We no longer list the autodefs filters at all. With [[-index]], the dbosk +fork's [[-autolang]] discovers which languages a document contains and +stacks the matching autodefs filter for each, so the defaults index +whatever a project mixes into its documents---program modules, helper +scripts and their build files---without naming Python, shell or make +explicitly. We use [[-index]] rather than the previous [[-x]] because the autodefs -output feeds the identifier index, which [[-x]] does not build. -Projects in other languages override the choice, \eg -[[NOWEAVEFLAGS.tex]] with [[-autodefs c]] or [[-autodefs rust]]; the -[[?=]] operator makes that a one-line change in the project Makefile. -Extending the stack is unconstrained: the dbosk fork's [[noweave]] -accumulates filters in a single variable and joins them with [[|]], so -any number of [[-autodefs]] filters stack alongside [[-autolang]], -[[-index]] (which inserts two filters, [[finduses]] and [[noidx]]) and -[[tominted]] in one weave. Older, pre-fork [[noweave]] enumerated only -seven filter slots---a portability artifact of the array-less -[[/bin/sh]] pipeline builder---so against such a [[noweave]] the -default three [[-autodefs]] plus [[-autolang]] and [[tominted]] fill all -seven, and a fourth [[-autodefs]] or a project [[-filter]] would -overflow; the fork removes that ceiling. +output feeds the identifier index, which [[-x]] does not build (and it is +also what switches discovery on). +A project whose chunk names do not reveal their language can still name it: +adding [[-autodefs c]] or [[-langrule]] to [[NOWEAVEFLAGS.tex]] is a +one-line change through the [[?=]] operator, and an explicit [[-autodefs]] +is merged with the discovered set without duplication. +The pipeline has no fixed size: the dbosk fork's [[noweave]] accumulates +filters in a single variable and joins them with [[|]], so any number of +autodefs filters stack alongside [[-autolang]], [[-index]] (which inserts +two filters, [[finduses]] and [[noidx]]) and [[tominted]] in one weave. +Older, pre-fork [[noweave]] enumerated only seven filter slots---a +portability artifact of the array-less [[/bin/sh]] pipeline builder---and +also lacked discovery, so against such a [[noweave]] one would both list +the [[-autodefs]] filters by hand and keep their number within the slot +budget; the fork removes both constraints. The [[tominted]] filter typesets each chunk with the minted package, choosing the lexer per chunk from the same language annotations. From 473484ca2a016b16c5d893095f9f7ec3e6e2dd03 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Tue, 23 Jun 2026 07:56:22 +0200 Subject: [PATCH 09/11] noweb.mk: add default -langrule rules for test chunks Test chunks follow the convention <>, labelling the file rather than being it, and autolang deliberately does not infer a language from such a labelled name. Left unclassified, a test chunk is scanned by every autodefs filter and salts the index with cross-language false matches. Add one -langrule per supported language (python, shell, make) so each test chunk is classified, and hence indexed, by the same filter as the module it exercises. Co-Authored-By: Claude Opus 4.8 --- noweb.mk | 3 +++ noweb.mk.nw | 26 +++++++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-) diff --git a/noweb.mk b/noweb.mk index 7d3f398..9266a5e 100644 --- a/noweb.mk +++ b/noweb.mk @@ -3,6 +3,9 @@ NOWEB_MK = true NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ + -langrule '^test \[\[.*\.py\]\]=python' \ + -langrule '^test \[\[.*\.sh\]\]=bash' \ + -langrule '^test \[\[Makefile\]\]=make' \ -index -filter 'tominted -lexer ${NOWEB_LEXER}' NOWEB_LEXER?= noweb_lexer.py NOWEB_LIB?= $(shell sed -n 's/^LIB=//p' \ diff --git a/noweb.mk.nw b/noweb.mk.nw index c8c6057..1ef6ad8 100644 --- a/noweb.mk.nw +++ b/noweb.mk.nw @@ -75,6 +75,9 @@ Now, for the first case, we let <>= NOWEAVE.tex?= noweave ${NOWEAVEFLAGS.tex} $< > $@ NOWEAVEFLAGS.tex?= ${NOWEAVEFLAGS} -n -delay -t2 -autolang \ + -langrule '^test \[\[.*\.py\]\]=python' \ + -langrule '^test \[\[.*\.sh\]\]=bash' \ + -langrule '^test \[\[Makefile\]\]=make' \ -index -filter 'tominted -lexer ${NOWEB_LEXER}' @ These flags weave syntax-highlighted documentation with a language-aware identifier index; they require noweb with the [[autolang]] and [[tominted]] @@ -100,9 +103,26 @@ explicitly. We use [[-index]] rather than the previous [[-x]] because the autodefs output feeds the identifier index, which [[-x]] does not build (and it is also what switches discovery on). -A project whose chunk names do not reveal their language can still name it: -adding [[-autodefs c]] or [[-langrule]] to [[NOWEAVEFLAGS.tex]] is a -one-line change through the [[?=]] operator, and an explicit [[-autodefs]] + +Discovery rests on a chunk's name revealing its language, which the +tangling convention guarantees for modules but not for their tests: a +module's tests live in a sibling chunk that \emph{labels} the file rather +than being it, such as @<> for the module [[greet.py]]. +[[autolang]] leaves such a labeled name unclassified by design---a test +chunk could be a shell script that drives the Python module rather than +Python itself---so without help it would carry no language, and an +unclassified chunk is scanned by \emph{every} autodefs filter, salting the +index with cross-language false matches (Python keywords read as make +rules, say). +The three [[-langrule]] options supply that help for the languages we use: +each matches a name that \emph{opens} with the [[test]] label---the [[^]] +anchor keeps a word that merely ends in \emph{test}, or a [[test]] in +mid-name, from matching---and ends in one file extension, mapping it to +that extension's language, so a test chunk is indexed by the same filter as +the module it exercises. +A project that labels its chunks differently---a [[check]] prefix, another +extension---overrides [[NOWEAVEFLAGS.tex]] through the [[?=]] operator; +likewise a project in another language adds its own [[-autodefs c]], which is merged with the discovered set without duplication. The pipeline has no fixed size: the dbosk fork's [[noweave]] accumulates filters in a single variable and joins them with [[|]], so any number of From 6a0555b82c68fa710be95ce1a25b7510d22bc6f2 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Tue, 23 Jun 2026 10:03:17 +0200 Subject: [PATCH 10/11] Commits compiled tex.mk --- tex.mk | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/tex.mk b/tex.mk index fad6198..047c49c 100644 --- a/tex.mk +++ b/tex.mk @@ -12,14 +12,10 @@ LATEXFLAGS?= PREPROCESS.tex?= ${PDFLATEX} ${LATEXFLAGS} $< PREPROCESS.dtx?= ${PREPROCESS.tex} TEX_OUTDIR?= ltxobj -# The rerun loop is bounded: latexmk already reruns internally and gives -# up when the document does not converge, leaving "Rerun to get cross" -# in the log. An unbounded loop would then relaunch latexmk forever. COMPILE.tex?= \ ${PDFLATEX} ${LATEXFLAGS} -output-directory=${TEX_OUTDIR} $<; \ - for i in 1 2 3 4 5; do \ - grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} || break; \ - ${PDFLATEX} ${LATEXFLAGS} -output-directory=${TEX_OUTDIR} $<; \ + while (grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log}); \ + do ${PDFLATEX} ${LATEXFLAGS} -output-directory=${TEX_OUTDIR} $<; \ done COMPILE.dtx?= ${COMPILE.tex} TEX_BBL?= @@ -85,8 +81,7 @@ ${TEX_OUTDIR}/%.pytxcode: ${TEX_OUTDIR}/%.aux %.dvi ${TEX_OUTDIR}/%.dvi: %.tex ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $< - for i in 1 2 3 4 5; do \ - grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} || break; \ + while ( grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} ); do \ ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $<; \ done -${LN} ${TEX_OUTDIR}/$@ $@ @@ -101,8 +96,7 @@ latexmkrc: %.dvi ${TEX_OUTDIR}/%.dvi: %.dtx ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $< - for i in 1 2 3 4 5; do \ - grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} || break; \ + while ( grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} ); do \ ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $<; \ done -${LN} ${TEX_OUTDIR}/$@ $@ From 8075f8cfc35cd30b87e0210f413e92db9e720f27 Mon Sep 17 00:00:00 2001 From: Daniel Bosk Date: Tue, 23 Jun 2026 10:06:35 +0200 Subject: [PATCH 11/11] Updates compiled version of tex.mk --- tex.mk | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tex.mk b/tex.mk index 047c49c..5ae50da 100644 --- a/tex.mk +++ b/tex.mk @@ -14,8 +14,9 @@ PREPROCESS.dtx?= ${PREPROCESS.tex} TEX_OUTDIR?= ltxobj COMPILE.tex?= \ ${PDFLATEX} ${LATEXFLAGS} -output-directory=${TEX_OUTDIR} $<; \ - while (grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log}); \ - do ${PDFLATEX} ${LATEXFLAGS} -output-directory=${TEX_OUTDIR} $<; \ + for i in 1 2 3 4 5; do \ + grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} || break; \ + ${PDFLATEX} ${LATEXFLAGS} -output-directory=${TEX_OUTDIR} $<; \ done COMPILE.dtx?= ${COMPILE.tex} TEX_BBL?= @@ -81,7 +82,8 @@ ${TEX_OUTDIR}/%.pytxcode: ${TEX_OUTDIR}/%.aux %.dvi ${TEX_OUTDIR}/%.dvi: %.tex ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $< - while ( grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} ); do \ + for i in 1 2 3 4 5; do \ + grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} || break; \ ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $<; \ done -${LN} ${TEX_OUTDIR}/$@ $@ @@ -96,7 +98,8 @@ latexmkrc: %.dvi ${TEX_OUTDIR}/%.dvi: %.dtx ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $< - while ( grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} ); do \ + for i in 1 2 3 4 5; do \ + grep "Rerun to get cross" ${TEX_OUTDIR}/${<:.tex=.log} || break; \ ${LATEX} -output-directory=${TEX_OUTDIR} ${LATEXFLAGS} $<; \ done -${LN} ${TEX_OUTDIR}/$@ $@