spec conversion by aaronmbauman · Pull Request #81 · OpenGridForum/DFDL

aaronmbauman · 2026-09-01T01:50:52Z

Convert spec from MS-Word to AsciiDoc

The specification builds to an ISO-formatted PDF, HTML and semantic XML for both
the OGF and ISO editions, from one body, on one command, in a pinned container.

Why AsciiDoc when the issue asks for XML

The XML is here. It is produced and schema-validated on every build, and it is
what the property index and the cross-reference checker read: both were built by
querying it, not by parsing prose. What differs from a literal reading of #68 is
that editors write AsciiDoc rather than angle brackets.

A 12,000-line hand-edited XML document raises the barrier to contributing,
makes merge conflicts worse, and adds a failure mode AsciiDoc does not have: a
document that is no longer well-formed. Machines still get the XML.

DFDL's own concepts are expressible too. A property written as a typed
requirement block emits name, value type, default, valid values, owning
annotation and normative constraints as separate elements. That is demonstrated
but deliberately not adopted here; see what is deferred, below.

Why the conversion reproduces the source's mistakes

The conversion is faithful, including defects. pattern.at stays misspelled, an
em dash Word autocorrected into an XML comment stays an em dash, a missing full
stop stays missing.

A conversion that improves its source cannot be verified against it. Every
difference then has to be argued about individually, and nobody can say which
were intended. Holding the line makes the whole document checkable by machine,
which is what lets the claim "nothing was lost or altered" mean anything.

Seven classes of defect the conversion surfaced are recorded rather than fixed,
including a Property Index listing 7 of the specification's 106 properties, and
six dfdl: names in prose that match no property. Those are the working group's
to decide on.

Why there are three validators

Text can be provably intact while the document is wrong, so make check asks
three different questions.

A comparison that normalises typography away on both sides will report a
conversion clean while every 0x55 in an encoding table has become 0×55, a
normative regex has lost a parenthesis, and <= has turned into an arrow in
normative grammar. Those substitutions live in exactly the characters such a
check is taught to ignore. So the fidelity check is strict by default and counts
punctuation, not only words.

Resolving is a separate question from surviving, so the link checker proves every
cross-reference and citation lands on exactly one target.

Neither of those can see a document that is internally incoherent. The structural
checker reads the rendered page and asserts it says what it means: no sentence
naming the wrong figure, no caption stranded as a body paragraph, no markup
leaking into text, nothing published that Word hides.

The link and structure checks stay useful for as long as the document is edited,
so they are what make check runs. spec/EDITING.md covers the conventions an
editor needs: how clauses are named and added, how cross-references are
anchored, why an untitled listing has to be marked unnumbered.

Coverage is 81,709 of 81,884 source words; the shortfall is the cover page, which
becomes document metadata. A clause that pairs but is not compared is an error
rather than a note, because a gate that silently compares nothing still reports
PASS.

Why the two editions share one body

The source states the requirement in its own preamble: section and page numbers
must not change between the OGF and ISO editions. JTC 1 SD-9 §7.3.1 says the same
from the other side, that ISO intends no divergence between a transposed PAS and
the originator's version 1.

One body with two front matters makes that mechanical rather than a discipline
somebody has to maintain. spec/frontmatter/ is the only thing that differs.

What is deferred, and why

The DFDL property model, expressing property tables as typed requirement
blocks, is a second pull request. It re-shapes 15 tables, which is a different
review question from "is this the same specification", and mixing the two makes
both unanswerable.

Publication is untouched. Nothing here changes GitHub Pages, the README's
htmlpreview link, or which document is authoritative. The series is purely
additive: it adds spec/, tools/ and a Makefile, and changes nothing that
exists, so merging it cannot break anything in use today.

Errata and ISO Directives edits change the specification's text. Neither is
conversion work.

Notes

Comparing the build against the Word source is not part of make check. It
proves the conversion, and fails on any deliberate change to the specification,
so leaving it in the default checks would make the first erratum look like a
defect. It runs on its own as make check-conversion.

Six plain-text references were left unlinked because linking them would change
what the sentence says: Metanorma labels a top-level clause "Clause 11" where the
specification writes "Section 11".

The workflow has no push or pull request trigger, so opening this will show no
check. Whether this repository runs CI automatically, and on what, is a working
group decision rather than something a conversion should assume; the triggers to
add are commented at the top of .github/workflows/spec.yml. Until then it runs
from the Actions tab, and make check runs the same validators locally.

Closes: #68
Closes: #78
Closes: #79

The specification is an ISO Publicly Available Specification, and the rules for transposing one decide several things that would otherwise be guessed at: whether Directives conformance is mandatory, whether change bars are needed, and when the next review falls. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Establishes the master document and the ISO metadata the build needs. Front matter is kept in its own file from the start. The OGF edition differs from the ISO one only there, so nothing added to the body later can become specific to one edition. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

The specification ships in two editions. Both use ISO formatting, and the OGF one carries OGF front matter and copyright, so section and page numbers stay identical between them (OpenGridForum#76). JTC 1 SD-9 (6th ed. 2021) §7.3.1 states the same intention from the other side, that ISO avoids divergence between a transposed PAS and the version published by the originator [1]. [1]: https://jtc1info.org/wp-content/uploads/2022/01/SD-9-Guide-to-the-Transposition-of-PAS-2021.pdf Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Pin metanorma-cli so local and CI builds render identically. Output is keyed by edition. The OGF and ISO editions render identically-named artifacts, so they need separate directories from the start rather than a rename once the second edition exists. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

ruff and black run on the host rather than in the Metanorma container, which carries neither. check runs the validators in tools/ over the build. Both targets tolerate an empty tools/. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Rendering happens inside the pinned Metanorma image, so the runner installs nothing. The uploaded artifacts are how a reviewer reads the rendered spec without a local toolchain. Both editions are built, because a change can break one and not the other. There is no push or pull_request trigger. This repository has never run CI, and whether it should, and on what, is a working group decision; the workflow is here to be run from the Actions tab until that is settled. Read-only permissions and no secrets, so it behaves identically for pull requests from a fork. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Separate from the build job so a lint failure and a rendering failure are distinguishable at a glance. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Converting the whole document in one pass produces a diff nobody can review, so the tool addresses one top-level clause at a time. Word owns the heading numbering and switches it off for the front matter, so the specification's own clause keys are reconstructed rather than read out of the text. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Word splits a logical listing across consecutive paragraphs, so a run of them collapses into one source block. Schema fragments left in body text are listings too: read as prose their quotes curl and their tags are taken for markup. Prose passes through AsciiDoc's inline markup and then Metanorma's smart formatting, and both claim sequences that are normative here. 0x55, #,##,##0, <= and ' are wrapped per sequence, so ordinary emphasis still works around them. Editors' notes live in Word's hidden text and are kept as AsciiDoc comments, where the next editor has them and no reader does. Listings the source neither numbers nor names are marked unnumbered, so they do not take figure numbers from the diagrams that carry one. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Ids come from heading and caption text rather than Word's internal bookmark names, so a cross-reference reads as something in the specification and survives reconversion. Because the document is converted a clause at a time, a reference can point outside the text being emitted; those keep Word's cached result so the sentence still reads. Word writes a reference as a pair of fields, one resolving to the target's number and one to its title. Only the number is Metanorma's to regenerate, so the pair becomes one reference followed by the title. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Word tolerates rows whose cells do not add up to the table grid and AsciiDoc drops the surplus silently, so every row is fitted to the grid first. Word places a caption after the object it describes; AsciiDoc wants the title in front and Metanorma numbers tables itself. Some captions are laid out as a row of the table, where a title left in place is read as cell markup and takes the cell down with it. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Metanorma resolves image macros against the source tree, so the media parts of the package are written out beside the AsciiDoc. Where a drawing carries both an SVG and a raster fallback the SVG is taken, as it stays sharp in the rendered PDF and HTML. An image is rewritten only when its bytes differ, so re-running the conversion leaves the working tree alone. A footnote is a macro, so brackets in its text have to be escaped; those belonging to the passthroughs that carry technical sequences are left alone, since escaping them would undo the protection. A footnote on a heading is not carried into the rendered title, so it moves to the text the heading introduces. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Word and Metanorma disagree on escaping and whitespace even where the conversion is faithful, so the two can only be compared through a normalised form. Every character folded away is one no comparison can see differ, so the default folds only what the formats must disagree about and typography folding is opt-in. Clause numbers are reconstructed from structure because neither format stores them as text, and the unnumbered front-matter regions are numbered too, so a difference there can be located. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Counting occurrences of normative phrases passes while an entire property table is dropped, so the check is a differential over the whole of both documents. It compares strictly by default: fold quotes, dashes and symbols away and a target that turned every 0x55 in an encoding table into 0x55 with a multiplication sign compares identical to its source. Tokens rather than words carry the verdict, since punctuation is what means something in a regex or a number pattern. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

The conversion lands one clause at a time, so a run over the whole document calls lost what has simply not been written yet. The built output is the authority on what has been converted. A clause the build carries is compared whether or not it has a body, and a clause the build carries that answers to no source clause is content nobody can trace back. Both fail the run. Clauses pair by title because the ISO edition renumbers them, and repeated titles pair in document order: 25 to 28 are four siblings all titled "Removed". Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

The cover page is the publisher's imprint rather than specification text: the title, the document status, the copyright line and the abstract are metadata each edition states in its own form, and ISO generates all of it from the document attributes in the master file. The abstract is the one part both editions print as prose, so it is carried here, in the source's own words. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>

Verbatim output of tools/docx2adoc.py at ed1aa48. The preamble carries the source's own statement of the two-edition requirement — that section and page numbers stay identical between the OGF and ISO editions, and that sections before the Introduction are unnumbered. It is retained rather than treated as editorial scaffolding; whether the MS-Word cross-referencing instructions belong in the specification is the working group's decision, not the conversion's. The intellectual property statement, the disclaimer and the full copyright notice are the specification's own text, headed and numbered as it heads and numbers them, and both editions carry them as converted clauses rather than as wording chosen here. Everything before the Introduction is preface in both editions, so that the numbered clauses start at the same number in each. Signed-off-by: Aaron Bauman <bauman.aaron@gmail.com>