feat(reader): add an internal content-stream interpreter

[Tim81]

Part of #98 (v2.4 PR 3 of 16). #398 (page tree) is merged; nothing else outstanding.

What

An internal ISO 32000-2 §7.8.2 operand-stack interpreter over the page tree that PR 5 (text extraction) and PR 11 (image extraction) will share. Nothing public calls it in this PR; the public surface is one option and ten diagnostic codes.

• Content/ContentInterpreter.cs: Run(PdfReadPage, IContentVisitor). Owns the operand stack (cap 64; Annex C.2 Table C.1 recorded 32 as the DeviceN colourant count earlier versions recommended and §8.6.6.5 allows an arbitrary number, so scn with 32 components plus a pattern name is a legal 33-operand call), the TJ array cap (8192 elements), the q/Q state stack (cap 64; an unbalanced Q is ignored with a diagnostic), BX/EX nesting, marked-content nesting (cap 64, mirroring the q/Q treatment), gs with an /ExtGState /Font entry, and Form XObject recursion on Do (/Subtype /Form): depth from PdfReaderOptions.MaxFormXObjectDepth (default 32, floor 1, tighten-only), an on-stack cycle guard (FormXObjectCycle, reported once), 4096 invocations per page (FormXObjectBudgetExceeded; the page still completes), and the §8.10.1 a)/b)/e) treatment of the form's content: the graphics state is saved, the form's /Matrix is premultiplied into the CTM (§8.3.4) so OnFormBegin and every operator inside the form see the composed value, and the state is restored on return. Marked-content depth and BX/EX depth are saved the same way, implemented as stack floors so the implicit save costs nothing against the 64-deep cap and a form's own Q cannot pop below its entry; the invoker's text matrices and its inside-a-text-object flag are saved too, because the form's content starts outside any text object of its own (§9.4.1) and its BT/Td/ET must not leak back. An /XObject entry that resolves to something other than a Form or Image stream (a non-stream, a stream without /Subtype, an unknown /Subtype) reports ResourceMissing with the shape it found. A form without its own /Resources falls back to the invoking stream's per §8.10.2 Table 93 (the PDF 1.1 shape; "optional but strongly recommended" since 1.2). /Matrix, /BBox and the ExtGState /Font resolve through an indirect reference before the shape check (§7.3.10).
• Inline images per §8.9.7, Table 91 and Table 92 abbreviations expanded before the visitor sees the dictionary, inside a /CS array too ([/I /RGB 1 <…>] becomes [/Indexed /DeviceRGB 1 <…>], the one composite inline colour space §8.9.7 allows). Delimiting in three tiers: /L (PDF 2.0, Table 91) when present and the bytes at the computed end are EI; else, for unfiltered data, Height × rowBytes computed from /W, /H, /BPC and the component count of the colour space (named /ColorSpace resources resolved through the page's resources; anything this reader cannot count falls through to the scan); else a whitespace-EI-whitespace/delimiter scan. A scan candidate is accepted when a probe over at most 8 tokens inside a 128-byte window after it lexes in content mode with every keyword either a Table A.1 operator, true/false/null, a one-byte {/}/>, or an unknown name made only of printable ASCII; a token that runs off that window is re-probed once over a 4096-byte window, and a token that runs off that one too rejects the candidate; a keyword carrying any other byte is binary noise and rejects the candidate; BI ends the probe with acceptance because what follows its ID is image data. When a tier-a or tier-b length does not land on EI, the interpreter reports InlineImageMalformed and falls back to the scan rather than losing the rest of the stream. After ID, CR LF counts as one EOL marker (§7.2.3), and when ASCIIHexDecode or ASCII85Decode appears anywhere in the filter list every following whitespace byte is skipped before the data starts (§8.9.7 and its NOTE 2). JBIG2Decode, JPXDecode and Crypt inline are InlineImageMalformed and skipped; a missing ID likewise; so are a negative or non-integer /L, a /W or /H that is missing, non-integer or non-positive, and a /BPC outside 1, 2, 4, 8 and 16 (Table 87).
• PdfLexer gains an opt-in content-stream mode (new PdfLexer(data, contentStreamMode: true)): {, } and a lone > become one-byte Keyword tokens instead of throwing, so a stray brace inside BX/EX no longer aborts the page. The default constructor path is unchanged; the 11 IConformanceRule implementations that construct a PdfLexer (one of them over CMap bytes rather than a content stream) keep their behaviour (Conformance.Tests below).
• /Contents arrays are concatenated with a separator so a token split across two streams is not glued (§7.7.3.3); a non-stream element or a stream whose filter chain this reader cannot decode reports ContentStreamLexError and the page continues with what it has. Decoded content bytes are capped at 64 MiB per Run, one budget shared between the page's /Contents and every Form XObject invocation it makes (each invocation counts again, since the cost being bounded is interpretation work). The budget is charged as each /Contents element decodes, so an array that names one oversized stream many times stops decoding at the cap instead of holding every copy first, and a form invocation with no budget left skips its decode; ContentStreamTooLarge is reported once and the operators before the cap are kept.
• An unknown operator outside BX/EX reports UnknownOperator (Warning) and clears the operand stack, since §7.8.2 says operands shall not be left over once an operator has run and a kept stack made the next valid operator fail its arity check with a second, spurious diagnostic. Inside BX/EX it is silent and its operands are dropped the same way, per Table 33 ("along with their operands").
• Two codes split what a consumer can act on differently: OperandStackMalformed (302) for producer-side malformations (unparseable or double-signed number, misplaced dictionary operand, wrong arity, an operand of the wrong type at the right arity such as a TJ whose operand is not an array (§9.4.3), a Do inside a text object (§8.2 Figure 9 admits no operator of Table 50's XObjects category there), unbalanced Q or EMC) and ContentLimitExceeded (309) for this reader's own ceilings (operand count, TJ elements, q depth, marked-content depth).
• ContentStreamTooLarge and FormXObjectBudgetExceeded go through DiagnosticSink.ReportRetained (the feat(reader): walk the page tree and expose PdfReadPage #398 precedent for a condition that ends a walk): at most two retained entries per Run, one of each.
• Graphics and text state (GraphicsState, TextState, Matrix) live beside the interpreter; the text lane reads them, the image lane ignores them.
• PdfDocumentReader.CreateContentDiagnosticScope() is the first caller of DiagnosticSink.CreateScope (Reader: a diagnostics channel for notify-and-continue conditions #385): one scope per page interpreted.
• Run never lets InvalidDataException escape: the lexer and parser paths were already caught per stream, and an outermost catch now covers a malformed reference chain met while resolving /Contents, a resource or a form (reported as ContentStreamLexError).

Public API (PublicAPI.Unshipped.txt, +12 lines)

PdfReaderOptions.MaxFormXObjectDepth (get/init) and PdfReaderDiagnosticCode 300 to 309: ContentStreamLexError, UnknownOperator, OperandStackMalformed, FormXObjectDepthExceeded, FormXObjectCycle, FormXObjectBudgetExceeded, ResourceMissing, InlineImageMalformed, ContentStreamTooLarge, ContentLimitExceeded. All ten are Warning; UnknownOperator moved from Info to Warning in round 1 because PdfReaderDiagnosticSeverity.Info is defined as "produced correct output" and a skipped operator is not that. PdfReaderDiagnosticCodeTests pins the 3xx range, uniqueness and every severity. PublicAPI.Shipped.txt untouched; the file is UTF-8 without BOM, LF.

Departures from the brief, recorded for the reviewers

• The brief cited Table 93 for the inline /L key. ISO 32000-2 §8.9.7 puts L in Table 91 ("Entries in an inline image object", the PDF 2.0 addition); Table 93 is the Type 1 form dictionary. The code cites Table 91.
• Operand discipline is stricter than the brief's "operands stay": an operator with the wrong operand count reports OperandStackMalformed and is dropped, and the stack is cleared. Since round 2 an unknown operator clears the stack as well, so 1 0 0 1 10 20 R cm (the stray-R case) reports UnknownOperator for R and then OperandStackMalformed for a cm with no operands; the cm is dropped either way.
• UnknownOperator is reported once per page, not once per name: the sink's dedupe key is (code, object, page) and the interpreter reports with no object number, so a second distinct unrecognised name on the same page is deduped away. The XML doc says so.
• ScanForEi strips exactly one whitespace byte before the EI; a producer that pads with more leaves the extra bytes in the image data. Unfiltered and DCT decoders tolerate trailing bytes; this is disclosed rather than modelled.
• The marked-content cap (64) mirrors the q/Q cap rather than being separately justified; §14.6.1 sets no nesting limit.
• No Info diagnostic when tier c (the scan) delimits an image: tier c is the normal path for every filtered inline image in a PDF older than 2.0, so it would fire on ordinary content.
• MaxFormXObjectDepth is public while the sibling caps are private constants: the plan's minimal-surface decision.

Fix-ups on top of the first commit

99777c2: LooksLikeResyncPoint originally accepted any candidate EI whose following bytes lexed without throwing; binary noise lexes as one long Keyword, so the probe accepted almost everything. It then required every keyword in the window to be a Table A.1 operator; DctInlineImage_withAFalseEiFollowedByBinaryNoise_isSkippedByTheScan is the case the first version passed for the wrong reason. Also: ReaderLimits.Resolve doc counts four knobs, the CreateScope remark no longer says it is unused, the per-name set dropped, one comment reword.

Round 1 (three reviewers at 99777c2: inline delimiting; operands and forms with a corpus sweep; lexer regression, API freeze and prose)

Two HIGH, eleven MEDIUM, all confirmed and fixed in 3d98820 (+ prose tidy 298693e):

• HIGH: Concatenate used the three-argument Array.Copy, which copies to index 0, so a multi-stream /Contents over the cap had its truncated last chunk written over the first. The single-stream test passed because written was 0. Now Array.Copy(chunk, 0, capped, written, take); ContentExceeding64MiB_acrossTwoStreams_appendsTheSecondStreamAfterTheFirst_notOverIt.
• HIGH: the resync probe lexed to the end of the buffer from every candidate, so BI /F /DCT ID followed by N copies of " EI (" was quadratic (100 KB of content took 18 s, 400 KB 305 s, from a 1.2 KB Flate-compressed file). The probe is now bounded to 128 bytes and 8 tokens; a token that runs off a clipped window is inconclusive and accepts, so the " EI (" shape costs one string read by the outer lexer and ends in ContentStreamLexError. ManyFalseEiCandidates_doesNotThrow_andReportsADiagnostic (asserts the diagnostic, not the time, per Tests: wall-clock assertions flake on the shared CI runner #400).
• MEDIUM: Do did not save and restore the graphics state (§8.10.1 a) and e)), so a form's cm leaked, a form's stray Q popped the page's save, and 70 invocations of a q-only form exhausted the cap. Marked-content and BX/EX depth leaked the same way. Floors as described under What; four Do_on… tests.
• MEDIUM: inside BX/EX an unknown operator kept its operands (Table 33 says they are ignored with it). UnknownOperatorInsideBX_dropsItsOperands_perTable33.
• MEDIUM: the 64 MiB cap covered only the page's /Contents; a 46 KB file drawing one 8.4 MB form 40 times interpreted 336 MB in 108 s with no diagnostic. Per-Run budget shared with forms; FormDrawnRepeatedly_countsTowardTheSameContentBudgetAsThePage.
• MEDIUM: the probe's "every keyword must be a Table A.1 operator" rule rejected a valid EI followed by an unknown operator or by BX { } EX, losing the rest of the stream, and an 8-token window could swallow a second inline image with a short dictionary. Printable unknown keywords are neutral, BI stops the probe; four FalseEiCandidate_…/TwoConsecutiveInlineImages_… tests.
• MEDIUM: a tier-a/tier-b length that did not land on EI (/L one short, ID + CR LF, /W 0) abandoned the rest of the stream while /L past the end recovered. Symmetric fallback to the scan with a diagnostic; CR LF consumed as one separator per §7.2.3, with a one-byte-earlier retry for data that itself starts with LF; four tests.
• MEDIUM: /CS [/I /RGB 1 <…>] was not abbreviation-expanded and had no component count. Fixed; CsArray_tableNinetyTwoAbbreviationsExpand_andIndexedComponentCountDrivesTierB.
• MEDIUM: ContentStreamTooLarge and FormXObjectBudgetExceeded used Report, so they were lost once MaxDiagnostics was spent (the feat(reader): walk the page tree and expose PdfReadPage #398 ReportRetained precedent). Switched; ContentStreamTooLarge_isRetainedEvenOnceMaxDiagnosticsIsAlreadySpent.
• MEDIUM: UnknownOperator XML doc said "once per (page, operator name)"; it is once per page. Rewritten with the dedupe explanation.
• MEDIUM: this body claimed content mode lets "a Type 4 function body" lex. Type 4 function bodies are separate streams the interpreter never lexes; the clause is gone. The PdfLexer.cs comment inverted §7.2.3 (braces have their only syntax inside Type 4 function bodies, §7.10.5, outside content streams, and it cited §7.2.2); rewritten.
• MEDIUM: the fuzz test wrapped Open, GetPage and Run in one try admitting InvalidDataException, so a Run-time throw was indistinguishable from an open-time one. Split; Run admits only UnsupportedPdfFeatureException. The split exposed a pre-existing gap: resolving /Contents, a resource or a form through a malformed reference chain could throw InvalidDataException past Run. The outermost catch under What closes it; confirmed against the failing seed and a 30 000-iteration run.

LOWs taken in the same commit: /Matrix, /BBox and ExtGState /Font resolve indirect references; OnFormBegin documents a null boundingBox; operand cap 32 to 64 (Annex C.1); code 309 split from 302; UnknownOperator Info to Warning; §14.6.1 instead of §14.6.2 at the two nesting citations; reader-guide gotcha lists MaxFormXObjectDepth; negative /L and invalid /W//H//BPC reported; a second sign character rejected (the lexer already splits --5 into two tokens, so the check is defensive and has no dedicated test); "> >" and a Seek-after-one-byte-keyword case in PdfLexerContentModeTests. The pre-existing §7.2.2 mentions elsewhere in PdfLexer.cs were not touched.

The round-1 probe accepted a candidate when a token ran off a clipped window (more buffer beyond it) and treated the true end of a short buffer as the unbounded lexer would; round 2 replaced the accept with the second 4096-byte window described under What, because "ran off a clipped window" and "the token never closes" were indistinguishable from 128 bytes alone.

Round 2 (three reviewers at 298693e, same lenses)

One HIGH, six MEDIUM, all confirmed and fixed in b892675 (Sonnet) and 5f5f84b:

• HIGH (operands): AddElement decoded every /Contents element and held every chunk before Concatenate applied the 64 MiB cap, so a 103 KB file whose /Contents array named one 20 MiB stream 128 times reached a 4.6 GB peak heap. The budget is now charged per element as it decodes and resolution stops at zero; ContentStreamsDecoded (internal) pins that ten references to one oversized stream decode only four elements. The Concatenate cap stays as a backstop.
• MEDIUM (operands): HandleDo decoded the form before the exhausted-budget check (8 MiB form drawn 256 times allocated 6.9 GB). The check is hoisted above the decode.
• MEDIUM (inline): the clipped-window inconclusive accept flipped on byte count: a DCT image without /L whose data held an unclosed ( straddling the 128-byte mark was delimited correctly at 100 filler bytes and truncated at 110, reporting ContentStreamLexError and losing the trailing Q. Replaced by the two-window probe; tests sit on both sides of the boundary.
• MEDIUM (inline): §8.9.7 exempts ASCIIHexDecode/ASCII85Decode data from the single-whitespace rule after ID, and NOTE 2 says the /L count starts after that whitespace; the code consumed one byte for every filter, so /L 48 on an A85 image padded with two spaces was judged malformed and its payload re-lexed. Fixed as described under What.
• MEDIUM (prose): the 302 doc's "Covers" list omitted the wrong-type-at-right-arity case the code and TjOperand_notAnArray_... already handled.
• MEDIUM (prose): the 307 doc claimed interpretation always continues; it continues when the image data was still delimited (disallowed filter, length recovered by the scan) and stops for that stream when it was not (no ID, no EI, length past the end). Rewritten to say which.
• MEDIUM (prose): two invented ISO quotations. Matrix.cs now quotes §8.3.4 as written ("shall be multiplied before (premultiplied with) the existing transformation matrix"); TextState.cs cites §9.4.1, not §9.4.2, and quotes its own sentence ("may be specified only within a text object and shall not persist from one text object to the next").

LOWs and nits taken: an over-cap q/BMC push now leaves a credit its matching Q/EMC spends, so 65 balanced nests report 309 once and no 302; unknown operators clear the stack everywhere (What); the invoker's text matrices are saved around Do and a Do inside a text object reports 302; §8.10.1 b) applied and IContentVisitor.OnFormBegin says the matrix is already in the CTM; the cycle guard runs before the depth cap so a self-referencing form at MaxFormXObjectDepth = 1 reports FormXObjectCycle; unusable /XObject entries report instead of silently skipping; the CR LF retry runs before the 307 report; real-typed /L and /BPC outside Table 87's set reported; ContentOperators.IsKnown takes a span (no per-keyword string in the probe); DiagnosticSink and IContentVisitor doc lines corrected; Annex C.2 Table C.1, Table 50 and the Table 351/352 note at their citations; three more PdfLexerContentModeTests cases (}5, }/N, }(s)); the 307 dedupe (one per page, so a second image's distinct malformation is folded into the first) is documented rather than keyed on offset.

Declined, with the standard as the reason: the request to move the ColorSpace-resource comment's citation from §8.6.3 to §8.6.8 Table 73. The rule that CS/cs name only DeviceGray, DeviceRGB, DeviceCMYK and some Pattern spaces directly and everything else through the ColorSpace subdictionary is stated in §8.6.3 itself; Table 73 restates it and points back.

5f5f84b corrects two things in that fix-up: it described Do as a general graphics state operator (Table 50 lists it under XObjects; §8.2 Figure 9 is what excludes that category from a text object), and it saved the text matrices around a form but not the inside-a-text-object flag, so a form invoked from within BT/ET had its own Do judged by the invoker's state and a form ending in an unclosed BT left the invoker inside a text object. Both tests fail without the bracket. The §8.10.1 b) quotation it added was a paraphrase and now reads as the standard does.

Tests

ContentInterpreterTests (91: operand shapes, caps at the boundary with the 302/309 split, q/Q and cm, text-state operators, gs font, forms at depth 32 and 33, cycle, 4097 invocations, resources fallback, matrix and BBox handed over directly and by reference, the four form save/restore cases, the shared content budget, the three inline-image tiers with their fallbacks, the false-EI cases (binary noise, unknown printable operator, BX/EX, short second dictionary, a terminated string straddling the 128-byte window, unterminated literal and hex strings rejected at every length on both sides of it and past the 4096-byte window, a terminated 4000-byte literal after a real EI, 20 000 candidates), the AHx/A85 whitespace rule, the CR LF retry ordering, a real-typed /L, /BPC 3, the budget charged per /Contents element (ten references to one 20 MiB stream decode four, count pinned) and an 8 MiB form skipped from its ninth invocation once the budget is spent, the over-cap q/BMC credit, unknown operators clearing the stack, the form /Matrix in the CTM, Do inside a text object with the form judged on its own state and an unbalanced form BT not leaking out, cycle before depth at MaxFormXObjectDepth = 1, the three unusable /XObject shapes, abbreviation KATs incl. the /CS array, named colour spaces, JPX inline, missing ID, negative /L, invalid /W, split tokens, non-stream /Contents, image-filtered /Contents, the 64 MiB cap in one and two streams, retention past MaxDiagnostics, and a CsCheck run over byte-mutated content streams asserting Run never throws and always terminates), PdfLexerContentModeTests (27 cases: default mode still throws on {, }, lone >; content mode yields keywords; every other token kind identical across modes), PdfReaderDiagnosticCodeTests, PdfReaderOptionsTests.

Gates (local, REQUIRE_ORACLES=1, REQUIRE_VERAPDF=1, QPDF_HOME, POPPLER_HOME, VERAPDF_HOME set) at 5f5f84b

• dotnet build VellumPdf.slnx: 0 warnings, 0 errors
• dotnet format --verify-no-changes: clean
• eng/clean-room-check.ps1: passed
• Reader.Tests: 1271 total, 0 failed, 12 skipped (the 12 expected with oracles set)
• Conformance.Tests: 1281 total, 0 failed, 0 skipped
• Branch adds 0 em dashes and 0 -- against origin/main
• Not run locally: the AOT smoke and the coverage gate (CI runs both; the Reader floor MIN_VALID_LINES is unchanged in this PR and is re-measured after PR 9 and PR 11 per the plan).

Review lenses

1. §8.9.7 inline delimiting: /L, the computed length (component counts, /BPC packing, /IM masks), EI inside DCT data, the ID whitespace rule, the bounded probe.
2. §7.8.2 operand discipline (the exact-count rule and the stray-R consequence above) and Form XObject bombs: depth, cycle, budget, the shared byte budget, the implicit save/restore, the resources fallback, /Matrix and /BBox shapes.
3. Lexer-mode regression (every existing PdfLexer consumer unchanged, the mode opt-in) and the API-freeze checklist on MaxFormXObjectDepth and the ten codes.