fix(kernel): always set /P bit 10 when writing an encryption dictionary

[Tim81]

Closes #397.

Why

ISO 32000-2 §7.6.4.2 Table 22, bit 10 of /P: "Not used. This bit was previously used to determine whether content could be extracted for the purposes of accessibility, however, that restriction has been deprecated in PDF 2.0. PDF readers shall ignore this bit and PDF writers shall always set this bit to 1 to ensure compatibility with PDF readers following earlier specifications." This library writes %PDF-2.0, and StandardSecurityHandler set the bit only when the caller passed PdfPermissions.Extract. A Permissions = All & ~Extract document therefore violated Table 22 and failed PDF/UA-1 §7.16-1 (the veraPDF test is (P & 512) == 512), which is why #396 had to commit a pre-fix fixture.

What

• StandardSecurityHandler.cs: the always-on mask becomes 0xFFFFF2C0 (bits 7, 8 and 10 forced on; bits 1 and 2 cleared; bits 13 to 32 reserved as 1). ComputePerms (Algorithm 10) reads PValue, so the /R 6 /Perms seal moves with it.
• PdfPermissions.Extract stays on the Shipped enum, no [Obsolete], no PublicAPI change; its XML doc now says the writer sets the bit regardless of the flag and readers ignore it. The flag still describes files written to earlier specifications and still feeds PdfDocument's PDF/UA-1 guard.
• PdfDocument's PDF/UA-1 guard (encrypting a PDF/UA-1 document without Extract throws) stays, by maintainer decision: it names a caller's declared-intent mistake. Its comment no longer implies the written bit depends on the flag.
• Reader unaffected: EncryptionSetup reports /Perms ?? /P masked with PdfPermissions.All and never refuses a clear bit 10 (readers "shall ignore this bit"). The committed Conformance: PDF/UA §7.16-1 — validate encrypted-file /P permission bit #138 fixtures (enc-aes-256-p-bit10-clear.pdf and its -4 twins) open and report exactly as before; Conformance.Tests with veraPDF required is green on them.

Expected values, derived from Table 22 by hand

P = (0xFFFFF2C0 | (bits & 0xFFF)) & ~3, with All = 0xF3C, Copy = 0x10, Extract = 0x200:

Permissions	Low 12 bits	/P
None	0x2C0	0xFFFFF2C0 = -3392 (was -3904)
Copy	0x2D0	0xFFFFF2D0 = -3376 (was -3888)
All & ~Extract	0xD3C | 0x2C0 = 0xFFC	0xFFFFFFFC = -4 (was -516)
All	0xFFC	-4 (unchanged)

Tests

• EncryptionTests: Permissions_None_clears_user_bits now expects PValue & 0xF3C == 0x200; a new Theory (PValue_matchesHandDerivedKnownAnswer) pins the four values above on StandardSecurityHandler.PValue; one end-to-end test (EncryptedDocument_allWithoutExtract_writesExpectedP) pins -4 in the written /Encrypt dictionary for All & ~Extract and re-checks it through the decrypted /Perms block; the /Perms seal test (Permissions = Print) now expects the reader to report Print | Extract, the direct consequence of the fix.

• PdfValidatorOracleTests (qpdf, Permissions = Copy): pinned P = -3888 becomes P = -3376, and the test now also asserts extract for accessibility: allowed. qpdf reports bit 10 on that line at /R >= 3, and on the same document the pre-fix writer produced extract for accessibility: not allowed, the discriminating pre-fix measurement. qpdf 12.3.2 output on the new bytes:

  P = -3376
  User password = openme
  Supplied password is user password
  extract for accessibility: allowed
  extract for any purpose: allowed
  

• StandardSecurityDecryptorTests p: -3904 literals are parsed inputs and are unchanged.

• Comments in UaEncryptionPermissionsRuleTests and Assets/README.md that described the pre-fix writer in the present or future tense now describe it as the pre-Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397 writer; the arithmetic they cite still yields the committed fixture's /P -516.

Review fix-ups (round 1)

• The reserved-bit comment in EncryptionTests counted bit positions in two frames at once; it now says which positions are counted from the LSB (0-based: 2..5, 8, 10, 11 cleared; 6..7 and 9 forced on) and maps them to Table 22's 1-based bits 7-8 and 10.

• docs/kernel-guide.md described the PDF/UA-1 guard as if omitting Extract still changed the written /P; both paragraphs now say the emitted bit is set either way, so the guard checks the caller's declared intent against the profile's promise.

• The Conformance Assets/README.md provenance block presented its recipe as reproducible; it is labelled as a run against the pre-Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397 writer at 1a85a66, and the arithmetic paragraph names the pre-Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397 mask it uses.

• The CHANGELOG bullet adds the All & ~Extract value (-4, same as All) and the reader-visible consequence: a document written without Extract reads back with Extract in PdfEncryptionInfo.Permissions.

• The handler's mask comment quotes Table 22's "Reserved. Must be 1." for bits 7-8 (it is unconditional there, not tied to /R); the /Perms test doc says "same permissions" and names the inputs DecryptPermsBlockForTest fixes.

• The Permissions = All fixture doc in UaEncryptionPermissionsRuleTests credited bit 10 to PdfPermissions.Extract; the mask forces it now, and the doc says so.

Review fix-ups (round 2)

• The AOT smoke harness (eng/aot/VellumPdf.AotSmoke/Program.cs) tampers /P and checks that the /Perms seal wins, but still expected the pre-fix writer's Print alone; with bit 10 forced on the sealed copy reads Print | Extract, so the aot-smoke job failed on all three runners. The expectation now matches the Kernel suite's twin assertion, and the surrounding comment says why. Found by the round-2 reviewer running run-aot-smoke.ps1 on a pristine checkout; it was not in the Verification list before, and is now.
• The Permissions = All hand-derivation in EncryptionTests claimed 0x2C0's bits were "already covered by 0xF3C", which is backwards. It now says the only bit All adds over All & ~Extract is Extract (0x200), which the mask supplies anyway.
• EncryptedDocument_allWithoutExtract_writesExpectedPAndPerms verified the /Perms of a second handler, not the one on disk; renamed to …writesExpectedP. Its /P regex now requires exactly one match in the written document rather than taking the first.
• Wording: the handler comment says "positions 0-1 (Table 22 bits 1-2)"; the qpdf oracle comment drops "for R >= 3" (Table 22 makes no revision distinction for bits 7-8); bit 10 is "extraction for accessibility" in the reserved-bits KAT; the PDF/UA-1 guard comment says the bit no longer "records" the caller's choice; the Conformance csproj comment and the Assets README describe the fixture's provenance in past tense ("the pre-Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397 writer", "same recipe").

Review fix-ups (round 3)

• The "Tests" bullet above described KATs that do not exist in that shape (four values "on the written /Encrypt dictionary"); it now names the Theory on PValue and the single end-to-end test that reads the file.
• The reserved-bits KAT in EncryptionTests still said bits 7-8 "must be 1 for R >= 3"; Table 22 has no revision qualifier there; round 1 had removed the same words from the handler comment and round 2 from the qpdf oracle comment.
• "Table 22 deprecates this bit" (CHANGELOG, PdfPermissions.Extract, the PDF/UA-1 guard, the handler comment) was imprecise: Table 22 deprecates the restriction the bit expressed, and the bit itself is live, with writers required to set it. All four now say so.
• PdfPermissions.Extract and PdfEncryptionInfo.Permissions XML docs now state that the reader reports Extract for any file whose bit 10 is set, including every file this library writes since Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397, so the "readers shall ignore it" quote does not read as a claim about this reader.
• The CHANGELOG entry gives the pre-fix value for the headline case (All & ~Extract: -516, not only None and Copy) and says which permission sets produced the violation (round 4 corrected that to "any set omitting Extract"); the kernel guide's PdfEncryptionSettings table marks Extract as no longer changing the written /P.
• Smaller wording: the /P regex comment gives the real reason there is exactly one match (no structure tree, AcroForm or signature in the fixture, since the writer's other /P keys are page back-references the regex would also match); the KAT summary says a mask bug cannot be blessed by an expectation taken from its own output; the Assets README calls -516 "that mask's arithmetic" rather than Table 22's; the qpdf oracle comment says qpdf consults bit 10 for the line at R >= 3.

The round-3 CI-equivalence lens re-ran every gate and all seven test assemblies at 9504460 with oracles required (0 failures; the 12 Reader skips are pre-existing ThirdParty reconstruction cases) and the mutation 0xFFFFF2C0 -> 0xFFFFF0C0, which six tool-free Kernel tests, the qpdf oracle KAT and the AOT smoke all catch. The blast-radius lens confirmed the only committed /P snapshot (GoldenTests.Encrypted_projection.verified.txt, -4 from All) is unchanged and that patching a written /P -3376 file to -3888 makes qpdf report a /Perms mismatch (exit 3), so the oracle's exit == 0 assertion is not vacuous.

Review fix-ups (round 4)

• The CHANGELOG entry said any narrower permission set produced the Table 22 violation. Only sets that omitted Extract did: All & ~Modify includes Extract and always wrote bit 10 set. The entry and its last sentence now say "any permission set that omitted Extract" and "a document written without Extract".
• Two comments that afe54d1 missed still called the bit itself deprecated (the reserved-bits KAT in EncryptionTests, the value-pinning comment in PdfValidatorOracleTests); both now say the restriction is what PDF 2.0 deprecates.
• The /P-regex comment called the writer's other /P keys page back-references; PdfStructureTree writes /P as a structure element's parent. It now says "indirect references (a widget's or signature's page, a structure element's parent)", which is the property the regex argument rests on.
• PdfEncryptionSettings.Permissions, the property a caller sets, gains a sentence saying that omitting Extract no longer clears a bit in the written /P but still fails the PDF/UA-1 guard; PdfPermissions.Extract points at the /Perms copy for /R 5 and 6.
• Smaller: the Assets README quotes Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397's full title; PdfEncryptionInfo.Permissions reads "this library's writer has since Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397"; the orphaned short comment lines the earlier rewordings left behind are reflowed (one survived into round 5, see below).

The round-4 CI-equivalence lens re-ran every gate and the Kernel, Layout, Conformance, Reader and Cli test assemblies at afe54d1 with oracles required (0 failures, the same 12 pre-existing Reader skips) and the mutation 0xFFFFF2C0 -> 0xFFFFF0C0 (7 kills: six Kernel tests and the qpdf KAT). The blast-radius lens confirmed the Reader reports Extract only when bit 10 is set in whichever value it decodes (/Perms at /R 5-6, /P below), so the committed -516 fixture still reads back without Extract, and that no README, guide, snapshot or workflow outside the diff pins a /P value.

Review fix-ups (round 5)

• The round-4 reflow of the PDF/UA-1 guard comment in PdfDocument.cs moved the mid-sentence break instead of removing it, so "Omitting Extract from" still sat alone on a line while 31e05e8 and the round-4 bullet above said the orphans were gone. The paragraph is now wrapped end to end.
• PdfEncryptionSettings.Permissions said omitting Extract "still fails the PDF/UA-1 guard" without the precondition every other doc site states; it now says a PdfConformance.PdfUA1 document fails the guard.
• Smaller: the Conformance test csproj comment no longer says "bit" twice for the committed fixture, the Assets README keeps one tense inside its /P derivation, and a filler "actually" leaves a test comment. The round-4 bullet above that said "all five test assemblies" now names them.
• Not changed: three commit messages (bf20cfd, 73-character subject; 9390847, 72 characters, one over the house limit; 8a8ae13, which still calls the bit itself deprecated) are history on this branch. The repository's squash default is COMMIT_MESSAGES, so a bare gh pr merge --squash would copy all of them into main; this PR is merged with an explicit --subject (the PR title) and --body that say the restriction is deprecated, so neither wording reaches main.

The round-5 CI-equivalence lens re-ran all eight gates at 31e05e8, including the coverage-threshold script from ci.yml (89.2 % merged over 31 869 lines, every assembly above the 40 % floor) and the PDF 2.0 inventory --check; the full solution in Release ran 5904 total, 0 failed, 14 skipped (the 12 Reader fixture-layout skips plus two transient pdftoppm probe timeouts that OracleGate.Transient never escalates), and the 0xFFFFF2C0 -> 0xFFFFF0C0 mutation reproduced round 4's 7 kills. The blast-radius lens listed all ten writer sites that emit a /P key and confirmed the nine outside /Encrypt are indirect references, that 31e05e8 changed no code token (comment-stripped hashes identical), and that the only committed /P snapshot is unchanged.

Review fix-ups (round 6)

• The round-5 bullet above claimed the squash-merge body is "written at merge time" as if that were how the repository merges; its squash default is COMMIT_MESSAGES (gh api repos/Tim81/VellumPDF --jq .squash_merge_commit_message), which would concatenate every branch commit message. The bullet now says the merge passes --subject and --body explicitly, and discloses 9390847's 72-character subject alongside bf20cfd's.
• The csproj comment for the bit-10-clear fixture had a 64-column line between two 90-plus ones, left by the round-5 shortening; reflowed.
• The CHANGELOG filed the change under ### Fixed only. Encryption: /P bits 7-8 are never set, contradicting the code comment #189, the previous Table 22 mask widening (2.1.0), paired its root-cause entry under Fixed with a Changed entry saying encrypted output bytes differ from earlier versions; Kernel: always set /P bit 10 in the encryption dictionary (ISO 32000-2 Table 22) #397 is the same change class with a wider consequence, so it now has the same pair. The Changed entry says which documents are affected (those written without Extract), that they now report Extract on re-opening, and that documents written with Extract carry the same /P value as before.

The round-6 prose lens re-read Table 22 at physical pp. 94-95 and confirmed all six deprecation sites scope it to the restriction, re-derived the mask and the four /P values, counted the em dashes over the whole branch diff (2 added, 6 removed, both additions rewrites of lines that already carried one), and confirmed every test count and the ten /P writer sites named in this body. The blast-radius lens re-derived every /P literal the branch adds or changes from Table 22 by hand (all matching), enumerated every consumer of PValue and of the Algorithm 10 seal across src/ and eng/ (two changed, the rest shown not to need an edit), read all 22 committed encrypted PDFs with qpdf (exactly one bit-10-clear file, P = -516; the other 21 are -4), recomputed the four SHA-256 pins, and confirmed no PublicAPI* file changes.

CI at 9390847 and at eb0ce36 both failed on the same test this branch does not touch, EncryptDictionaryDenialOfServiceTests.HugeEncryptDictionary_opensUnderTimeout ([Fact(Timeout = 10_000)], from #215), cancelled at 10.1 s and 10.0 s on the shared runner; it takes 0.33 s here in Release, and the same runner ran a 30 s-bounded Reader test from #398 at 33 s and 52 s the same day. The AOT smoke jobs were green on all three runners both times. A rerun of the eb0ce36 job passed with no code change, and the next push (0626bb1) passed first time. That timeout is a runner-load flake in a main-branch test; #400 tracks it together with the same-day flake in #398.

Verification

• dotnet build, dotnet format --verify-no-changes, clean-room check: green at ea2c5c5, bf20cfd, 9504460, afe54d1, 31e05e8, 9390847, eb0ce36, 0626bb1 and 6a5c49a. At 31e05e8: Kernel.Tests 1398 total, 0 failed; the qpdf oracle class 39 total, 0 failed; UaEncryptionPermissionsRuleTests 5 total, 0 failed (oracles required); pwsh ./eng/aot/run-aot-smoke.ps1 prints AOT smoke PASSED, exit 0 (it failed at 2f35691, see round 2). At 9390847 (comment-only): EncryptionTests 58/0, UaEncryptionPermissionsRuleTests 5/0, EncryptedFixtureDigestTests 4/0.
• Through dotnet exec with REQUIRE_ORACLES=1, QPDF_HOME, POPPLER_HOME and VERAPDF_HOME set (PowerShell): Kernel.Tests 1398 total, 0 failed, 0 skipped; Layout.Tests 220 total, 0 failed, 0 skipped. Reader.Tests 1029 total, 0 failed, 12 skipped; Conformance.Tests with veraPDF required 1281 total, 0 failed, 0 skipped (skip counts re-measured by the round-1 reviewer from a pristine checkout of ea2c5c5 with all three *_HOME set; the earlier figures came from a run without POPPLER_HOME).