Repository navigation
rdocx and rpptx: validate a schema subset, references and the stored package, refuse a ZIP name stored twice, open shared bookmark ids - #439
Conversation
The ZIP index of zip 9 keeps the last of two central directory entries that share a name, and other readers keep the first, so a review made with one tool and an edit made with another saw different parts. The open now compares the end record's entry count with the index, and when an entry is missing reads the central directory names and refuses the package with DuplicatePartName, naming the first repeated one. Without an entry limit, an end record the scan cannot read is left to the ZIP reader, as before. Part of tensorbee#435 (case 3).
…et in rdocx rdocx validate checked the relationship targets, the content types, the well-formedness of the parts the main document relates to and the style ids, and no content-model, value or reference rule. Every XML part, docProps included, is now checked for well-formed XML 1.0 through oxml_core::xml::validate_strict_xml_1_0, so a literal U+0001 is reported with its line and column. The main part and every header, footer, note and comment part are scanned once for a subset of the schema and for references: child order and allowed children of w:body, w:p and w:r, text where only elements may stand, the w:jc and w:color values, the w:id and w:author of a tracked change, wp:extent and a:ext sizes, the required children of m:rad, r: ids that name no relationship of the part, note references to no note, and w:t inside w:del. Each finding names the part and the byte offset, at most ten per part. Two bookmarks that share an id are a warning. Part of tensorbee#435 (cases 1, 2, 3 and 5).
Presentation::validate checks the deck as rpptx parsed it, serialized again from its model. A misordered presentation.xml or a p:cNvPr without name reads into the same model as a valid one, and a save keeps the stored bytes of every part it does not change, so such faults passed and survived a save. The new rpptx::validate_presentation_file reads the stored package first: XML parts that are not well-formed XML 1.0, a slide, layout, master, notes or handout part without its own content type, presentation.xml children out of the CT_Presentation order, unknown shape tree children, a p:cNvPr without id or name, and table rows whose a:tc count differs from the a:gridCol count. It then appends Presentation::validate. The findings reuse InvalidXmlChild and MissingContentTypeOverride, so ValidationIssue is unchanged. rpptx validate calls it. Part of tensorbee#435 (cases 3, 4 and 5).
DocumentIdentifiers::define refused a repeated w:bookmarkStart/@w:id with "duplicate bookmark id N in imported or preserved XML", so opening, text, render and validate all failed on such a file. Word has saved them (the _GoBack bookmark reusing the id of a _Toc bookmark of the body or of a header), and Word and LibreOffice open them. The id only pairs a start with its end, and the names stay distinct. A repeated id is now counted in repeated_bookmark_ids, as repeated_drawing_ids already counts wp:docPr ids. The ids are kept as they are. A staged edit that adds a repetition the opened document did not have is still refused, with "duplicate bookmark id N in authored XML", and new bookmarks take an unused id. Part of tensorbee#435 (case 6).
add_textbox and add_shape wrote a negative width or height into a:ext, which ST_PositiveCoordinate forbids, so rpptx validate rejected the saved deck and rpptx could not open it again. add_table already refuses it. sized_transform now refuses a negative width or height for textboxes and preset shapes, styled or not, before anything is written. A zero extent stays allowed, as the schema allows it. Part of tensorbee#435 (case 7).
Re-record the oxml-opc, rdocx, rdocx-cli, rpptx and rpptx-cli archives, whose member bytes change. The other rows are kept.
The stored-package check reported a row whose a:tc count differed from the a:gridCol count. LibreOffice writes a spanning cell with gridSpan and leaves out the hMerge cells after it, so a valid row of its pinned corpus deck LIBRE_OFFICE-100610-0.pptx was reported as short. Each cell now covers its gridSpan, and a row is reported only when the cells cover fewer columns than the grid has. Part of tensorbee#435 (case 4).
Re-record the rpptx archive, whose member bytes change with the gridSpan fix. The other rows are kept.
The rpptx archive with the gridSpan rule of tensorbee#439 on top of the integration. Tests, python3 scripts/readme_doctests.py --record-measurements Harness, unchanged, measurements only
|
Released in rpptx-v0.15.0 (tag), v0.17.0 (tag). The releases include a hardened equivalent of the stored-validation reference work. Bounded Word/PowerPoint schema and reference checks use existing channels, exact duplicate decoded ZIP names and forbidden XML characters refuse, and negative authored dimensions refuse while zero remains valid. Valid opaque payloads stay preserved. Complete OOXML validation is outside this delivered subset. Thank you @hadim for contributing the reference implementation. |
|
Thank you @hadim for the stored-validation reference implementation and for identifying its remaining open/save gap. F-X280 contract and evidence and children integrated a hardened equivalent, including forbidden-character opening refusal, exact duplicate ZIP-name rejection, the declared stored-schema/reference checks, shared-bookmark tolerance and negative-dimension refusal. Closing this PR as superseded by eb980e94, released in Word 0.17.0 and shared OOXML/PowerPoint 0.15.0 and qualified by successful integrated CI. It was not merged unchanged. #435's seven finite cases are addressed, including the missing case-5 opening protection. Full OOXML schema validation remains outside this subset. |
Part of #435
Fixed: cases 1, 2, 3, 4, 6 and 7, and the
validatehalf of case 5. Left: the open and save half of case 5 (see Limits). Case 2's script also builds its base document withadd_footnote, whose run ordervalidatenow reports, so its first line passes once #432 fixes that writer.rdocx validatechecks every XML part for well-formed XML 1.0, and the body, header, footer, note and comment parts against a subset of the WordprocessingML schema and against the package: the child order and the allowed children ofw:body,w:pandw:r, text where only elements may stand, thew:jcandw:colorvalues, the requiredw:idandw:authorof a tracked change,wp:extentanda:extsizes, the required children ofm:rad,r:ids that name no relationship of the part, footnote and endnote references to no note, and deleted text inw:t. Each finding names the part and the byte offset.rpptx validatenow reads the stored package before the deck: XML parts that are not well-formed, a slide, layout, master or notes part without its own content type,presentation.xmlchildren out of theCT_Presentationorder, unknown shape tree children, ap:cNvPrwithoutidorname, and table rows whose cells, each covering itsgridSpan, cover fewer columns than thea:gridColcount. LibreOffice writes a spanning cell without thehMergecells after it, so the count ofa:tcalone is not the rule.Both libraries refuse to open a ZIP that stores one entry name twice, naming it. rdocx opens a document whose bookmarks share an id, and
validatewarns about the shared id.add_textboxandadd_shaperefuse a negative width or height, asadd_tabledoes.Cause
validate_document_package(crates/rdocx/src/document.rs) checked the relationship targets, the content types, well-formedness of the parts the main document relates to and the style ids, and no content-model, value or reference rule.Document::from_packageopens the header case, and only the story readers (rdocx inspect, the Python constructor) refuse its danglingw:headerReference, sovalidatenever met it.ZipArchive(zip 9) indexes the central directory by name, so the second entry of a repeated name replaces the first andOpcPackage::from_reader_with_limitsnever saw the repetition. ItsDuplicatePartNamecheck only met aliases that normalize to one name. rdocx only checked the parts the main document relates to, and rpptx no part, for well-formedness.Presentation::validatechecks the deck as rpptx parsed it, serialized again from its model. A misorderedpresentation.xmlor ap:cNvPrwithoutnamereads into the same model as a valid one, and the save keeps the stored bytes of every part it does not change, so the stored fault survived. A slide part that falls back to thexmldefault content type has a content type, sopart_has_content_typepassed it.Charproduction, so the well-formedness scan accepted U+0001.DocumentIdentifiers::definerefused a repeatedw:bookmarkStart/@w:idwithduplicate bookmark id N in imported or preserved XML, so every open of such a file failed.textbox_memberandpreset_member(crates/rpptx/src/lib.rs) wrote the size intoa:extunchecked.CT_Table::newrefuses a non-positive size foradd_table.Change
oxml-opc,fix(opc):from_reader_with_limitsreads the end record's entry count on every open. When the ZIP index holds fewer names, it reads the central directory names and refuses the package withDuplicatePartNamenaming the first repeated one. Without an entry limit, an end record the scan cannot read is left to the ZIP reader, as before.rdocx validate,rpptx validate,Document()andPresentation()all reportduplicate OPC part name: word/document.xml.rdocx,feat(validate):document_xml_lexical_faultrunsoxml_core::xml::validate_strict_xml_1_0on every UTF-8 XML part and names the first forbidden character with its line and column. Every XML part, docProps included, is checked for well-formedness.document_story_findingsscans the main part and each story part once for the schema subset above (at most 10 findings per part listed), anddocument_note_idsreads the note ids. The CLI rustdoc andcrates/rdocx-cli/README.mdsay what is checked and that it is not a full schema validator. Fourrdocx-clivalidation tests built schema-invalid fixtures: content afterw:sectPr, andw:ins,w:delandw:moveFromwithoutw:idandw:author. They now insert beforew:sectPrand carry the attributes, and their footnote and endnote reference cases (which name a note the package does not hold) expect the new error.rpptx,feat(validate): newrpptx::validate_presentation_file(path)checks the stored package (stored_package_issues) and then appendsPresentation::validate(). If the deck does not open and the stored checks found something, it returns those. Findings reuseValidationIssue::InvalidXmlChildandMissingContentTypeOverride, so the enum is unchanged.rpptx validatecalls it.crates/rpptx-cli/README.mdgets a Validation section.rdocx,fix: a repeated bookmark id is counted inrepeated_bookmark_idsinstead of refused, asrepeated_drawing_idsalready does forwp:docPr/@id. A staged edit that adds a repetition the opened document did not have is still refused (duplicate bookmark id N in authored XML), throughreject_new_repeated_bookmark_idsnext toreject_new_repeated_drawing_ids. Ids are kept as they are, no renumbering. New bookmarks take an unused id.rpptx,fix:sized_transformrefuses a negative width or height foradd_textbox,add_shapeand the unstyled preset shapes of HTML and ODP imports, before anything is written. A zero extent stays allowed (ST_PositiveCoordinateincludes 0), so a flat line shape still works.Limits
w:body,w:pandw:rare not checked against their content model, nor enums other thanw:jcandw:color, nor required children of math elements other thanm:rad. Elements in other namespaces (mc:,w14:and later) are not checked. On the five pinned corpus documents ofscripts/docx-corpus-manifest.tsv(Word and LibreOffice) and the seven samples,validatereports nothing new.docs/hld/04-opc-and-packaging.mdstates this on purpose ("An entry that already held such a character when the package was read is written back while its bytes are unchanged, so producer content passes through verbatim"), and an edit that re-serializes the part already refuses the save. Refusing such a file at open, or rewriting the character, reverses that rule, so I left the choice to you.validatenow reports the part.add_equation(r"\sqrt{x}")(anm:radwithoutm:deg) andadd_footnote(w:rPrafterw:footnoteRef) now failvalidate. Both writers are fixed under rdocx: markup the library writes breaks the schema or differs from Word (equations, notes, hyperlinks, cells, numbering, TOC, replace) #432. Until then a document that holds such a note or radical failsrdocx validateandDocument.validate().w:tinsidew:delis reported because ECMA-376 Part 1 17.3.3.7 (delText) says deleted run text usesw:delText, as I remember it. I could not check what Word does with such a file.rpptx::validate_presentation_fileis new public API inrpptx. The PythonPresentation.validate()andDocument.validate()keep their signatures.Document.validate()gets the new rdocx checks, andPresentation.validate()still checks the deck as parsed, not the stored package.Tests
New tests, one per case:
crates/oxml-opc/src/package.rsissue435_an_entry_name_stored_twice_is_refused_naming_the_part(case 3), with and without an entry limit. Fails without the fix: the package opens holding the second copy.crates/rdocx/tests/regression_test.rs:issue435_validate_reports_wordprocessingml_schema_faults(case 1): the base document validates, and each of the ten faults is an error naming/word/document.xml.issue435_validate_reports_dangling_references(case 2): picture, header, footnote andw:tinw:del.issue435_validate_reports_a_malformed_document_properties_part(case 3).issue435_validate_reports_a_character_xml_forbids(case 5).issue435_bookmarks_sharing_an_id_open_and_render(case 6): the same part and body plus header, open, text, layout, validate (valid, with the warning), edit and reopen.imported_and_preserved_collisions_fail_before_mutationnow opens its bookmark collision. Twodocument.rsunit tests on identifier scan order are updated, and the authored repetition case now expectsduplicate bookmark id 1 in authored XML.crates/rpptx/tests/integration.rs:issue435_validate_presentation_file_reports_stored_faults(case 4, the six faults),issue435_validate_presentation_file_reports_a_malformed_document_properties_part(case 3),issue435_validate_presentation_file_reports_a_character_xml_forbids(case 5) andissue435_new_shapes_refuse_a_negative_size(case 7, including a zero-height shape), andissue435_a_spanned_row_without_hmerge_cells_covers_its_grid(agridSpan="2"cell with nohMergecell after it, as LibreOffice writes it, is not a short row).crates/rdocx-py/tests/test_core.pytest_issue435_bookmarks_sharing_an_id_open(case 6, both layouts) andcrates/rpptx-py/tests/test_documented_examples.pytest_issue435_add_textbox_and_add_shape_refuse_a_negative_size(case 7).Runs on linux-x86_64, debug,
RUST_MIN_STACK=8388608:cargo test -p oxml-opc: 89 passed, 1 ignored.cargo test -p rdocx --test regression_test: 2474 passed, 7 ignored.--test integration_test: 467 passed, 4 failed, all four on this machine's LibreOffice 26.2.6.3 against the pinned 26.2.5.2.--lib: 821 passed, 1 failed,fx195_variable_svg_paint_matches_independent_static_instances, which needshb-subset14.4.0, not installed here.body_lookup_work_is_linear_at_250_and_1000was skipped in debug (over 30 minutes) and passes with--release.cargo test -p rdocx-cli: 101 passed.cargo test -p rpptx, with the fetchedcorpus/anduvon PATH for the python-pptx oracle: lib 199 passed,integration437 passed, 25 ignored.cargo test -p rpptx-cliwith the corpus: 61 passed,validate_rejects_corruption_and_accepts_the_pinned_corpusincluded. A first version counteda:tcwithoutgridSpanand failedcorpus/pptx/LIBRE_OFFICE-100610-0.pptx(slide 8, a row of 10 cells, one spanning 3, over 12 columns).fix(validate): count gridSpan when rpptx validate checks a table rowfixes it.rpptx validateon the 51 decks ofcorpus/gives the same verdict as on main.cargo clippy -p oxml-opc -p rdocx -p rpptx -p rdocx-cli -p rpptx-cli --all-targets --all-features -- -D warnings,cargo fmt --all --checkandscripts/prose_check.py: clean.test_core.py,test_formatting_tables.py,test_python_docx_parity.py,test_shared.pyandtest_documented_examples.pygive 2105 passed, 2 skipped, 56 failed. 48 are fx195 and fx206 tests that need theF_X195_*oracle environment, 3 need the pinned LibreOffice 26.2.5.2, 1 needs a coordinator-authenticatedRDOCX_CLI, and 4 aretest_fx268_picture_normalization_reuses_actual_checked_rounds, whose producer path reportsproducer_currentinstead ofproducer.seed_eligiblehere. It fails the same way on main 24af95d in this repository's own CI (Python bindings job,KeyError: 'producer.seed_eligible'), and nothing in this branch touches picture normalization.scripts/hash_harness.py --check: 49 entries match. No sample output changes.scripts/readme_doctests.pypasses after re-recording theoxml-opc,rdocx,rdocx-cli,rpptxandrpptx-cliarchives, and therpptxarchive again after the gridSpan fix, in the measurement commits.rdocx validateon the five pinned corpus documents ofscripts/docx-corpus-manifest.tsvand the seven samples reports nothing new.rdocx validateandrpptx validategive a new verdict only on the issue's fault files, among the files the issue's scripts write.Repro
The issue's seven scripts on this branch (debug CLIs and bindings):