Add native ClawDog round trip - #55
Conversation
|
xbrlkit PR 55 - native ClawDog round trip |
jfrench9
left a comment
There was a problem hiding this comment.
Thanks Andrew — this is the PR I asked for and it is close. Before reading the asks, the measurements, because they are the good news.
What I checked, on this branch merged with current main (the branch is 11 commits behind; the merge is clean):
- Six holons from the corpus (3M 10-K
0000066740-25-000006, Apple0000320193-23-000106, NVIDIA0001045810-26-000021and-000052,0001193125-26-277521, and a RoboLedger-authored report) →to_clawdog→from_clawdog_json: periods, units, facts and networks come back identical on all six. Write is 0.8 s on the 3M filing, read 0.1 s. - An Arelle parse of the 3M 10-K (3,150 facts, 559 networks, cubes included) → ClawDog → back keeps all 3,150 facts and 559 networks. The
disclosuresmap and the segmentinformation_blockare byte-identical from the parse and from the ClawDog reload — so, unlike TAVI, this format carries the definition networks and every tool on main works over it. from_clawdog_jsonrefuses a holon and a TAVI;json_kindclassifies all three correctly.
So the shape is right. What follows is what stops the merge, then what I'd like, then nits.
Required
Three reader fallbacks overwrite a real empty value, and one field is dropped silently:
deserialize/clawdog.py:158—reporting_style … or "clawdog". Every SEC filing comes back withreporting_style="clawdog"; the source hasNone. KeepNone.deserialize/clawdog.py:160withserialize/clawdog.py:106— the writer putsform or report_idinlabel, and the reader takeslabelas the form. A report with no form (the RoboLedger one) comes back withform == report_id. Write the form underlg:formonly and stop readinglabelas a form.deserialize/clawdog.py:253—namespace … or (prefix if local else "")._text("")is falsy, so a concept whose namespace is""comes back with its prefix as the namespace: 426 of 1,329 concepts on the 3M filing (country:,cyd:, …). Only fall back when the key is absent, not when it is empty.filing.primary_documentis not written and not read, and it is not in the gap report either — the one thing the module docstring says will never happen. Write it, or name it ingaps.README.md:210now says the platform's SEC pipeline projects withto_clawdog_report. It does not — please revert that line. The serialize/deserialize READMEs are fine.- The PR body is the empty template. Output impact is not INTERNAL:
build --format allnow writes two more files andexport_filinggains a format — that is additive coverage plus a CLI change, and it should say so. And name the filings you ran it against.
Suggested
- Add the whole-model gate.
tests/test_deserialize.py::test_holon_round_trips_the_whole_modelis the bar the holon meets; the ClawDog test compares a hand-picked signature on a three-fact fixture. Put the existingmodelfixture throughto_clawdog/from_clawdog_jsonand compare field by field the way that test does. Items 1–4 above all fail it, which is the point. - Empty the gap list. On a real parse the writer reports
concept references,fact raw_value,fact source_hashandnetwork role_id. Nothing in the format forbids carrying them — the format is defined in this module. Carry them and the corpus gate becomesmodel == back, the same fixed point the holon has. Related:serialize/clawdog.py:219reportsunmentioned conceptfor concept nodes the writer has just written unconditionally; that is not a gap. Drop it, or make it a note. - In
serialize/README.mdthe ClawDog row sits above holon. Put it last; the holon is the package's own projection.
Nits
deserialize/clawdog.py:122— the fallback sniff is"lg" in str(@context). RequiredocumentType, or@context["lg"] == CLAWDOG_VOCAB.serialize/clawdog.py:33— thexbrlprefix is declared and never used.- The
lg:AccountingEquationnodes restate every calculation arc, so the document is ~90% of the holon's size rather than "compact". Fine if you want them for ClawDog's own consumers; just not compact. - Node ids are slugs under the vocab namespace (
lg:fact:<slug>) while identity actually travels inlg:internalId. Works for our round trip (4,628 ids on 3M, no collisions), but another JSON-LD consumer gets slugs. If ClawDog has an identity scheme of its own, use it here. - The doc-type IRI is under
lodgeit.orgwith no document behind it. A sentence in the module docstring saying this module defines v1 would settle where the spec lives. periods.py:3–5— "Four sources" and then "all three have to agree".
Ping me when it is pushed and I will merge and release it the same day.
8ca0a38 to
1e3e030
Compare
1e3e030 to
a472ff5
Compare
jfrench9
left a comment
There was a problem hiding this comment.
Verified on the new head: whole-model round trip identical on the six corpus holons and on an Arelle parse of the 3M 10-K (references, raw_value, source_hash, role_id all carried; writer gaps empty), and the disclosures / information_block output is byte-identical from the parse and from the ClawDog reload. Thanks Andrew — merging.
The ClawDog round trip (#55, released as 0.14.0) and the published-first loader both touched _load_json: main adds the clawdog kind to the served set, this branch inlines text-block fragments on load. Both kept. Claude-Session: https://claude.ai/code/session_01GD7d1Rjg5PPV8swJUDFdyr
Summary
Adds native ClawDog report JSON-LD as a first-class xbrlkit projection and importer beside holon and TAVI, so authored ClawDog reports and converted filings can round-trip through
XbrlModeland use the existing serve/view/tooling path.Changes
to_clawdog/to_clawdog_reportandfrom_clawdog_json/from_clawdog_report.build,serve,export_filing, JSON sniffing, and view pass-through.raw_value,source_hash),primary_document, and networkrole_id.Output Impact
BROADER COVERAGE / CLI CONTRACT:
xbrlkit build --format allnow writes ClawDog output in addition to the existing projections.export_filinggains aclawdogformat.serve/viewcan load and pass through.clawdog.jsonldreports.Existing holon, TAVI, OIM, and property-graph output is intended to remain unchanged.
Testing
focused ClawDog gate -> 6 passed, 3 warnings
focused PR gate -> 53 passed, 99 warnings
full pytest -> 467 passed, 2 skipped, 162 warnings
ruff check -> All checks passed
ruff format --check -> 7 files already formatted
git diff --check -> clean