Skip to content

Add native ClawDog round trip - #55

Merged
jfrench9 merged 1 commit into
RoboFinSystems:mainfrom
futureWA:streamace/clawdog-native-roundtrip
Sep 13, 2026
Merged

Add native ClawDog round trip#55
jfrench9 merged 1 commit into
RoboFinSystems:mainfrom
futureWA:streamace/clawdog-native-roundtrip

Conversation

@futureWA

@futureWA futureWA commented Sep 12, 2026

Copy link
Copy Markdown

Summary

Adds native ClawDog report JSON-LD as a first-class xbrlkit projection and importer beside holon and TAVI, so authored ClawDog reports and converted filings can round-trip through XbrlModel and use the existing serve/view/tooling path.

Changes

  • Adds to_clawdog / to_clawdog_report and from_clawdog_json / from_clawdog_report.
  • Wires ClawDog into build, serve, export_filing, JSON sniffing, and view pass-through.
  • Carries report/entity/period/unit/concept/fact/network structure, dimensions, calculation equations, fact provenance, concept references, parser fidelity fields (raw_value, source_hash), primary_document, and network role_id.
  • Adds the LodgeiT equivalence fixture across ClawDog, TAVI, and holon, plus a whole-model ClawDog round-trip gate covering the review edge cases.

Output Impact

BROADER COVERAGE / CLI CONTRACT:

  • xbrlkit build --format all now writes ClawDog output in addition to the existing projections.
  • export_filing gains a clawdog format.
  • serve / view can load and pass through .clawdog.jsonld reports.

Existing holon, TAVI, OIM, and property-graph output is intended to remain unchanged.

Testing

focused ClawDog gate -> 6 passed, 3 warnings
focused PR gate -> 53 passed, 99 warnings
full pytest -> 467 passed, 2 skipped, 162 warnings
ruff check -> All checks passed
ruff format --check -> 7 files already formatted
git diff --check -> clean

@futureWA

Copy link
Copy Markdown
Author

xbrlkit PR 55 - native ClawDog round trip

@jfrench9 jfrench9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Andrew — this is the PR I asked for and it is close. Before reading the asks, the measurements, because they are the good news.

What I checked, on this branch merged with current main (the branch is 11 commits behind; the merge is clean):

  • Six holons from the corpus (3M 10-K 0000066740-25-000006, Apple 0000320193-23-000106, NVIDIA 0001045810-26-000021 and -000052, 0001193125-26-277521, and a RoboLedger-authored report) → to_clawdogfrom_clawdog_json: periods, units, facts and networks come back identical on all six. Write is 0.8 s on the 3M filing, read 0.1 s.
  • An Arelle parse of the 3M 10-K (3,150 facts, 559 networks, cubes included) → ClawDog → back keeps all 3,150 facts and 559 networks. The disclosures map and the segment information_block are byte-identical from the parse and from the ClawDog reload — so, unlike TAVI, this format carries the definition networks and every tool on main works over it.
  • from_clawdog_json refuses a holon and a TAVI; json_kind classifies all three correctly.

So the shape is right. What follows is what stops the merge, then what I'd like, then nits.

Required

Three reader fallbacks overwrite a real empty value, and one field is dropped silently:

  1. deserialize/clawdog.py:158reporting_style … or "clawdog". Every SEC filing comes back with reporting_style="clawdog"; the source has None. Keep None.
  2. deserialize/clawdog.py:160 with serialize/clawdog.py:106 — the writer puts form or report_id in label, and the reader takes label as the form. A report with no form (the RoboLedger one) comes back with form == report_id. Write the form under lg:form only and stop reading label as a form.
  3. deserialize/clawdog.py:253namespace … or (prefix if local else ""). _text("") is falsy, so a concept whose namespace is "" comes back with its prefix as the namespace: 426 of 1,329 concepts on the 3M filing (country:, cyd:, …). Only fall back when the key is absent, not when it is empty.
  4. filing.primary_document is not written and not read, and it is not in the gap report either — the one thing the module docstring says will never happen. Write it, or name it in gaps.
  5. README.md:210 now says the platform's SEC pipeline projects with to_clawdog_report. It does not — please revert that line. The serialize/deserialize READMEs are fine.
  6. The PR body is the empty template. Output impact is not INTERNAL: build --format all now writes two more files and export_filing gains a format — that is additive coverage plus a CLI change, and it should say so. And name the filings you ran it against.

Suggested

  • Add the whole-model gate. tests/test_deserialize.py::test_holon_round_trips_the_whole_model is the bar the holon meets; the ClawDog test compares a hand-picked signature on a three-fact fixture. Put the existing model fixture through to_clawdog / from_clawdog_json and compare field by field the way that test does. Items 1–4 above all fail it, which is the point.
  • Empty the gap list. On a real parse the writer reports concept references, fact raw_value, fact source_hash and network role_id. Nothing in the format forbids carrying them — the format is defined in this module. Carry them and the corpus gate becomes model == back, the same fixed point the holon has. Related: serialize/clawdog.py:219 reports unmentioned concept for concept nodes the writer has just written unconditionally; that is not a gap. Drop it, or make it a note.
  • In serialize/README.md the ClawDog row sits above holon. Put it last; the holon is the package's own projection.

Nits

  • deserialize/clawdog.py:122 — the fallback sniff is "lg" in str(@context). Require documentType, or @context["lg"] == CLAWDOG_VOCAB.
  • serialize/clawdog.py:33 — the xbrl prefix is declared and never used.
  • The lg:AccountingEquation nodes restate every calculation arc, so the document is ~90% of the holon's size rather than "compact". Fine if you want them for ClawDog's own consumers; just not compact.
  • Node ids are slugs under the vocab namespace (lg:fact:<slug>) while identity actually travels in lg:internalId. Works for our round trip (4,628 ids on 3M, no collisions), but another JSON-LD consumer gets slugs. If ClawDog has an identity scheme of its own, use it here.
  • The doc-type IRI is under lodgeit.org with no document behind it. A sentence in the module docstring saying this module defines v1 would settle where the spec lives.
  • periods.py:3–5 — "Four sources" and then "all three have to agree".

Ping me when it is pushed and I will merge and release it the same day.

@futureWA
futureWA force-pushed the streamace/clawdog-native-roundtrip branch from 8ca0a38 to 1e3e030 Compare September 13, 2026 01:11
@futureWA
futureWA force-pushed the streamace/clawdog-native-roundtrip branch from 1e3e030 to a472ff5 Compare September 13, 2026 05:33

@jfrench9 jfrench9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on the new head: whole-model round trip identical on the six corpus holons and on an Arelle parse of the 3M 10-K (references, raw_value, source_hash, role_id all carried; writer gaps empty), and the disclosures / information_block output is byte-identical from the parse and from the ClawDog reload. Thanks Andrew — merging.

@jfrench9
jfrench9 merged commit 8d85c3d into RoboFinSystems:main Sep 13, 2026
4 of 7 checks passed
jfrench9 added a commit that referenced this pull request Sep 13, 2026
The ClawDog round trip (#55, released as 0.14.0) and the published-first
loader both touched _load_json: main adds the clawdog kind to the served
set, this branch inlines text-block fragments on load. Both kept.

Claude-Session: https://claude.ai/code/session_01GD7d1Rjg5PPV8swJUDFdyr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants