docs(hashing): add hash computation site reference document (ITL-529) by kurodo3[bot] · Pull Request #238 · nauticalab/orcapod-python

kurodo3 · 2026-07-21T07:57:42Z

Summary

Adds docs/reference/hashing.md — a developer reference cataloguing all 14 hash computation sites across 6 usage groups in orcapod-python, with a master index table, per-site structured templates (inputs, algorithm, output format, uniqueness guarantee, exclusions), and a worked example section with concrete hash values
Adds superpowers/scripts/hash_audit_example.py — a runnable script that constructs a deterministic example pipeline (scores + attendance DictSources → Join → FunctionPod/FunctionJobNode + SideEffectPod + PipelineJob) with fixed source_id values and prints all concrete hash values for copy-paste into the doc
Wires reference/hashing.md into mkdocs.yml nav under a new Reference section

Hash sites covered

Group	Sites
1. Framework Object Identity	`content_hash()` (1), `pipeline_hash()` (2)
2. Source Provenance & System Tags	Schema hash (3), default `source_id` (4), per-row `record_id` (5), Join system tag suffix (6)
3. Pipeline DB Entry Keys	`compute_base_entry_id()` (7), `compute_pipeline_entry_id()` (8)
4. Side-Effect Record ID & Invocation Hash	Side-effect `record_id` (9), `invocation_hash` (10)
5. Data Function URI Hash	Output schema hash (11)
6. Pipeline Run Identity	`run_id` (12), `snapshot_hash` (13), `datagram_uuid` (14)

Test plan

uv run python superpowers/scripts/hash_audit_example.py runs without error and prints all hash values
uv run mkdocs build passes (no new warnings from reference/hashing.md)
Full test suite: 4583 passed, 93 skipped, 6 xfailed

Closes ITL-529

🤖 Generated with Claude Code

…t spec

… doc

…-529)

…tags (ITL-529)

…ation hash (ITL-529)

…L-529) Add Part 3 covering site 11 (DataFunction URI / output_schema_hash), sites 12+13 (PipelineJob run_id and snapshot_hash), and site 14 (datagram UUID). Also add the if __name__ == "__main__" guard and fix the SideEffectPod callback signature to accept **kwargs for data columns. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ash (ITL-529)

… Run Identity (ITL-529)

…es (ITL-529)

codecov · 2026-07-21T07:59:19Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…ew (ITL-529) - Fix site 9 preimage: remove NODE_CONTENT_HASH_COL (removed in ITL-533); fix INPUT_DATA_HASH_COL type pa.large_string → pa.large_binary - Fix site 9/10 uniqueness guarantee: pod-version scoping is via table path (uri + pipeline_hash()), not preimage — consistent with sites 7–8 - Fix §4 intro: remove stale NODE_CONTENT_HASH_COL narrative - Fix FunctionPod identity_structure(): bare data_function (not 1-tuple) when no ctx - Fix RootSource identity_structure(): actual return shape (class_name, schemas, source_id) - Fix ArrowTableStream fallbacks: document actual 3-tuple and schema-pair cases - Fix §1b FunctionPod note: ctx-aware and plain pods have different uri and DB table paths; shared pipeline_hash value does not imply shared table path - Fix site 6 algorithm: n_char is system_tag_n_char, not schema_n_char - Fix site 7 known exclusions: table path scoped by uri + pipeline_hash(), not pipeline_hash() alone

…sites-document-inputs-algorithm

Copilot

Pull request overview

Adds a developer-facing hashing reference for orcapod-python, plus a reproducible script to print concrete hash values used in the documentation, and wires the page into the MkDocs nav.

Changes:

Add docs/reference/hashing.md, cataloguing 14 hash computation sites and including a worked example with concrete values.
Add superpowers/scripts/hash_audit_example.py, a runnable script that constructs a deterministic pipeline and prints the relevant hash outputs.
Update documentation scaffolding (MkDocs nav + supporting superpowers artifacts / ignore rules).

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
superpowers/specs/2026-07-21-itl-529-hash-audit-design.md	Design spec for the hashing reference doc and audit script
superpowers/scripts/hash_audit_example.py	Script to generate concrete hash values for documentation
mkdocs.yml	Adds the new reference page to the MkDocs navigation
docs/reference/hashing.md	New hashing reference document (14 sites + worked example)
docs/metamorphic/plans/2026-07-21-itl-529-hash-audit-reference.md	Implementation plan for generating the doc + script
.gitignore	Adjusts ignore rules to allow committing superpowers scripts

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

kurodo3 · 2026-07-21T09:29:12Z

+Side-effect pods (and `FunctionPod` instances with a `ctx_arg_name`) use a parallel but
+distinct record key scheme. The row-level preimage structure mirrors §3; pod-version
+scoping is handled at the delivery log's table-path level via `pipeline_hash()` and `uri`,
+not at the preimage level — consistent with how `FunctionJobNode` scopes function cache
+entries in §3.


Fixed. This was a genuine code oversight from ITL-533: _build_invocation_context() in function_pod.py still included NODE_CONTENT_HASH_COL (as pa.large_string()) and used to_string()/large_string() for INPUT_DATA_HASH_COL — inconsistent with FunctionJobNode and SideEffectPod which were both updated by ITL-533.

The fix (commit bdae918):

Replaced the manual preimage construction in _build_invocation_context() with the shared _build_record_id_preimage() helper from function_node.py (deferred import to avoid circular dependency), then appends only _SIDE_EFFECT_RECOMPUTATION_INDEX_COL. This removes NODE_CONTENT_HASH_COL and switches INPUT_DATA_HASH_COL to to_prefixed_digest()/large_binary() — matching SideEffectPod exactly.

Updated hashing.md §4 / site 10 to reflect that both paths now share the same preimage structure.

Added a regression test in test_node_content_hash_redundancy.py asserting NODE_CONTENT_HASH_COL is absent and INPUT_DATA_HASH_COL is large_binary in the invocation context preimage.

All 4565 tests pass.

kurodo3 · 2026-07-21T09:29:21Z

+
+**Format:** Prose explaining the two-component structure, then per-site template.
+
+**Opening prose:** Side-effect pods (and `FunctionPod` instances with a `ctx_arg_name`) use a parallel but distinct record key scheme. The key difference from Section 3 is the inclusion of `NODE_CONTENT_HASH_COL` — so that changing the pod's implementation invalidates prior delivery records, even for the same input. The `invocation_hash` string is composed from two `ContentHash` components and exposed to the pod function as an idempotency key.


This is a historical planning spec artifact (superpowers/specs/), not a shipped library document. The authoritative documentation is docs/reference/hashing.md, which has been corrected. The spec pre-dates ITL-533 removing NODE_CONTENT_HASH_COL from the preimage and does not need updating.

kurodo3 · 2026-07-21T09:29:29Z

+| Inputs | System tags + `INPUT_DATA_HASH_COL` (as `pa.large_string()`) + `NODE_CONTENT_HASH_COL` (pod `content_hash().to_string()`, as `pa.large_string()`) + `_SIDE_EFFECT_RECOMPUTATION_INDEX_COL` (fixed `0`, `pa.int32`) |
+| Algorithm | `StarfixArrowHasher.hash_table(preimage)` → `.to_prefixed_digest()` |
+| Output format | `ContentHash` (internally); `.to_prefixed_digest()` → `bytes` when stored in the delivery log |
+| Uniqueness guarantee | Unique per `(tag lineage, input_data content, pod version)`; recomputation index is always `0` — side-effect pods do not version recomputations |
+| Known exclusions | Unlike sites 7–8, this includes `NODE_CONTENT_HASH_COL` — a deliberate difference ensuring that changing the pod version invalidates the delivery record even for unchanged inputs |


Same as the previous comment — this is a historical planning spec artifact that pre-dates ITL-533. The authoritative reference is docs/reference/hashing.md, which correctly documents both FunctionPod._build_invocation_context() and SideEffectPod using to_prefixed_digest()/large_binary() with no NODE_CONTENT_HASH_COL in the preimage (after the fix in commit bdae918).

kurodo3 · 2026-07-21T09:29:39Z

+    def _log_fn(data, ctx: InvocationContext) -> None:  # noqa: ANN001
+        captured_ctx.append(ctx)


This is a historical implementation plan artifact (docs/metamorphic/plans/). The actual script (superpowers/scripts/hash_audit_example.py) was already written with the correct signature: def _log_fn(ctx: InvocationContext, **kwargs) -> None: — matching how SideEffectPod calls the function with **{ctx_arg_name: ctx, **data_dict}.

…n_context preimage ITL-533 removed NODE_CONTENT_HASH_COL from the record_id preimage in FunctionJobNode and SideEffectPod but missed _build_invocation_context() in function_pod.py. This commit aligns the ctx-aware FunctionPod path with the same preimage structure used everywhere else: system_tags + INPUT_DATA_HASH_COL (large_binary) + recomputation_index=0. Also switches INPUT_DATA_HASH_COL from to_string()/large_string() to to_prefixed_digest()/large_binary() by delegating to the shared _build_record_id_preimage() helper from function_node.py (deferred import to avoid circular dependency). Adds regression test confirming NODE_CONTENT_HASH_COL is absent and INPUT_DATA_HASH_COL uses large_binary in the invocation context preimage. Updates hashing.md site 10 to correctly attribute pipeline_hash_ch to the pod's own pipeline_hash() rather than SideEffectPodStream. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

kurodo3 · 2026-07-21T09:29:55Z

Review response summary

Addressed all 4 Copilot review comments.

Genuine code fix (comment 1)

The review correctly identified that FunctionPod._build_invocation_context() still included NODE_CONTENT_HASH_COL in its preimage — an oversight from ITL-533, which removed it from FunctionJobNode and SideEffectPod but missed the ctx-aware FunctionPod path.

Commit bdae918:

Replaced the manual preimage construction in _build_invocation_context() with the shared _build_record_id_preimage() helper from function_node.py (deferred import to avoid circular dependency via cached_function_pod). This removes NODE_CONTENT_HASH_COL and switches INPUT_DATA_HASH_COL from to_string()/large_string() to to_prefixed_digest()/large_binary() — matching SideEffectPod exactly.
Fixed hashing.md site 10 to attribute pipeline_hash_ch to "the pod's own pipeline_hash()" rather than "SideEffectPodStream".
Added a regression test in test_node_content_hash_redundancy.py::TestPreimageShape asserting NODE_CONTENT_HASH_COL is absent and INPUT_DATA_HASH_COL uses large_binary in the invocation context preimage.

All 4565 tests pass.

Historical artifact comments (comments 2–4)

Comments on superpowers/specs/ and docs/metamorphic/plans/ point to pre-ITL-533 planning documents. These are archived artifacts; the authoritative reference is docs/reference/hashing.md. Replied inline to each.

kurodo3 Bot and others added 16 commits July 21, 2026 05:24

docs(specs): add hash audit design spec for ITL-529

f759b8c

docs(specs): fix FunctionPod ctx_arg_name exclusion note in hash audi…

257b607

…t spec

docs(specs): add worked example section to hash audit spec (ITL-529)

97e26bc

docs(plans): add implementation plan for ITL-529 hash audit reference…

b21e046

… doc

chore: create docs/reference and superpowers/scripts directories (ITL…

2f8246e

…-529)

feat(scripts): add hash_audit_example.py Part 1 — sources and schema …

b7fc539

…tags (ITL-529)

feat(scripts): add hash_audit_example.py Part 2 — entry IDs and invoc…

10b7a1b

…ation hash (ITL-529)

docs(hashing): add header, intro, and 14-row hash site index (ITL-529)

2ee2875

docs(hashing): add Section 1 — Framework Object Identity (ITL-529)

d7d38e4

docs(hashing): add Section 2 — Source Provenance & System Tags (ITL-529)

e00a554

docs(hashing): add Section 3 — Pipeline DB Entry Keys (ITL-529)

076a604

docs(hashing): add Section 4 — Side-Effect Record ID and Invocation H…

02a0950

…ash (ITL-529)

docs(hashing): add Sections 5+6 — Data Function URI Hash and Pipeline…

427111e

… Run Identity (ITL-529)

docs(hashing): add Section 7 — Worked Example with concrete hash valu…

0f56e73

…es (ITL-529)

docs(hashing): wire reference/hashing.md into mkdocs nav (ITL-529)

3309769

eywalker requested a review from Copilot July 21, 2026 08:47

chore: remove .gitkeep placeholder files (ITL-529)

d9e74c8

Copilot started reviewing on behalf of eywalker July 21, 2026 08:48 View session

Merge branch 'main' into eywalker/itl-529-audit-all-hash-computation-…

3b91fec

…sites-document-inputs-algorithm

Copilot AI reviewed Jul 21, 2026

View reviewed changes

eywalker approved these changes Jul 21, 2026

View reviewed changes

eywalker merged commit 6aa1cd9 into main Jul 21, 2026
11 checks passed

eywalker deleted the eywalker/itl-529-audit-all-hash-computation-sites-document-inputs-algorithm branch July 21, 2026 09:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs(hashing): add hash computation site reference document (ITL-529)#238

docs(hashing): add hash computation site reference document (ITL-529)#238
eywalker merged 20 commits into
mainfrom
eywalker/itl-529-audit-all-hash-computation-sites-document-inputs-algorithm

kurodo3 Bot commented Jul 21, 2026

Uh oh!

codecov Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

kurodo3 Bot Jul 21, 2026

Uh oh!

kurodo3 Bot Jul 21, 2026

Uh oh!

kurodo3 Bot Jul 21, 2026

Uh oh!

kurodo3 Bot Jul 21, 2026

Uh oh!

kurodo3 Bot commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		Format: Prose explaining the two-component structure, then per-site template.

		Opening prose: Side-effect pods (and `FunctionPod` instances with a `ctx_arg_name`) use a parallel but distinct record key scheme. The key difference from Section 3 is the inclusion of `NODE_CONTENT_HASH_COL` — so that changing the pod's implementation invalidates prior delivery records, even for the same input. The `invocation_hash` string is composed from two `ContentHash` components and exposed to the pod function as an idempotency key.

		def _log_fn(data, ctx: InvocationContext) -> None: # noqa: ANN001
		captured_ctx.append(ctx)

Uh oh!

Conversation

kurodo3 Bot commented Jul 21, 2026

Summary

Hash sites covered

Test plan

Uh oh!

codecov Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

kurodo3 Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

kurodo3 Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

kurodo3 Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

kurodo3 Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

kurodo3 Bot commented Jul 21, 2026

Review response summary

Genuine code fix (comment 1)

Historical artifact comments (comments 2–4)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jul 21, 2026 •

edited

Loading