Independently verify real Git pack and index bytes

.github/workflows/ci.yml

@@ -64,16 +64,22 @@ jobs:
           cmp docs/demo/git-dag-v1/verify.txt "$RUNNER_TEMP/verify.txt"
           env -u PYTHONHOME -u PYTHONPATH python -B -m git_dag_lab inspect --compact > "$RUNNER_TEMP/inspect.json"
           cmp evidence/git-dag-v1.json "$RUNNER_TEMP/inspect.json"
+          env -u PYTHONHOME -u PYTHONPATH python -B -m git_dag_lab pack-verify > "$RUNNER_TEMP/pack-verify.txt"
+          cmp docs/demo/git-pack-index-v1/verify.txt "$RUNNER_TEMP/pack-verify.txt"
+          env -u PYTHONHOME -u PYTHONPATH python -B -m git_dag_lab pack-inspect --compact > "$RUNNER_TEMP/pack-inspect.json"
+          cmp evidence/git-pack-index-v1.json "$RUNNER_TEMP/pack-inspect.json"
 
       - name: Check tests, evidence freshness, and shell boundary
         shell: bash
         run: |
           set -euo pipefail
           env -u PYTHONHOME -u PYTHONPATH python -W error -m unittest discover -s tests -v
           env -u PYTHONHOME -u PYTHONPATH python -B tools/generate_evidence.py --check
+          env -u PYTHONHOME -u PYTHONPATH python -B tools/generate_pack_evidence.py --check
           bash -n tools/capture_report.sh
+          bash -n tools/capture_pack_report.sh
           test "$(shellcheck --version | awk '$1 == "version:" {print $2}')" = "0.9.0"
-          shellcheck tools/capture_report.sh
+          shellcheck tools/capture_report.sh tools/capture_pack_report.sh
           test "$(git status --porcelain=v1 --untracked-files=normal --ignore-submodules=none)" = ""
 
   distribution:
@@ -119,6 +125,11 @@ jobs:
             env -u PYTHONHOME -u PYTHONPATH "$runtime/bin/python" -B -m git_dag_lab verify
           ) > "$RUNNER_TEMP/wheel-verify.txt"
           cmp docs/demo/git-dag-v1/verify.txt "$RUNNER_TEMP/wheel-verify.txt"
+          (
+            cd "$RUNNER_TEMP"
+            env -u PYTHONHOME -u PYTHONPATH "$runtime/bin/python" -B -m git_dag_lab pack-verify
+          ) > "$RUNNER_TEMP/wheel-pack-verify.txt"
+          cmp docs/demo/git-pack-index-v1/verify.txt "$RUNNER_TEMP/wheel-pack-verify.txt"
           test "$(git status --porcelain=v1 --untracked-files=normal --ignore-submodules=none)" = ""
 
       - name: Upload verified distributions

.github/workflows/codeql.yml

@@ -0,0 +1,46 @@
+name: CodeQL
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  security-events: write
+
+concurrency:
+  group: codeql-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  analyze:
+    name: Analyze (${{ matrix.language }})
+    runs-on: ubuntu-24.04
+    timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        language:
+          - python
+    steps:
+      - name: Check out the immutable event revision
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 1
+          persist-credentials: false
+
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6
+        with:
+          languages: ${{ matrix.language }}
+          queries: security-extended
+
+      - name: Analyze
+        uses: github/codeql-action/analyze@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6
+        with:
+          category: /language:${{ matrix.language }}

README.md

@@ -1,12 +1,14 @@
 # Git DAG Evidence Lab
 
-> A dependency-free Python systems lab that constructs a real Git object database from plumbing commands and independently verifies every byte-addressed object.
+[![CI](https://github.com/omar07ibrahim/hello-git/actions/workflows/ci.yml/badge.svg)](https://github.com/omar07ibrahim/hello-git/actions/workflows/ci.yml) [![CodeQL](https://github.com/omar07ibrahim/hello-git/actions/workflows/codeql.yml/badge.svg)](https://github.com/omar07ibrahim/hello-git/actions/workflows/codeql.yml)
+
+> A dependency-free Python systems lab that constructs real Git object storage, then independently verifies loose-object DAGs plus closed-subset pack v2/index v2 bytes.
 
 The experiment demonstrates a subtle but important property: a merge commit and a rebase-shaped replay can resolve to **exactly the same tree** while preserving **different histories**.
 
 ![Real offline Git DAG evidence report showing the verified graph, checks, CLI receipt, and object envelope](docs/assets/git-dag-report.png)
 
-<p align="center"><sub>Attested 1440×1800 Chromium capture of the checked-in offline report. No external assets, JavaScript, network, secrets, or host-repository data.</sub></p>
+<p align="center"><sub>Attested 1440×1800 Chromium capture of the checked-in offline report. Provenance: report receipt `2da1ccd8…69c84`; screenshot SHA-256 `e539db11…e17e`. No external assets, JavaScript, network, secrets, or host-repository data.</sub></p>
 
 ## Quick start
 
@@ -15,6 +17,8 @@ Requirements: Python 3.10+ and Git 2.29+. There are no runtime Python dependenci
 ```bash
 python3 -m git_dag_lab verify
 python3 -m git_dag_lab inspect
+python3 -m git_dag_lab pack-verify
+python3 -m git_dag_lab pack-inspect
 python3 -m unittest discover -s tests -v
 ```
 
@@ -34,6 +38,29 @@ The fixture contains three blobs, six trees, and five commits. `merge` stores th
 
 The replay is called **rebase-shaped** because it is constructed directly with `git commit-tree`. The lab does not claim to execute porcelain `git rebase`.
 
+
+## A second systems slice: verify pack and index bytes
+
+The pack path stores three fixed synthetic blobs in a fresh private SHA-1 repository, invokes real `git pack-objects`, then removes the repository after independently decoding both generated files. The CLI receipt below is actual stdout from that production path:
+
+![Exact real pack-verify stdout with actual pack and receipt digests](docs/assets/git-pack-cli.svg)
+
+The offline report is rendered from the same canonical receipt and captured by digest-pinned Chromium in a read-only, network-disabled container:
+
+![Actual pinned-Chromium screenshot of the verified pack v2 and index v2 report](docs/assets/git-pack-report.png)
+
+<p align="center"><sub>Attested 1440×1500 Chromium capture. Provenance: pack report receipt `a884aed4…a81b0`; screenshot SHA-256 `ad1f615d…4d63`. Three deterministic synthetic blobs; no delta support, arbitrary-repository compatibility, authentication, network, or host data.</sub></p>
+
+The verifier does not trust the pack filename or Git's index. It parses the variable-length pack entry headers, bounds each zlib stream, reconstructs logical blob IDs, verifies the pack trailer, then parses the 256-entry cumulative fanout table, sorted OIDs, CRC32 rows, 32/64-bit offsets, pack binding, and index checksum.
+
+![Actual byte offsets and sizes decoded from the generated pack and index](docs/assets/git-pack-layout.svg)
+
+![Actual populated fanout buckets, sorted object IDs, and lookup ranges](docs/assets/git-pack-fanout.svg)
+
+![Receipt-derived integrity chain from pack header through index checksum](docs/assets/git-pack-integrity.svg)
+
+This is deliberately a closed subset: pack v2 and index v2, at most 64 objects, 1 MiB files, 256 KiB expanded objects, and non-delta entries only. Delta entries (OFS/REF), other object formats, arbitrary repositories, reachability, and pack optimization are not claimed. SHA-1 and CRC32 model Git storage integrity here; neither is presented as authentication, a signature, or collision-resistant security.
+
 ## The hard part: verify Git without trusting Git
 
 Writing an object with Git and asking Git to identify it would only prove that Git agrees with itself. This lab reads the raw stored bytes and independently computes:
@@ -51,7 +78,7 @@ SHA-1 is used because this scenario models a SHA-1 Git object database. Here it
 ## Isolation and execution boundaries
 
 - Git is resolved once to an absolute executable and invoked with argument arrays, never a shell.
-- 10 local subcommands are allow-listed, while one fixed isolated `git init` creates the bare database; transport commands and remote-looking arguments are rejected.
+- 11 local subcommands are allow-listed, including the bounded `pack-objects` path, while one fixed isolated `git init` creates each bare database; transport commands and remote-looking arguments are rejected.
 - `HOME`, `XDG_CONFIG_HOME`, and `TMPDIR` are private; inherited Git config, hooks, replacement objects, identity, and object-directory redirects are ignored.
 - fixed synthetic identity `dag-lab@example.invalid`, fixed UTC timestamps, and fixed LF payloads make object IDs reproducible.
 - symlinked workspace components are rejected; stdout/stderr are spooled privately and checked before bounded reads.
@@ -61,24 +88,26 @@ See [SECURITY.md](SECURITY.md) for the threat model and trusted-input boundary.
 
 ## Evidence pipeline
 
-Every README visual begins with the same canonical CLI document. The generator runs fresh experiments twice, requires byte-identical outputs, derives the SVG and offline HTML, and binds each artifact into a hash manifest. Digest-pinned Chromium then captures the report in a read-only container with `--network none`. A separate attestation binds the exact report, rendered DOM, PNG, browser binary/version, container digest, isolation policy, viewport, and capture-script hash; without that attestation, the generator refuses to call the screenshot verified.
+Every README visual begins with a canonical production CLI document. The DAG and pack generators each run fresh experiments twice, require byte-identical outputs, derive their SVGs and offline HTML, and bind every artifact into a hash manifest. Digest-pinned Chromium captures both reports in read-only containers with `--network none`. Separate attestations bind each exact report, rendered DOM, PNG, browser binary/version, container digest, isolation policy, viewport, and capture-script hash; without the matching attestation, a generator refuses to call its screenshot verified.
 
 ![Architecture of the fixed scenario, real Git plumbing, independent verification, and evidence publication pipeline](docs/assets/evidence-pipeline.svg)
 
 ### Reproduce the checked-in evidence
 
 ```bash
-# Verify JSON, transcripts, SVGs, HTML, source hashes, and the existing PNG.
+# Verify both JSON/transcript/visual/report/manifest packages and PNG attestations.
 python3 -B tools/generate_evidence.py --check
+python3 -B tools/generate_pack_evidence.py --check
 
-# Rebuild evidence and recapture the report with the pinned browser container.
+# Rebuild and recapture either offline report with pinned Chromium.
 tools/capture_report.sh
+tools/capture_pack_report.sh
 
-# Run all engine, boundary, CLI, evidence, and provenance tests.
+# Run all parser, boundary, CLI, evidence, and provenance tests.
 python3 -W error -m unittest discover -s tests -v
 ```
 
-Current verified baseline: **61 tests**, **9/9 graph invariants**, **57 isolated Git invocations**, report receipt `2da1ccd8…69c84`, and screenshot SHA-256 `e539db11…e17e`.
+Current verified baseline: **89 tests**, **9/9 graph invariants**, **7/7 pack/index checks**, **57 isolated Git invocations** in the DAG evidence run, two independently replayed evidence packages, and two attested offline browser captures.
 
 | Artifact | What it proves |
 |---|---|
@@ -89,13 +118,22 @@ Current verified baseline: **61 tests**, **9/9 graph invariants**, **57 isolated
 | [`rendered-dom.html`](docs/demo/git-dag-v1/rendered-dom.html) | Actual DOM emitted by Chromium during the attested capture |
 | [`capture-attestation.json`](docs/demo/git-dag-v1/capture-attestation.json) | Report/DOM/PNG hashes plus verified browser, container, isolation, viewport, and script provenance |
 | [`manifest.json`](docs/demo/git-dag-v1/manifest.json) | SHA-256, byte size, role, source hashes, normalized argv, and attestation receipt |
-| [`git-dag-report.png`](docs/assets/git-dag-report.png) | Actual Chromium rendering of the report at 1440×1800 |
+| [`git-dag-report.png`](docs/assets/git-dag-report.png) | Actual Chromium rendering of the DAG report at 1440×1800 |
+| [`evidence/git-pack-index-v1.json`](evidence/git-pack-index-v1.json) | Canonical real pack/index receipt, physical entry order, cross-bound rows, and non-claims |
+| [`git-pack-cli.svg`](docs/assets/git-pack-cli.svg) | Exact production `pack-verify` stdout rendered as an accessible terminal panel |
+| [`git-pack-layout.svg`](docs/assets/git-pack-layout.svg) | Actual pack offsets/sizes and index-table byte counts |
+| [`git-pack-fanout.svg`](docs/assets/git-pack-fanout.svg) | Actual non-empty fanout buckets and sorted OID ranges |
+| [`git-pack-integrity.svg`](docs/assets/git-pack-integrity.svg) | Receipt-derived pack/index checksum and row-binding workflow |
+| [`git-pack-report.png`](docs/assets/git-pack-report.png) | Actual Chromium rendering of the pack/index report at 1440×1500 |
+| [`git-pack-index-v1/manifest.json`](docs/demo/git-pack-index-v1/manifest.json) | Hash/size/source/command/capture inventory for every pack visual and output |
 
 ## Test coverage by risk
 
 The standard-library suite exercises more than happy-path graph construction:
 
 - independent blob, tree, and commit envelope hashes;
+- pack v2 headers, bounded zlib streams, logical OIDs, trailer checksum, and explicit delta rejection;
+- index v2 fanout, sorted OIDs, CRC32 rows, small/large offsets, pack binding, and checksum mutations;
 - exact object/ref inventories, parent ordering, reachability, and ancestry;
 - Git's special `directory/` tree ordering, truncated binary objects, and malformed headers;
 - hostile inherited Git environment and fake global identity/config;

SECURITY.md

@@ -1,6 +1,6 @@
 # Security model
 
-Git DAG Evidence Lab creates a temporary bare repository below the selected workspace and removes it after each run. It does not inspect the repository that contains this source code.
+Git DAG Evidence Lab creates a temporary bare repository below the selected workspace for each DAG or pack experiment and removes it after the run. It does not inspect the repository that contains this source code.
 
 The command boundary is deliberately narrow:
 
@@ -13,6 +13,12 @@ The command boundary is deliberately narrow:
 - temporary roots with symlinked path components are rejected.
 - stdout and stderr are captured in private temporary files and rejected before loading into memory when either exceeds 1 MiB.
 
+## Pack/index closed subset
+
+The pack experiment passes only three fixed synthetic blob IDs to `git pack-objects`; it does not accept a repository path, revision, ref, or caller-provided object list. Generated `.pack` and `.idx` files must be regular, single-link files no larger than 1 MiB and must remain the same inode and size across the bounded read.
+
+The independent parser accepts pack v2 and index v2 only. It rejects OFS/REF deltas, more than 64 objects, objects expanding beyond 256 KiB, invalid or unterminated zlib streams, duplicate logical objects, non-canonical fanout/large-offset tables, and any mismatch among logical object IDs, CRC32 rows, offsets, pack trailer, index pack binding, or index checksum. These checks establish the fixed fixture's storage integrity; they do not establish provenance, authenticity, repository reachability, or safety of arbitrary Git data.
+
 ## SHA-1 scope
 
 The lab uses SHA-1 because the scenario explicitly models a SHA-1 Git object database. The independent envelope calculation demonstrates deterministic content addressing and detects accidental changes in these fixtures. It is not a signature, authentication mechanism, or claim of modern collision resistance.