docs: record cloud source bundle materialization

Author: rohans0509

docs/codealmanac-launch/auth-api-contract.md

@@ -285,6 +285,17 @@ not echo transcript text. `POST /v1/capture/turns` accepts normalized routing
 metadata plus `artifactRef`. SQL stores refs and metadata, not copied transcript
 content.
 
+Implemented Slice 30 internal worker artifact read route:
+
+```text
+GET /api/internal/source-artifacts?ref=<source-artifacts-ref>
+```
+
+The route authenticates with `X-Internal-Secret`, returns raw artifact bytes,
+and includes source ref, sha256, and byte length headers. It is intentionally
+not a public `/v1` route. Hosted workers use it to materialize source refs into
+worker-local `sources/` folders.
+
 Implemented Slice 28 local capture state:
 
 ```text
@@ -313,6 +324,8 @@ POST /v1/capture/artifacts
 POST /v1/capture/turns
 POST /v1/auth/token/refresh
 
+GET  /api/internal/source-artifacts?ref=<source-artifacts-ref>
+
 GET  /v1/repositories
 GET  /v1/repositories/resolve?provider=github&owner=<owner>&repo=<repo>
 GET  /v1/repositories/{repo_id}

docs/codealmanac-launch/next-agent-brief.md

@@ -10,65 +10,73 @@ verification, launch-folder updates, commit, push, and RelayForge update.
 
 ## Last Completed Slice
 
-Slice 29 implemented capture transcript artifact upload through the narrow
-`cap_...` credential.
+Slice 30 implemented hosted source-bundle materialization for captured
+conversation refs.
 
 Implemented:
 
 - hosted worktree at
   `/Users/rohan/.config/superpowers/worktrees/usealmanac/hosted-baseline-convergence`
 - hosted branch `codex/workos-authkit-api-foundation`
-- hosted source-artifact service seam with filesystem-backed development store
-- hosted `/v1/capture/artifacts` and `/v1/capture/turns`
-- hosted `conversation_sources.source_ref` and matching migration
-- hosted/frontend DTO mirrors for capture upload DTOs
-- local Codex/Claude transcript normalizer for hook payloads
-- local capture Git routing probe for repo/branch/head metadata
-- local capture HTTP upload methods for raw artifact bytes and turn metadata
-- hidden provider hook upload path with `upload_status` diagnostic events
+- hosted `ConversationBatchSource.source_refs` instead of inline
+  `source_text`
+- hosted conversation ingest scheduling only for completed, routable,
+  ref-backed capture turns
+- hosted source-artifact `read(ref)` service/store port
+- protected internal route `GET /api/internal/source-artifacts?ref=...`
+- Modal worker materialization into
+  `.codealmanac-worker/sources/<batch-id>/manifest.json` and
+  `sessions/<provider>/<provider-session-id>-<hash>.jsonl`
+- hosted worker command bridge to the current Python CLI:
+  `codealmanac dev ingest <sources-dir> --foreground --using codex`
+- Modal image install of Python CodeAlmanac from a pinned git ref instead of
+  the old npm package
 
 Verified:
 
 ```text
 cd /Users/rohan/.config/superpowers/worktrees/usealmanac/hosted-baseline-convergence/backend
-uv run pytest tests/test_capture_tokens_api_contract.py tests/test_capture_upload_api_contract.py tests/test_hosted_conversation_sync_contract.py tests/test_conversation_ingest_scheduler.py tests/test_architecture_contract.py -q
+uv run pytest tests/test_conversation_ingest_scheduler.py tests/test_capture_upload_api_contract.py tests/test_internal_route_contract.py tests/test_modal_worker_contract.py tests/test_updates_contract.py tests/test_architecture_contract.py -q
+uv run pytest -q
 uv run ruff check .
+uv run ruff format --check .
 
-cd /Users/rohan/Desktop/Projects/codealmanac
-uv run pytest tests/test_capture_transcript_upload.py tests/test_cloud_capture_service.py tests/test_cli.py tests/test_public_contract.py tests/test_architecture.py -q
-uv run ruff check .
+cd /Users/rohan/.config/superpowers/worktrees/usealmanac/hosted-baseline-convergence
+git diff --check
 ```
 
-Counts: hosted focused backend `88 passed, 1 warning`; local focused tests
-`147 passed`.
+Counts: focused hosted backend `119 passed, 1 warning`; full hosted backend
+`301 passed, 1 warning`.
 
 ## Next Pressure Test
 
-Slice 30 should materialize source-artifact refs into cloud worker source
-bundles.
+Remove or narrow the hosted worker subprocess bridge.
 
 Pressure points:
 
-- teach the cloud worker/source-bundle path to read `source_ref` artifacts
-  instead of assuming inline `conversation_messages`
-- preserve pass-by-reference semantics in prompts and worker request files
-- decide whether capture-token turn uploads should mark ingest state due in the
-  same slice that source refs become materializable
-- keep the old `/api/cli/.../conversation-turns/*` inline-message route
-  compatibility-only
+- the hosted worker still calls `codealmanac` as a subprocess; the launch
+  contract says workers should call the Python engine/model API directly
+- the Modal image currently pins a CodeAlmanac git SHA, so the next slice must
+  either update the pin intentionally or replace the package-level bridge
+- cloud run storage still needs SQL-backed `runs`, `run_events`, and durable
+  run/source artifact refs that parallel the local control DB
+- hosted delivery still needs the expected-head check and commit/PR delivery
+  policy path
+- old inline-message conversation routes should remain compatibility-only
 
 ## Known Repo State
 
-The CodeAlmanac branch is `dev`. Slice 28 is pushed to `origin/dev` at
-`40ff8e0c feat: add cloud capture setup`; Slice 29 is currently dirty and
-ready for full verification, commit, and push.
+The CodeAlmanac branch is `dev`. Slice 29 is pushed to `origin/dev` at
+`d7a33cba feat: upload captured transcripts`. Slice 30 only changes launch docs
+in this repo.
 
 The hosted auth branch is
 `/Users/rohan/.config/superpowers/worktrees/usealmanac/hosted-baseline-convergence`
 on `codex/workos-authkit-api-foundation`. Slice 27 is pushed to origin at
 `c91e162 feat: add v1 CLI auth routes`; Slice 28 is pushed to origin at
-`36211ba feat: add capture credential API`. Slice 29 is currently dirty and
-ready for full verification, commit, and push.
+`36211ba feat: add capture credential API`; Slice 29 is pushed to origin at
+`5644a65 feat: add capture transcript upload`; Slice 30 is pushed to origin at
+`191d8d8 feat: materialize capture source refs`.
 
 The local wiki command currently fails on this checkout with:
 

docs/codealmanac-launch/progress.md

@@ -8,7 +8,7 @@ Percentages are planning estimates, not accounting metrics.
 
 ## Latest RelayForge Update
 
-Sent: 2026-07-02 after Slice 28 cloud capture install.
+Sent: 2026-07-02 after Slice 30 cloud source-bundle materialization.
 
 Route:
 
@@ -26,11 +26,11 @@ It was not present in `codealmanac/prd` when checked.
 
 | Area | Latest | Previous | Basis |
 | --- | ---: | ---: | --- |
-| CodeAlmanac backend/local | 90% | 88% | Slice 29 makes local capture hooks upload transcript artifacts and metadata; local trigger/run/workspace/delivery maturity is unchanged. |
-| CodeAlmanac CLI/public UX | 64% | 62% | Hidden capture hook now performs cloud upload, while repo, runs, status, and open commands remain. |
-| CodeAlmanac-hosted backend/auth/API | 45% | 40% | Hosted now accepts capture-token artifact and turn uploads; repo API, run storage, worker source-bundle materialization, and rate limits remain. |
-| Hosted frontend/onboarding | 15% | 15% | Slice 28 added DTO parity only; browser onboarding/configuration screens are still not implemented. |
-| Infra/deploy rename | 10% | 10% | Slice 29 changed capture/API/CLI only. |
+| CodeAlmanac backend/local | 90% | 90% | Slice 30 changed hosted worker/source-bundle behavior only; local trigger/run/workspace/delivery maturity is unchanged. |
+| CodeAlmanac CLI/public UX | 64% | 64% | Slice 30 does not add public CLI commands; cloud capture hook upload remains the latest CLI movement. |
+| CodeAlmanac-hosted backend/auth/API | 52% | 45% | Hosted conversation ingest now requires source refs, exposes internal artifact reads, materializes worker `sources/`, and passes the folder to CodeAlmanac. |
+| Hosted frontend/onboarding | 15% | 15% | Browser onboarding/configuration screens are still not implemented. |
+| Infra/deploy rename | 12% | 10% | Modal runtime now installs the Python CodeAlmanac package from a pinned git ref instead of the old npm package. |
 
 ## Update Rule
 

docs/codealmanac-launch/schema-contract.md

@@ -32,6 +32,26 @@ The capture-token path stores raw transcript bytes in the source-artifact store
 and stores `source_ref` plus routing metadata in SQL. It does not write
 conversation message content into `conversation_messages`.
 
+Slice 30 makes cloud conversation update runs ref-backed:
+
+```text
+runs.source_json.source_refs[]
+```
+
+`ConversationBatchSource` contains `batch_id` and source artifact refs. It does
+not contain rendered conversation markdown, transcript text, or copied session
+payloads. The hosted worker reads refs through the internal artifact edge and
+materializes a temporary worker-local folder:
+
+```text
+<checkout>/.codealmanac-worker/sources/<batch-id>/
+  manifest.json
+  sessions/<provider>/<provider-session-id>-<sha-prefix>.jsonl
+```
+
+That folder is an execution input, not durable product state. Durable state is
+the SQL batch/run rows plus source-artifact refs.
+
 ## Local Storage
 
 Local control DB:

docs/codealmanac-launch/verification-matrix.md

@@ -78,6 +78,19 @@ Current evidence:
   hosted backend capture/conversation/architecture tests (`88 passed, 1
   warning`), hosted ruff, local capture/CLI/public/architecture tests
   (`147 passed`), and local ruff.
+- Slice 30 added internal source-artifact reads:
+  `GET /api/internal/source-artifacts?ref=...`.
+- `backend/tests/test_internal_route_contract.py` proves the source-artifact
+  read route rejects missing internal secrets and returns raw bytes, content
+  type, source-ref, sha256, and byte-length headers when authenticated.
+- Slice 30 changed hosted conversation ingest to require captured source refs
+  before scheduling worker runs.
+- `backend/tests/test_capture_upload_api_contract.py` proves capture-token turn
+  upload stores `source_ref` and schedules ingest state once the turn is
+  completed, routable, and ref-backed.
+- `backend/tests/test_conversation_ingest_scheduler.py` proves run source JSON
+  contains source refs and no inline `source_text`, and proves turns without
+  source refs are not scheduled.
 
 ## CodeAlmanac Local Repo
 
@@ -421,6 +434,25 @@ Current evidence:
   a comment containing `m-* utility`.
 - Slice 26 does not implement versioned public API, CLI login/capture
   credentials, onboarding configuration screens, or hosted worker/run storage.
+- Slice 30 materialized cloud source-artifact refs in the Modal worker into
+  `.codealmanac-worker/sources/<batch-id>/manifest.json` and
+  `sessions/<provider>/<provider-session-id>-<hash>.jsonl`.
+- `backend/tests/test_modal_worker_contract.py` proves conversation batch runs
+  pass the materialized source folder to
+  `codealmanac dev ingest <sources-dir> --foreground --using codex`.
+- `backend/tests/test_modal_worker_contract.py` proves PR and branch update
+  commands use the current Python CodeAlmanac CLI bridge:
+  `codealmanac dev ingest github:pr:<n> --foreground --using codex` and
+  `codealmanac init --using codex --yes`.
+- `backend/tests/test_modal_worker_contract.py` proves the Modal image installs
+  Python CodeAlmanac from a pinned git ref and no longer installs the old npm
+  package.
+- Slice 30 full hosted backend verification passed with `uv run pytest -q`
+  (`301 passed, 1 warning`), `uv run ruff check .`,
+  `uv run ruff format --check .`, and `git diff --check`.
+- Remaining hosted worker risk: the worker still calls CodeAlmanac through a
+  subprocess bridge. A later slice should import and call the Python
+  engine/model API directly.
 
 ## Provider / Deployment
 

docs/codealmanac-launch/worklog.md

@@ -2,6 +2,38 @@
 
 ## 2026-07-02
 
+- Planned Slice 30 in
+  `docs/plans/2026-07-02-slice-30-cloud-source-bundle-materialization.md`.
+- Changed hosted conversation-batch update runs to store source artifact refs,
+  not rendered conversation text, in `ConversationBatchSource`.
+- Added hosted source-artifact reads through the service/store port and
+  protected internal route:
+  `GET /api/internal/source-artifacts?ref=...`.
+- Changed the hosted conversation ingest scheduler to select only captured
+  turns with non-null `source_ref` values and to schedule ingest when capture
+  uploads a completed routable turn with a source ref.
+- Added Modal worker source materialization:
+  `.codealmanac-worker/sources/<batch-id>/manifest.json` plus
+  `sessions/<provider>/<provider-session-id>-<hash>.jsonl`.
+- Updated hosted worker command construction to use the current Python
+  CodeAlmanac maintenance surfaces:
+  `codealmanac dev ingest <sources-dir> --foreground --using codex`,
+  `codealmanac dev ingest github:pr:<n> --foreground --using codex`, and
+  `codealmanac init --using codex --yes`.
+- Changed the Modal image to install Python CodeAlmanac from a pinned git ref
+  instead of installing the old npm `codealmanac@latest` package.
+- Kept the hosted worker on a process bridge for this slice. The next worker
+  slice should replace the process command with a direct Python engine/model API
+  call.
+- Verified Slice 30 focused hosted backend gates with
+  `uv run pytest tests/test_conversation_ingest_scheduler.py
+  tests/test_capture_upload_api_contract.py tests/test_internal_route_contract.py
+  tests/test_modal_worker_contract.py tests/test_updates_contract.py
+  tests/test_architecture_contract.py -q` (`119 passed, 1 warning`).
+- Verified Slice 30 full hosted backend with `uv run pytest -q`
+  (`301 passed, 1 warning`).
+- Verified Slice 30 hosted lint/format/diff gates with `uv run ruff check .`,
+  `uv run ruff format --check .`, and `git diff --check`.
 - Planned Slice 29 in
   `docs/plans/2026-07-02-slice-29-capture-transcript-upload.md`.
 - Added hosted source-artifact service seam with a filesystem-backed