fix(ducklake): pin HTTPS for delta-kernel staging reads via duckgres

[EDsCODE]

Problem

On cache-proxy-enabled duckgres clusters, the org's ambient S3 secret deliberately carries USE_SSL false so httpfs traffic flows through the logging cache proxy. But delta_scan's delta-kernel object store ignores DuckDB's proxy configuration and dials the plain-HTTP S3 endpoint directly — and direct port-80 egress is silently dropped by the duckgres worker network policy.

Verified in mw-prod-us CP logs (2026-07-22): every duckgres-side delta_scan in the data-imports and data-modeling copy workflows fails with DeltaKernel ObjectStoreError … GET http://s3.us-east-1.amazonaws.com/... after 10 retries at ~58s per attempt, continuously re-attempted across many teams (team 2, devex, 19618, 35787, …) — 29 failures/15min, ~2 worker-minutes burned per minute, zero successes, each attempt pinning a dedicated worker under duckgres's one-session-per-worker model.

Change

create_staging_read_secret(conn, catalog_bucket) in posthog/ducklake/storage.py, called after setup_duckgres_session at the three duckgres delta_scan sites (imports create, imports verification, modeling create):

CREATE OR REPLACE SECRET posthog_staging_delta_https (
  TYPE S3, PROVIDER credential_chain, USE_SSL true,
  SCOPE 's3://<bucket>/__posthog_staging')

• The longer SCOPE wins DuckDB's prefix resolution for the staging tree and nowhere else — DuckLake data traffic keeps the proxy transport.
• credential_chain resolves the worker's own ambient (org-scoped) credentials and region; no secret material crosses the wire.
• Session-scoped by design: duckgres wipes non-persistent secrets at the next session create.
• Fixing it here rather than in duckgres keeps PostHog's staging-path convention out of duckgres, and avoids weakening the proxy default for anything but the staging prefix.

How did you test this code?

I'm an agent; checks actually run:

• TDD: extended test_copy_data_imports_to_ducklake_activity_via_duckgres to require the secret DDL (provider/scope/USE_SSL) before the first delta_scan; watched it fail, then implemented.
• test_ducklake_copy_data_imports_workflow.py -k duckgres: 4 passed; test_ducklake_copy_data_modeling_workflow.py -k duckgres: 1 passed.
• Pre-commit hooks (ruff, format, ty) green.
• Root cause verified end-to-end in production logs before writing any code (full trail in the linked investigation).

Rollback

Revert; behavior returns to today's failing-loop state (no data loss — the loop never succeeded).

🤖 Agent context

Authored by Claude Code (Fable 5) during the duckgres-sink throughput investigation; the failure signature (silent port-80 egress drop) matches the documented worker warmup air-gap incident.

🤖 Generated with Claude Code

[trunk-io]

😎 This pull request was merged.

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(ducklake): pin HTTPS for delta-kerne..." | Re-trigger Greptile}

[stamphog]

Small, contained fix that forces HTTPS via a narrowly-scoped, session-only S3 secret using ambient credential_chain (no hardcoded creds, strengthens rather than weakens transport security); covered by a TDD-added test, and corroborated by an active Greptile review plus positive reactions from Greptile and a security-focused bot with no unresolved concerns.

• Author wrote 0% of the modified lines and has 30 merged PRs in these paths (familiarity MODERATE).
• 👍 on the PR from greptile-apps[bot], hex-security-app[bot].

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	27L, 3F substantive, 41L/4F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1b-small (41L, 4F, single-area, fix)
stamphog 2.0.0b3		.stamphog/policy.yml @ b954a44 · reviewed head cc3e43d

[EDsCODE]

CI triage: the failing Django tests – Temporal (3/14) shard also fails on master's own latest ci-backend runs (same job name, plus a Core shard) — inherited breakage, not this change. The duckgres-path tests this PR touches pass locally (imports + modeling, plus the full storage suite). Will re-run checks once master is green.

🤖 Generated with Claude Code

[github-actions]

🤖 CI report

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend snapshots — 11 updated (11 modified, 0 added, 0 deleted)

Query snapshots: Backend query snapshots updated

Changes: 11 snapshots (11 modified, 0 added, 0 deleted)

What this means:

• Query snapshots have been automatically updated to match current output
• These changes reflect modifications to database queries or schema

Next steps:

• Review the query changes to ensure they're intentional
• If unexpected, investigate what caused the query to change

Review snapshot changes →

⚠️ Backend coverage — 50.0% of changed backend lines covered — 1 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ██████████░░░░░░░░░░ 50.0% (1 / 2)

File	Patch	Uncovered changed lines
posthog/ducklake/storage.py	50.0%	695

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 29967943894 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
batch_exports	████████░░░░░░░░░░░░ 39.6%	8,412 / 21,250
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
warehouse_sources_queue	████████████░░░░░░░░ 59.2%	148 / 250
tasks	██████████████░░░░░░ 69.1%	29,366 / 42,488
data_tools	██████████████░░░░░░ 70.0%	63 / 90
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
signals	████████████████░░░░ 79.1%	20,386 / 25,788
cdp	████████████████░░░░ 80.7%	3,117 / 3,863
data_modeling	█████████████████░░░ 82.8%	5,529 / 6,675
wizard	█████████████████░░░ 84.8%	935 / 1,102
notebooks	█████████████████░░░ 85.3%	7,266 / 8,522
agent_platform	█████████████████░░░ 86.4%	3,807 / 4,405
actions	█████████████████░░░ 86.6%	717 / 828
cohorts	██████████████████░░ 87.8%	4,488 / 5,114
product_tours	██████████████████░░ 87.9%	1,303 / 1,482
exports	██████████████████░░ 88.4%	6,949 / 7,861
data_warehouse	██████████████████░░ 88.8%	11,867 / 13,358
business_knowledge	██████████████████░░ 89.0%	4,391 / 4,936
engineering_analytics	██████████████████░░ 89.2%	5,641 / 6,323
conversations	██████████████████░░ 89.3%	16,820 / 18,834
dashboards	██████████████████░░ 89.4%	5,983 / 6,693
visual_review	██████████████████░░ 89.4%	5,818 / 6,505
error_tracking	██████████████████░░ 89.7%	10,174 / 11,342
alerts	██████████████████░░ 89.9%	4,054 / 4,508
early_access_features	██████████████████░░ 90.1%	1,031 / 1,144
mcp_analytics	██████████████████░░ 90.1%	2,763 / 3,065
streamlit_apps	██████████████████░░ 90.4%	2,501 / 2,767
links	██████████████████░░ 90.6%	183 / 202
slack_app	██████████████████░░ 90.7%	9,025 / 9,948
marketing_analytics	██████████████████░░ 91.0%	11,792 / 12,964
stamphog	██████████████████░░ 91.1%	4,056 / 4,450
product_analytics	██████████████████░░ 91.3%	5,811 / 6,362
mcp_store	██████████████████░░ 91.9%	4,257 / 4,634
notifications	███████████████████░ 92.6%	1,011 / 1,092
ai_observability	███████████████████░ 92.7%	14,944 / 16,117
web_analytics	███████████████████░ 92.9%	14,119 / 15,206
surveys	███████████████████░ 93.0%	5,730 / 6,163
managed_migrations	███████████████████░ 93.1%	1,424 / 1,530
posthog_ai	███████████████████░ 93.2%	1,326 / 1,422
approvals	███████████████████░ 93.3%	3,437 / 3,682
reminders	███████████████████░ 93.4%	468 / 501
workflows	███████████████████░ 93.5%	6,337 / 6,775
legal_documents	███████████████████░ 94.1%	1,568 / 1,667
endpoints	███████████████████░ 94.1%	8,640 / 9,177
messaging	███████████████████░ 94.3%	2,682 / 2,845
tracing	███████████████████░ 94.5%	2,670 / 2,826
skills	███████████████████░ 94.5%	2,893 / 3,061
revenue_analytics	███████████████████░ 94.6%	3,651 / 3,858
review_hog	███████████████████░ 94.7%	6,814 / 7,199
logs	███████████████████░ 95.4%	9,935 / 10,416
experiments	███████████████████░ 95.7%	24,882 / 25,996
growth	███████████████████░ 96.1%	3,245 / 3,376
replay_vision	███████████████████░ 96.2%	14,973 / 15,572
annotations	███████████████████░ 96.2%	732 / 761
feature_flags	███████████████████░ 96.5%	17,110 / 17,734
user_interviews	███████████████████░ 96.5%	2,638 / 2,734
access_control	███████████████████░ 96.8%	851 / 879
warehouse_sources	███████████████████░ 97.1%	327,769 / 337,448
customer_analytics	███████████████████░ 97.3%	8,380 / 8,610
data_catalog	███████████████████░ 97.4%	2,367 / 2,429
analytics_platform	████████████████████ 98.0%	2,153 / 2,197
metrics	████████████████████ 98.2%	2,491 / 2,536
pulse	████████████████████ 98.4%	2,017 / 2,049
live_debugger	████████████████████ 99.2%	613 / 618
field_notes	████████████████████ 99.4%	158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

[stamphog]

Retaining stamphog approval — delta since last review classified as trivial_paths.

[stamphog]

Retaining stamphog approval — delta since last review classified as trivial_paths.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-23 00:52 UTC	Run
prod-us	✅ Deployed	2026-07-23 01:05 UTC	Run
prod-eu	✅ Deployed	2026-07-23 01:11 UTC	Run