spider2-dbt — harbor_view dbt/DuckDB parity with ade-bench - #14
Merged
kentwelcome merged 19 commits intoJun 18, 2026
Conversation
Port ade_bench preflight patterns into spider2_dbt/preflight.py: named Spider2WorkspacePreflightError, _read_duckdb_tables round-trip, family-agnostic dbt sources cross-check. No static family contracts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ock + RIDER build-context - AC-1: _ensure_dbt_deps_image_layer gated on dbt_project/packages.yml (the spider2 divergence from ade-bench's project/) - AC-2 image side: _ensure_workspace_preflight_image_layer copies the preflight script and RUNs it --workspace /app before CMD - AC-3: locking test asserts gold/expected/golden + solution deny-globs strip planted answer files from the materialized view - RIDER (Codex finding 2): _ensure_spider2_build_context_layer stages dbt_project/ (incl. source .duckdb) into the environment/ build context and COPYs it to /app BEFORE the preflight RUN. Proven at build-context level: a test parses the COPY src from the Dockerfile and asserts a real .duckdb is present under it, so --workspace /app cannot fail on a missing project. Layer order: COPY dbt_project -> /app; dbt deps; preflight; CMD. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h corrupts source fixtures ACs 1-3 + RIDER verified from a clean checkout (28 passed; build-context proof; /app contract pinned; generic materializer unchanged; 72-passed non-spider2 regression; pre-existing score.load error confirmed not a regression). Code review + independent repro found a Critical defect: the three Dockerfile helpers write through the link-mode symlinked Dockerfile and corrupt the committed source fixtures (and the user's source Dockerfile in production). Gate: REJECTED -> implementation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under view_mode="link" the reflected environment/Dockerfile is a symlink into the source tree; the three layer-injection helpers' write_text calls followed the link and corrupted the version-controlled fixtures. Guard all three with the existing unlink-then-write pattern (materialize.py:140-146) so the view owns a real file. Add a regression test proving link mode never mutates the source Dockerfile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g from clean checkout Re-verified the unlink-then-write guard in all three Dockerfile helpers from a fresh clone of HEAD 7f31b7b: source fixtures unchanged (git diff tests/fixtures empty), view Dockerfile a real file with injected layers, regression test proven load-bearing (fails without the guard). 29 passed; generic materializer/leakage unchanged; /app contract pinned for r5. Gate: PASSED -> done. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rce check Validation gate cycle-2 feedback (two valid Codex findings): Finding 1 (high): the /app/<db_name>.duckdb contract was prose-only. The injected build-time preflight RUN passed only --task-id/--workspace, so it glob-firsted and could validate the wrong DB under multi/stale-DB drift. Wire the importable resolve_spider2_db_name (profiles.yml path: -> single *.duckdb -> task-slug; fails closed on >1 *.duckdb with none pinned) into _ensure_workspace_preflight_image_layer and thread --db-name into the RUN. The resolver is exposed for the r5 verifier to reuse the SAME resolution. Finding 2 (medium): the source-table check ignored schema. Track required and observed relations as (schema, table) pairs (honor dbt source/table schema fields, default to source name; compare against information_schema table_schema + table_name) so a table in the wrong schema no longer satisfies a source. AC-1/2/3, the build-context rider, and the cycle-1 unlink-then-write fix stay green: uv run pytest -k spider2_dbt --ignore=...test_task_identity_scoring.py -> 38 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…check verified load-bearing Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… in link mode The preflight helper wrote environment/razorback_spider2_preflight.py with no is_symlink() guard. Under view_mode="link", a source task shipping a file with that exact name is reflected as a symlink into the source tree, so the write followed the link and corrupted the version-controlled source — the same symlink-write-through class as the cycle-1 Dockerfile fix. Add the unlink-then-write guard mirroring the Dockerfile/task.toml guards, plus a link-mode regression test that seeds the source file and asserts it is unchanged (proven red without the guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ght symlink guard) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ified load-bearing Independently re-verified commit c0d9da1: the preflight-helper write guard (is_symlink/unlink before write_text) is real and load-bearing — seeding a colliding source file and materializing in link mode leaves the source byte-for-byte unchanged; stripping the guard corrupts the source and fails the regression test. Full symlink-write-through class now closed (4 guards). No regression: spider2_dbt acceptance 39 passed, generic materializer/leakage unchanged, ade_bench 13 passed, /app contract pinned. Code review: ready to merge, zero blocking findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…otherwise The profiles.yml resolver iterated outputs.values() and returned the first DuckDB path:, ignoring the profile's target:. dbt selects outputs[target], so on a multi-output profile (dev/prod) the preflight could pin the wrong DB. Now returns outputs[target]'s DuckDB path; fails closed when several outputs exist but target is missing/unknown (unresolved dbt target) or the target output is non-DuckDB (target output not duckdb). Single-output (no target) and glob fallbacks preserved. Shared resolver -> correctness matters for the r5 verifier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dbt target) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d load-bearing Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kentwelcome
added a commit
that referenced
this pull request
Jun 18, 2026
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR brings the spider2-dbt Harbor task-view materialization into dbt+DuckDB parity with ade-bench by adding a build-time “fail closed” workspace preflight, ensuring dbt dependencies are installed when declared, and hardening link-mode view generation against symlink write-through that could corrupt source tasks.
Changes:
- Add
spider2-dbtworkspace preflight logic + CLI (preflight_spider2_workspace,resolve_spider2_db_name) to validate DuckDB presence/readability and dbtsources:requirements. - Enhance
spider2-dbtHarbor view materialization to (a) stagedbt_project/into the Docker build context andCOPYit to/app, (b) inject adbt depsbuild layer whenpackages.ymlexists, and (c) run the build-time preflight pinned to the resolved/app/<db_name>.duckdb. - Add comprehensive unit tests for preflight failure modes, resolver behavior (including dbt
target:selection), Dockerfile layer injection ordering, and link-mode symlink-corruption regressions; plus implementation/validation documentation updates.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_spider2_dbt_workspace_preflight.py | New unit tests covering DuckDB preflight pass/fail cases, schema-aware dbt sources: enforcement, CLI behavior, and db_name resolver fail-closed semantics. |
| tests/unit/test_spider2_dbt_harbor_view.py | New unit tests asserting Dockerfile layer injection (dbt deps + preflight + build-context COPY), correct --db-name pinning, deny-glob behavior, and link-mode symlink write-through protection. |
| src/razorback/benchmarks/spider2_dbt/preflight.py | New preflight implementation and shared db_name resolver honoring dbt outputs[target], used by both the Docker build preflight and external verifier consumers. |
| src/razorback/benchmarks/spider2_dbt/harbor_view.py | Extends spider2-dbt task view materialization to stage dbt_project into build context, inject dbt deps + preflight layers, and guard all writes against link-mode symlink corruption. |
| docs/razorback-implementation/validation/spider2-dbt-harbor-view-ade-parity.md | Adds detailed validation evidence and multi-cycle verification notes for the parity work (including regression reproduction and fixes). |
| docs/razorback-implementation/spider2-dbt-harbor-view-ade-parity.md | Updates the workflow/implementation doc with the final implementation + validation cycle outcomes and rationale. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
kentwelcome
deleted the
spacedock-ensign/spider2-dbt-harbor-view-ade-parity
branch
June 18, 2026 11:08
kentwelcome
added a commit
that referenced
this pull request
Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings spider2-dbt task views to dbt+DuckDB runtime parity with ade-bench, so the agent can build the project and the workspace is validated before runtime.
What changed
sources:.dbt_project/packages.ymlis present.dbt_project/into the build context and COPY it to/appbefore the build-time preflight RUN.target:profile (/app/<db_name>.duckdb), shared with the verifier; fail closed on ambiguity.Evidence
materialize.py/leakage.pybyte-for-byte unchanged; resolver importable for the verifier.Review guidance
Four
is_symlink()guards inharbor_view.pyclose the source-corruption class; the db_name resolver honorsoutputs[target].ny