Declare search cells over a closed derived schema

[EnRaiha]

Summary

A derived relation over a closed-schema source refused s.distance with 42703, while the same projection over an open-schema source ran. The response layer answers a search-shaped query with two synthetic cells the schema cannot name: distance (Float64) and _surrogate (the internal row id, resolved to the user primary key). The declaration follows the plan: SqlPlan::carries_search_cells() matches the three variants whose rows lower to the vector-hit shape (VectorSearch | SparseSearch | MultiVectorSearch), and infer_subquery_relation reads the cells from the subquery's own plan — the source the planner already uses, never a second AST reading of it.

Closes #305.

Behaviour changes

Path	Before (main)	After
s.distance over a closed-schema derived relation whose body routes to a search (ORDER BY vector_* and the operator forms <=> / <#> / <->, sparse_score, routing WHERE forms, WHERE multi_vector_search(...))	42703	resolves; every returned hit carries a numeric value
s._surrogate over the same shapes	42703	resolves to the integer id
Shapes the planner sends to scalar evaluation (a comparison wrapped around a call, a one-argument ORDER BY vector_distance(..), a body whose FROM is neither Scan nor Join)	42703	42703 — a declared cell is never NULL
Queries that previously ran	unchanged	unchanged

Root cause

Derived-relation inference (resolver/derived.rs) re-derived "is this a search?" from the AST. The planner answers the same question in where_search / order_by/triggers, and the two diverged in both directions: declarations for shapes that route no search (silent NULL cells), and no declaration for SqlPlan::MultiVectorSearch.

What changed

Rebased onto a7f597257; two commits.

Commit e9f969f60 — the fix: SqlPlan::carries_search_cells() matches the three variants whose rows lower to the vector-hit shape; infer_subquery_relation takes the subquery's plan (derived_from passes inner_plan, entry passes cte_plan, TableScope's derived arm plans non-lateral factors for the same input — a correlated LATERAL factor cannot be planned at scope time and routes no cells). The four AST heuristic functions and the third registry singleton are deleted.

Commit e0a51d035 — the tests: the filed case (closed schema plus a vector index, SEARCH … USING VECTOR(…) in the derived body), the pinned outcomes for the previously tolerant assertions, and guards for the four divergences (sparse_score in WHERE, one-argument ORDER BY vector_distance, a body off a derived relation, and WHERE multi_vector_search).

Regression proof

On main without the change, the reworked wire case fails 5 of 20 — including the filed SEARCH … USING VECTOR(…) case and the _surrogate projection. On the branch head all 20 pass.

Tested

• cargo nextest run -p nodedb-sql --lib — 892 pass (derived-relation and carries_search_cells unit tests included)
• cargo nextest run -p nodedb --test wire -E 'test(cases::sql_search_subquery_composition)' — 20 pass; on main with the same tests: 15 pass / 5 fail
• lateral, recursive CTE, insert-select, vector-index and hybrid modules — 52 pass
• cargo fmt --all -- --check, cargo clippy -p nodedb-sql --all-targets, nodedb-preflight.sh: clean

Notes

• synthetic_column declares the cells as nullable Unknown type: the declaration resolves the name; the response layer supplies the value.
• WHERE multi_vector_search(...) plans SqlPlan::MultiVectorSearch, which the lowering step does not support yet (42601, variant named). The cell declaration is plan-derived, so that refusal is the honest one; 42703 no longer appears for it.
• A body off a derived relation gives order_by/triggers no Scan/Join to read, so it routes no search; its projection refuses the cells.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[farhan-syah]

Request changes.

Checked statically against the branch ref (no checkout):

Claim	Result
cargo fmt --all -- --check clean	Confirmed on every changed file.
Comment hygiene	One broken comment (inline). No issue numbers or links.
"every trigger and every shape"	Not true. The resolver re-derives routing from the AST and diverges from the planner in both directions (inline, blocker).
Commit list	The body cites cdb2e71a1, 4eddcfeb5, 235b7a28b, 994372149. The branch holds bec78889, 15cea2e, ca7baec, eeb56bb. Update the body after the rebase.

Blockers are the layer finding, the broken comment, and the doc/code contradiction. The registry singleton and the tolerant tests are should-fix; both fall out of the layer change.

What landed correctly: the closed-schema wire case reproduces #305, and _surrogate is now declared beside distance.

[EnRaiha]

Addressed and pushed — branch head 2a51f66ce (layer move 995215c8c, then merged with origin/main).

• Wrong layer (blocker): the decision now comes from the plan. SqlPlan::carries_search_cells() matches VectorSearch | SparseSearch | MultiVectorSearch; infer_subquery_relation takes the subquery's plan — derived_from.rs passes inner_plan, entry.rs passes cte_plan, and TableScope's derived arm plans non-lateral factors so it gets the same input. The four AST heuristics are deleted. A correlated LATERAL factor cannot be planned at scope time, so it carries no plan and routes no cells (stated in the body).
• Broken comment: it sat inside the block the layer change deletes.
• Doc contradiction: resolved by the source of truth, not the comment. WHERE sparse routing is the planner's Ok(None) and now declares nothing.
• Third registry singleton: builtin_registry() deleted; its only caller was the heuristic.
• Tolerant tests: :459 pins a numeric cell with no empty escape; :489 and the no-index case pin one outcome each. The divergence rows are pinned as guards: sparse WHERE → 42703, one-argument ORDER BY vector_distance(..) → refusal, body off a derived relation → refusal, WHERE multi_vector_search(..) → declared (lowering refuses 42601, never 42703).
• Commit list: body updated (bec788896, 15cea2eec, ca7baec72, eeb56bb4e, 995215c8c; merge 2a51f66ce).

Red proof: on main with the reworked case, 5 of 20 fail — including the filed SEARCH … USING VECTOR(…) case and the _surrogate projection; branch head 20/20 pass. nodedb-sql unit tests 921 pass; lateral, recursive-CTE and insert-select modules 52/52; preflight and fmt clean.

[farhan-syah]

Rebase, and force with release. Rebuild the commits

[EnRaiha]

Rebased onto a7f597257 and the history is rebuilt: e9f969f60 (fix) then e0a51d035 (tests). Force-pushed with lease.

The rebuilt tree is identical to the previously audited 2a51f66ce (git diff 2a51f66ce e0a51d035 is empty), so the parity audit still covers the content. Preflight passes on the rebuilt head.