Skip to content

fix: detect JSON columns in has() from the schema, not a name list - #193

Merged
richardwooding merged 1 commit into
mainfrom
fix/schema-based-json-detection
Sep 9, 2026
Merged

richardwooding merged 1 commit into
mainfrom
fix/schema-based-json-detection

Conversation

@richardwooding

Copy link
Copy Markdown
Contributor

Why

isDirectJSONFieldAccess and isJSONColumn recognised a column as JSON only when it was named one of:

[]string{"metadata", "properties", "content", "structure", "taxonomy", "analytics", "classification"}

Those names came from one application's schema — the introducing commit 54b8855 says "Supports expressions like: has(informationAsset.metadata.someField)" — and appear nowhere in docs/, README.md or the changelog.

They are the two survivors of #62 (#61, #59), which converted shouldUseJSONPath, hasJSONFieldInChain, isJSONArrayField, isJSONBField and getJSONArrayFunction to schema-based detection and added isFieldJSONB/isFieldArray to support them. These two were missed.

The bug

getJSONRootAndPath used the list as its only stopping condition when locating the JSON column boundary. A JSONB column named anything else never terminated the walk, so its name was swallowed into the path and the table alias was emitted as the JSON document:

-- has(record.payload.active), payload declared jsonb via WithSchemas
-- before: PostgreSQL rejects this — 'record' is the row, not a jsonb value
jsonb_extract_path_text(record, 'payload', 'active') IS NOT NULL
-- after
record.payload ? 'active'

Two existing tests covered exactly this and passed only because they asserted NotEmpty and t.Logf'd the SQL instead of comparing it (json_jsonb_coverage_test.go:49, json_escaping_test.go:94). Both now assert the generated SQL — that is the regression test.

docs/json-support.md:84-86 already documents the fixed behaviour (has(user.preferences.theme)user.preferences ? 'theme'); preferences isn't on the list, so the docs were describing something the code didn't do. No doc change needed — the fix makes them true.

The fix

Both helpers now use getTableAndFieldFromSelectChain + isFieldJSON — the same lookups shouldUseJSONPath already uses — and consult jsonVars. visitHasFunction was already calling the schema-driven isJSONBField on the line after isDirectJSONFieldAccess, so the right mechanism was one line away.

Deep paths are unchanged. In documents.content.metadata.corpus, metadata is still a path segment rather than a column, because a boundary now requires the operand to be a table identifier — exactly the guard isTableReference applied by hand. That function is now redundant and removed.

Also: isJSONBField learns that a variable declared via WithJSONVariables is itself a JSONB column, so has(tags.colour) produces tags ? 'colour' as docs/operators-reference.md:333 documents, rather than the JSON arrow form. The name-based implementation never consulted jsonVars at all.

Breaking

A column is no longer treated as JSON purely because of its name. Callers not passing WithSchemas or WithJSONVariables now get the ordinary column IS NOT NULL form; declaring the column restores the previous SQL. Callers already passing schemas are unaffected, and those with JSON columns outside the seven names get correct SQL for the first time.

Same class of change as the v3.7.0 removal of the name-based ::numeric heuristic, and flagged the same way in the changelog.

Tests

New json_has_detection_test.go: a jsonb column outside the old list, a json-not-jsonb column, a nested path below a non-listed column, a non-JSON column, the no-schema case (pins the intended break), and the WithJSONVariables case. Plus the two converted non-assertions, a non-listed column added to TestJSONColumnReferenceEdgeCases, and placeholder_style_test.go moved off metadata to a neutral name so it demonstrates schema-driven detection.

⚠️ Please rely on CI, not local results

Docker isn't available in my environment, so the 10 pg/ testcontainer tests fail locally here — identically on main, so it's environmental. But pg/provider_testcontainer_test.go holds the exact-SQL and row-count assertions for the nested has() paths this change touches, including TestJSONHasFieldExpressions. That job is the real gate for this PR. Everything else (go build, go vet, golangci-lint — 0 issues, go test -short) is clean.

Out of scope

Two name lists remain, gating numeric casts rather than JSON detection: isNumericJSONField (json.go:110) and isJSONObjectFieldAccess (json.go:180attr, item, element, obj, feature, review).

The second is worth knowing about because it interacts with this path: it keys off the CEL variable name, so has(obj.settings.theme) still renders its base as obj->>'settings' (text) rather than obj.settings, purely because the variable is called obj:

variable output
obj obj->>'settings' ? 'theme'
item item->>'settings' ? 'theme'
rec rec.settings ? 'theme'

I tried replacing it with the iterVars set that already tracks comprehension variables; that breaks three tests, so it needs its own investigation rather than widening this PR. I renamed the affected test's variable rather than encode the broken output as an expectation. Happy to open an issue.

🤖 Generated with Claude Code

isDirectJSONFieldAccess and isJSONColumn recognised a column as JSON only
when it was named one of metadata, properties, content, structure, taxonomy,
analytics or classification. Those names came from one application's schema
and were never documented. They are the two survivors of #62 (#61, #59),
which converted every other JSON detection path to use WithSchemas.

The list produced invalid SQL for any JSONB column named something else.
getJSONRootAndPath used it as the only stopping condition when locating the
JSON column boundary, so a column outside the set never terminated the walk:
its name was swallowed into the path and the table alias was emitted as the
JSON document.

  -- has(record.payload.active), payload declared jsonb via WithSchemas
  -- before: PostgreSQL rejects this, 'record' is the row not a jsonb value
  jsonb_extract_path_text(record, 'payload', 'active') IS NOT NULL
  -- after
  record.payload ? 'active'

Two tests covered exactly this and passed only because they asserted
NotEmpty and logged the SQL rather than comparing it. Both now assert the
generated SQL, which is the regression test for this fix.

Both helpers now use getTableAndFieldFromSelectChain + isFieldJSON, the same
lookups shouldUseJSONPath already uses, and consult jsonVars. Deep paths are
unchanged: in documents.content.metadata.corpus, metadata is still a path
segment rather than a column, because a boundary requires the operand to be a
table identifier — the guard isTableReference used to apply by hand, now
redundant and removed along with the function.

isJSONBField also learns that a variable declared through WithJSONVariables
is itself a JSONB column, so has(tags.colour) produces tags ? 'colour' as
docs/operators-reference.md already documents, rather than the JSON arrow
form. The name-based implementation never consulted jsonVars at all.

BREAKING: a column is no longer treated as JSON purely because of its name.
Callers not passing WithSchemas or WithJSONVariables now get the ordinary
column IS NOT NULL form; declaring the column restores the previous SQL.

Two name lists remain, gating numeric casts rather than JSON detection:
isNumericJSONField and isJSONObjectFieldAccess. Replacing the latter with the
iterVars set that already tracks comprehension variables breaks three tests,
so it needs its own change rather than widening this one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@richardwooding
richardwooding merged commit 61ee8c1 into main Sep 9, 2026
10 of 11 checks passed
@richardwooding
richardwooding deleted the fix/schema-based-json-detection branch September 9, 2026 16:36
richardwooding added a commit that referenced this pull request Sep 9, 2026
…mes (#194)

Two hardcoded name lists remained after #193, both in the comprehension
path. Removing them leaves none in the converter.

isJSONObjectFieldAccess treated field access on a comprehension variable as
JSON extraction whenever the variable was named attr, item, element, obj,
feature or review. Both directions were wrong. A jsonb[] column iterated as
"row" was treated as a composite and produced row.status; an array of a
composite type iterated as "item" produced item->>'name' for a real column.
The name leaked outside comprehensions too: with a schema declaring name as
text, item.name.size() rendered as LENGTH(item->>'name').

A comprehension variable is now bound along with whether its range yields
JSON, decided by the element type of the range, and a variable bound that way
is treated exactly like one declared through WithJSONVariables for as long as
the body is being written. That reuses the existing JSON path machinery
rather than duplicating it, which nested access needs: r.metadata.active over
a jsonb[] now routes through the path builder to r->'metadata'->>'active',
where the name-based branch extracted the first segment as text and then
dotted into it, producing r->>'metadata'.active.

The binding is scoped like iterVars, since comprehensions nest and an inner
one may reuse a name the outer bound to a different kind of collection.

isNumericJSONField carried nineteen field names — level, score, price,
rating, megapixels, vram, helpful and so on — deciding whether an extracted
JSON value was wrapped in ::numeric. visitCall already drives that from the
compared type, which is where v3.7.0 left it when it removed the matching
heuristic from visitIdent, so the list was redundant. (item->>'price')::numeric
> 10 is unchanged, now because 10 is numeric rather than because the field is
called price. With the branch gone, visitSelect loses its three-way switch.

Two tests pinned the old behaviour and are corrected: a filter over a jsonb[]
column expected r.metadata.active for a document, and the issue-85 test
expected item->>'name' for a text column.

BREAKING: field access on a comprehension variable follows the schema.
Iterating a composite array yields column access and iterating a jsonb range
yields extraction, whichever the variable is called.

Co-authored-by: Richard Wooding <richardwooding@Richards-Virtual-Machine.local>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant