fix: detect JSON columns in has() from the schema, not a name list - #193
Merged
Merged
Conversation
isDirectJSONFieldAccess and isJSONColumn recognised a column as JSON only when it was named one of metadata, properties, content, structure, taxonomy, analytics or classification. Those names came from one application's schema and were never documented. They are the two survivors of #62 (#61, #59), which converted every other JSON detection path to use WithSchemas. The list produced invalid SQL for any JSONB column named something else. getJSONRootAndPath used it as the only stopping condition when locating the JSON column boundary, so a column outside the set never terminated the walk: its name was swallowed into the path and the table alias was emitted as the JSON document. -- has(record.payload.active), payload declared jsonb via WithSchemas -- before: PostgreSQL rejects this, 'record' is the row not a jsonb value jsonb_extract_path_text(record, 'payload', 'active') IS NOT NULL -- after record.payload ? 'active' Two tests covered exactly this and passed only because they asserted NotEmpty and logged the SQL rather than comparing it. Both now assert the generated SQL, which is the regression test for this fix. Both helpers now use getTableAndFieldFromSelectChain + isFieldJSON, the same lookups shouldUseJSONPath already uses, and consult jsonVars. Deep paths are unchanged: in documents.content.metadata.corpus, metadata is still a path segment rather than a column, because a boundary requires the operand to be a table identifier — the guard isTableReference used to apply by hand, now redundant and removed along with the function. isJSONBField also learns that a variable declared through WithJSONVariables is itself a JSONB column, so has(tags.colour) produces tags ? 'colour' as docs/operators-reference.md already documents, rather than the JSON arrow form. The name-based implementation never consulted jsonVars at all. BREAKING: a column is no longer treated as JSON purely because of its name. Callers not passing WithSchemas or WithJSONVariables now get the ordinary column IS NOT NULL form; declaring the column restores the previous SQL. Two name lists remain, gating numeric casts rather than JSON detection: isNumericJSONField and isJSONObjectFieldAccess. Replacing the latter with the iterVars set that already tracks comprehension variables breaks three tests, so it needs its own change rather than widening this one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
richardwooding
added a commit
that referenced
this pull request
Sep 9, 2026
…mes (#194) Two hardcoded name lists remained after #193, both in the comprehension path. Removing them leaves none in the converter. isJSONObjectFieldAccess treated field access on a comprehension variable as JSON extraction whenever the variable was named attr, item, element, obj, feature or review. Both directions were wrong. A jsonb[] column iterated as "row" was treated as a composite and produced row.status; an array of a composite type iterated as "item" produced item->>'name' for a real column. The name leaked outside comprehensions too: with a schema declaring name as text, item.name.size() rendered as LENGTH(item->>'name'). A comprehension variable is now bound along with whether its range yields JSON, decided by the element type of the range, and a variable bound that way is treated exactly like one declared through WithJSONVariables for as long as the body is being written. That reuses the existing JSON path machinery rather than duplicating it, which nested access needs: r.metadata.active over a jsonb[] now routes through the path builder to r->'metadata'->>'active', where the name-based branch extracted the first segment as text and then dotted into it, producing r->>'metadata'.active. The binding is scoped like iterVars, since comprehensions nest and an inner one may reuse a name the outer bound to a different kind of collection. isNumericJSONField carried nineteen field names — level, score, price, rating, megapixels, vram, helpful and so on — deciding whether an extracted JSON value was wrapped in ::numeric. visitCall already drives that from the compared type, which is where v3.7.0 left it when it removed the matching heuristic from visitIdent, so the list was redundant. (item->>'price')::numeric > 10 is unchanged, now because 10 is numeric rather than because the field is called price. With the branch gone, visitSelect loses its three-way switch. Two tests pinned the old behaviour and are corrected: a filter over a jsonb[] column expected r.metadata.active for a document, and the issue-85 test expected item->>'name' for a text column. BREAKING: field access on a comprehension variable follows the schema. Iterating a composite array yields column access and iterating a jsonb range yields extraction, whichever the variable is called. Co-authored-by: Richard Wooding <richardwooding@Richards-Virtual-Machine.local> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
isDirectJSONFieldAccessandisJSONColumnrecognised a column as JSON only when it was named one of:Those names came from one application's schema — the introducing commit
54b8855says "Supports expressions like:has(informationAsset.metadata.someField)" — and appear nowhere indocs/,README.mdor the changelog.They are the two survivors of #62 (#61, #59), which converted
shouldUseJSONPath,hasJSONFieldInChain,isJSONArrayField,isJSONBFieldandgetJSONArrayFunctionto schema-based detection and addedisFieldJSONB/isFieldArrayto support them. These two were missed.The bug
getJSONRootAndPathused the list as its only stopping condition when locating the JSON column boundary. A JSONB column named anything else never terminated the walk, so its name was swallowed into the path and the table alias was emitted as the JSON document:Two existing tests covered exactly this and passed only because they asserted
NotEmptyandt.Logf'd the SQL instead of comparing it (json_jsonb_coverage_test.go:49,json_escaping_test.go:94). Both now assert the generated SQL — that is the regression test.docs/json-support.md:84-86already documents the fixed behaviour (has(user.preferences.theme)→user.preferences ? 'theme');preferencesisn't on the list, so the docs were describing something the code didn't do. No doc change needed — the fix makes them true.The fix
Both helpers now use
getTableAndFieldFromSelectChain+isFieldJSON— the same lookupsshouldUseJSONPathalready uses — and consultjsonVars.visitHasFunctionwas already calling the schema-drivenisJSONBFieldon the line afterisDirectJSONFieldAccess, so the right mechanism was one line away.Deep paths are unchanged. In
documents.content.metadata.corpus,metadatais still a path segment rather than a column, because a boundary now requires the operand to be a table identifier — exactly the guardisTableReferenceapplied by hand. That function is now redundant and removed.Also:
isJSONBFieldlearns that a variable declared viaWithJSONVariablesis itself a JSONB column, sohas(tags.colour)producestags ? 'colour'asdocs/operators-reference.md:333documents, rather than the JSON arrow form. The name-based implementation never consultedjsonVarsat all.Breaking
A column is no longer treated as JSON purely because of its name. Callers not passing
WithSchemasorWithJSONVariablesnow get the ordinarycolumn IS NOT NULLform; declaring the column restores the previous SQL. Callers already passing schemas are unaffected, and those with JSON columns outside the seven names get correct SQL for the first time.Same class of change as the v3.7.0 removal of the name-based
::numericheuristic, and flagged the same way in the changelog.Tests
New
json_has_detection_test.go: a jsonb column outside the old list, ajson-not-jsonbcolumn, a nested path below a non-listed column, a non-JSON column, the no-schema case (pins the intended break), and theWithJSONVariablescase. Plus the two converted non-assertions, a non-listed column added toTestJSONColumnReferenceEdgeCases, andplaceholder_style_test.gomoved offmetadatato a neutral name so it demonstrates schema-driven detection.Docker isn't available in my environment, so the 10
pg/testcontainer tests fail locally here — identically onmain, so it's environmental. Butpg/provider_testcontainer_test.goholds the exact-SQL and row-count assertions for the nestedhas()paths this change touches, includingTestJSONHasFieldExpressions. That job is the real gate for this PR. Everything else (go build,go vet,golangci-lint— 0 issues,go test -short) is clean.Out of scope
Two name lists remain, gating numeric casts rather than JSON detection:
isNumericJSONField(json.go:110) andisJSONObjectFieldAccess(json.go:180—attr,item,element,obj,feature,review).The second is worth knowing about because it interacts with this path: it keys off the CEL variable name, so
has(obj.settings.theme)still renders its base asobj->>'settings'(text) rather thanobj.settings, purely because the variable is calledobj:objobj->>'settings' ? 'theme'❌itemitem->>'settings' ? 'theme'❌recrec.settings ? 'theme'✅I tried replacing it with the
iterVarsset that already tracks comprehension variables; that breaks three tests, so it needs its own investigation rather than widening this PR. I renamed the affected test's variable rather than encode the broken output as an expectation. Happy to open an issue.🤖 Generated with Claude Code