Describe the bug
On the Calcite path (plugins.calcite.enabled=true), dedup on a dotted field that maps into an OpenSearch object hierarchy fails 100% of the time with a ClassCastException. The same field projected with | fields works, so the failure is specific to dedup (and any command that turns the field into a partition/group key).
To Reproduce
Index with an object hierarchy (e.g. OpenTelemetry logs):
PUT logs-otel/_mapping
{ "properties": { "resource": { "properties": { "attributes": { "properties":
{ "k8s": { "properties": { "namespace": { "properties": { "name": { "type": "keyword" }}}}}}}}}}
Query:
source=logs-otel
| dedup resource.attributes.k8s.namespace.name
| fields resource.attributes.k8s.namespace.name, severityText, body
| head 50
Result: java.sql.SQLException: exception while executing query: class java.lang.String cannot be cast to class java.util.Map.
Expected behavior
Dedup should deduplicate on the leaf value, exactly as | fields resource.attributes.k8s.namespace.name resolves it. No cast error.
Root cause
OpenSearchIndex.getFieldTypes() flattens the mapping and emits a column for every intermediate object node and leaf (OpenSearchDataType.traverseAndFlatten). Each object/STRUCT node becomes a Calcite MAP<VARCHAR,ANY> (OpenSearchTypeFactory ~L186-190). When the exact dotted leaf is not itself a distinct mapped column, QualifiedNameResolver matches the longest existing prefix (a MAP column) and wraps the remaining segments in an INTERNAL_ITEM/ITEM call — so the dedup key becomes a MAP-typed RexCall, not a scalar RexInputRef.
Two throw sites, both (Map<String,Object>) value on a scalar String:
- Agg-pushdown decode:
ObjectContent.map() L81 (dedup w/o keepempty is rewritten to composite-agg + top_hits by DedupPushdownRule), reached via OpenSearchExprValueFactory.parse STRUCT branch + OpenSearchResponse.handleAggregationResponse.
- JDBC decode:
OpenSearchTypeFactory.getExprValueByExprType STRUCT branch L347, via OpenSearchExecutionEngine.buildResultSet/processValue.
DedupPushdownRule's guards miss it: it only bails on bare-RexInputRef MAP/ARRAY, and the nested-path guard (Utils.resolveNestedPath) fires only for nested→ARRAY, never plain object→STRUCT. | fields avoids it by resolving the leaf scalar (ExprValueUtils.resolveRefPaths) without re-typing the parent object.
Plugins/paths affected
Calcite path only. V2/legacy is unaffected (no dedup pushdown; DedupeOperator resolves the leaf scalar in memory).
Proposed fix
QualifiedNameResolver: don't let a dedup/partition key silently become a MAP-typed ITEM when a scalar leaf is intended.
DedupPushdownRule: extend the guard to bail on MAP (STRUCT) / INTERNAL_ITEM keys, not just bare-RexInputRef MAP/ARRAY.
- Defense-in-depth: guard the two
Map casts to degrade a STRUCT-vs-scalar mismatch gracefully.
- Add a
CalcitePPLDedupIT case with an object hierarchy (+ CalciteNoPushdownIT). No current dedup test uses a dotted/object field.
Describe the bug
On the Calcite path (
plugins.calcite.enabled=true),dedupon a dotted field that maps into an OpenSearchobjecthierarchy fails 100% of the time with aClassCastException. The same field projected with| fieldsworks, so the failure is specific todedup(and any command that turns the field into a partition/group key).To Reproduce
Index with an
objecthierarchy (e.g. OpenTelemetry logs):Query:
Result:
java.sql.SQLException: exception while executing query: class java.lang.String cannot be cast to class java.util.Map.Expected behavior
Dedup should deduplicate on the leaf value, exactly as
| fields resource.attributes.k8s.namespace.nameresolves it. No cast error.Root cause
OpenSearchIndex.getFieldTypes()flattens the mapping and emits a column for every intermediateobjectnode and leaf (OpenSearchDataType.traverseAndFlatten). Eachobject/STRUCT node becomes a CalciteMAP<VARCHAR,ANY>(OpenSearchTypeFactory~L186-190). When the exact dotted leaf is not itself a distinct mapped column,QualifiedNameResolvermatches the longest existing prefix (a MAP column) and wraps the remaining segments in anINTERNAL_ITEM/ITEMcall — so the dedup key becomes a MAP-typedRexCall, not a scalarRexInputRef.Two throw sites, both
(Map<String,Object>) valueon a scalar String:ObjectContent.map()L81 (dedup w/okeepemptyis rewritten to composite-agg + top_hits byDedupPushdownRule), reached viaOpenSearchExprValueFactory.parseSTRUCT branch +OpenSearchResponse.handleAggregationResponse.OpenSearchTypeFactory.getExprValueByExprTypeSTRUCT branch L347, viaOpenSearchExecutionEngine.buildResultSet/processValue.DedupPushdownRule's guards miss it: it only bails on bare-RexInputRefMAP/ARRAY, and the nested-path guard (Utils.resolveNestedPath) fires only fornested→ARRAY, never plainobject→STRUCT.| fieldsavoids it by resolving the leaf scalar (ExprValueUtils.resolveRefPaths) without re-typing the parent object.Plugins/paths affected
Calcite path only. V2/legacy is unaffected (no dedup pushdown;
DedupeOperatorresolves the leaf scalar in memory).Proposed fix
QualifiedNameResolver: don't let a dedup/partition key silently become a MAP-typedITEMwhen a scalar leaf is intended.DedupPushdownRule: extend the guard to bail on MAP (STRUCT) /INTERNAL_ITEMkeys, not just bare-RexInputRefMAP/ARRAY.Mapcasts to degrade a STRUCT-vs-scalar mismatch gracefully.CalcitePPLDedupITcase with anobjecthierarchy (+CalciteNoPushdownIT). No current dedup test uses a dotted/object field.