TraceQL: unsupported predicates are silently dropped and the query answers with unfiltered traces

[wu-sheng]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Backend (apache/skywalking)

What happened

The TraceQL (Grafana Tempo API) query endpoint parses a query, applies the parts it implements, and silently discards the rest. The response is 200 with a full page of traces that do not satisfy the query the caller sent.

This is worse than an error for the use it is put to. An operator narrowing a trace search with span.http.method!="GET" gets back a page of traces that are all GET, under a filter that says otherwise, with nothing on the wire indicating the predicate was dropped. Nobody goes looking for a bug in a result that looks like data.

Grafana Tempo answers an unsupported or unparsable query with 400 and a message. This implementation answers 200 with the unfiltered result.

Environment

• Public demo, https://demo.skywalking.apache.org:3200/skywalking (OAP 11.1.0-SNAPSHOT-289b4b3)
• traceQL module with the SkyWalking datasource enabled
• Every query below ran against one fixed window in the past, so all of them see the same traces

How to reproduce

END=$(( $(date +%s) - 900 )); START=$(( END - 900 ))
search() { curl -s -u admin:<pw> --get --data-urlencode "q=$1" \
  --data "start=$START&end=$END&limit=20" \
  https://demo.skywalking.apache.org:3200/skywalking/api/search \
  | jq '[.traces[].traceID] | length'; }

search '{resource.service.name="agent::songs"}'                               # baseline: 20
search '{resource.service.name="agent::songs" && span.http.method!="GET"}'    # 20 — the same 20

Results

agent::songs in that window returns 20 traces with no predicate. Each row below adds one predicate that should exclude some or all of them.

A. The predicate is dropped and the answer is the unfiltered page — every one of these returned the byte-identical set of 20 trace ids as the baseline:

Query	Returned
{… && span.http.method!="GET"}	20 — identical to baseline
{… || resource.service.name="nope"}	20 — identical
{… && span:duration>10s}	20 — identical
{… && span.http.status_code="599"}	20 — identical
{… && event.name="nope"}	20 — identical
{… && !(span.http.method="GET")}	20 — identical
{… && span.nonexistent.tag}	20 — identical
{… && duration=99999ms}	20 — identical
{… && status="STATUS_CODE_ERROR"}	20 — identical
{resource.instance="no-such-instance"}	20 (a full unfiltered page)
{name="no-such-endpoint"}	20 (a full unfiltered page)

The last two are a distinct case worth calling out: an instance id and an endpoint id are built from the service id, so SkyWalkingTraceQLApiHandler applies them only when the query also names a service (SkyWalkingTraceQLApiHandler.java#L170-L185). Without one the condition is dropped and every trace matches. With a service the same query behaves correctly:

{resource.service.name="agent::songs" && resource.instance="no-such-instance"}   → 0 traces  ✔
{resource.service.name="agent::songs" && name="no-such-endpoint"}                → 0 traces  ✔

B. The predicate fails closed — wrong, but not dangerous, since an empty list is read as "nothing matched":

Query	Returned
{… && span.http.method=~"G.*"} (regex)	0
{… && span.http.method="GET|POST"} (alternation)	0
{… && span.http.method=~"GET|POST"}	0
{resource.service.name="agent::songs|agent::ui"}	0
{…} && {span.http.method="NOPE"} (second spanset)	0

The alternation cases matter for Grafana specifically: its TraceQL query builder emits an alternation for a multi-select value, so a perfectly ordinary Grafana panel silently returns nothing.

Where it comes from

TraceQLQueryVisitor maps the constructs it supports and takes no action on the others, rather than refusing them. Each branch is shaped like this (TraceQLQueryVisitor.java#L48-L112):

case "resource.instance":
    if ("=".equals(operator)) {
        params.setServiceInstance(value);
    }
    break;          // any other operator: parsed, then forgotten

duration handles only >, >=, <, <=, so duration= falls through both branches; status accepts any value and the handler reads only ok / error (SkyWalkingTraceQLApiHandler.java#L211-L218); unknown scopes such as event. / link. fall into the default branch and become ordinary tags that match nothing in particular.

The Zipkin datasource (/zipkin) shares the visitor and behaves the same way.

What I'd expect

In order of preference:

1. Refuse the query with 400, naming the construct that is not supported — this is what Tempo does, and it is the only answer that cannot be mistaken for data. A client can then show the operator why.
2. Failing that, return an empty result for a query carrying a predicate that was not applied. Empty is read as "nothing matched"; a full page is read as "these matched".

What should not happen in any case is answering a query as though the unsupported predicate were absent.

A smaller, separate fix that would help clients a great deal: the search response's spans carry no status and no error marker, so a trace list cannot show which traces failed without reading every trace. SearchResponse projects only the tags in skywalkingTracesListResultTags, and a SkyWalking span's failure is a field rather than a tag — including it (or the OTLP span status) in the search result would let a list render error state directly.

Anything else

Apache SkyWalking Horizon UI hit all of these while building its TraceQL tab; it currently refuses to send the affected expressions and explains why in the editor, because a confident wrong list of traces during an incident is the worst outcome available. That guard belongs upstream — every client of this API needs it.

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

[wu-sheng]

One more case of the same class, found while wiring a UI against this API — and this one is easy to hit by accident, because the query looks completely ordinary.

An attribute written without a scope is dropped, and the search answers with every trace.

TraceQL spells an unscoped attribute with a leading dot (.http.method) and a scoped one with span. / resource.. Written bare, the condition parses and then does nothing:

END=$(( $(date +%s) - 900 )); START=$(( END - 900 ))
search() { curl -s -u admin:<pw> --get --data-urlencode "q=$1" \
  --data "start=$START&end=$END&limit=20" \
  https://demo.skywalking.apache.org:3200/skywalking/api/search | jq '.traces | length'; }

query	returned
{}	20 (baseline)
{span.http.method="NOPE"}	0 ✔
{.http.method="NOPE"}	0 ✔
{http.method="NOPE"}	20 — the unfiltered page

The same holds for the service, which is the form an operator is most likely to type:

query	returned
{resource.service.name="agent::songs"}	20, all involving agent::songs ✔
{.service.name="agent::songs"}	20, all involving agent::songs ✔
{service.name="agent::songs"}	20, including traces of other services
{service.name="skywalking"}	20 — and no service by that name exists on the demo

TraceQLQueryVisitor has a case "service.name" branch, so the intent is clearly that the bare spelling works; whatever the parser produces for an unscoped-without-dot attribute never reaches it. Either way the outcome is the 200-with-everything behaviour this issue is about: the last row above is the clearest demonstration — a filter naming a service that does not exist returns a full page of traces.

Same request as in the issue body: refuse it with 400 naming the construct, or return an empty result — but not the unfiltered page.

Tested on the public demo, OAP 11.1.0-SNAPSHOT-289b4b3.