Skip to content

[BUG] PPL returns HTTP 200 with silently partial results when shards fail (no _shards failure surfaced) #5687

Description

@ahkcs

Describe the bug

When shards fail (e.g. Workload Management rejects queries at the shard level, or circuit-breaker/timeout on some shards), OpenSearch returns HTTP 200 with a _shards block such as {total:1000, successful:12, failed:988}. PPL discards the _shards info and returns the partial result as if it were complete — no error, no warning. Users get silently incorrect answers during load spikes (e.g. a returned count of 50 might really be 5000). DSL surfaces the shard failures; PPL does not.

To Reproduce

Run a PPL aggregation on a large multi-shard index under enough load (or WLM CPU enforcement) that some shards are rejected. Compare the PPL count to the equivalent _search DSL response's _shards.failed. PPL reports only the partial count with no indication.

Expected behavior

When failedShards > 0, PPL should either (a) attach a warning naming the shard-failure counts (total/successful/failed), or (b) optionally fail, so the consumer knows the answer is partial.

Root cause

OpenSearchResponse (opensearch module) ingests the SearchResponse but reads only getHits() and getAggregations() — it discards getTotalShards(), getSuccessfulShards(), getFailedShards(), getShardFailures(). The request also does not set allowPartialSearchResults(false), so it inherits the cluster default (true). Identical on the V2 and Calcite scan paths — both funnel through OpenSearchResponse.

Proposed fix

Capture failedShards/successfulShards/totalShards off the SearchResponse in OpenSearchResponse and, when failedShards > 0, raise a structured warning through the response warning channel (the Warning{type,message,detail}QueryResponse.warningsSimpleJsonResponseFormatter path added for mapping-conflict partial results). Optionally add a per-request/cluster setting to hard-fail instead of warn (mirrors DSL allow_partial_search_results).

Metadata

Metadata

Assignees

No one assigned

    Labels

    PPLPiped processing languagebugSomething isn't working

    Type

    No type

    Projects

    Status
    Not Started

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions