Skip to content

SOLR-18541: CombinedQueryComponent.mergeIds gives wrong numFound, approximateTotalHits and numFoundExact - #5117

Open
nick-boss-tech wants to merge 5 commits into
apache:mainfrom
nick-boss-tech:combinedquerycomponent-merge-totals
Open

nick-boss-tech wants to merge 5 commits into
apache:mainfrom
nick-boss-tech:combinedquerycomponent-merge-totals

Conversation

@nick-boss-tech

@nick-boss-tech nick-boss-tech commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖 (posted on behalf of Nick Shanin)

https://issues.apache.org/jira/browse/SOLR-18541

What happens today

Combined queries report totals that count shards the merge dropped, double-count shards with no approximateTotalHits header, and invert the exactness flag.

In CombinedQueryComponent.mergeIds, a shard's totals are added before the partial-results skip is tested (lines 342 to 343, lines 399 to 400). A skipped shard still adds its hits. With fabricated shard responses, the merged numFound was 7 where 3 documents were merged, and approximateTotalHits was 15 instead of 5. A shard with no approximateTotalHits header falls back to the running merged total, so two shards merge to 7 instead of 5. The exactness fold is inverted: all-exact merges report false, and all-inexact merges report true.

What this change does

The skip is tested before any total is added, the fallback uses the shard's own count, and numFoundExact is cleared when a shard is skipped or more than one sub-query contributes.

The skip now sits above the totals in the same loop (lines 389 to 467). A skipped shard adds nothing to numFound or approximateTotalHits, but it still clears numFoundExact, because it may hold unreported matches. With more than one contributing sub-query, numFoundExact is cleared even if every count is exact. numFound still keeps the largest sub-query count.

Proof

Nine tests in CombinedQueryComponentMergeTotalsTest pass at head 81a88e40dd42f90fe6c67b2aeb6f678c437d7c0d (verified 2026-10-16). Six of them fail on the base code.

  • Base (merge-base 597ed6774083, branch test file in place): 9 tests, 6 failures. skippedPartialShardApproximateTotalExcluded (approximateTotalHits 15, expected 5), fallbackUsesEachShardsOwnCount (7, expected 5), skippedPartialShardContributesNothing (numFound 7, expected 3), skippedPartialShardWithHitsClearsExactness (numFound 2, expected 0), and two single-sub-query exactness tests, where the base fold inverts numFoundExact.
  • Three tests pass on base only because of the inverted fold, so they are not fail-before proof.
  • Prior head a93f9f6a997e: exactly one failure, multipleContributingSubQueriesClearExactness (numFoundExact expected:<false> but was:<true>). It is the discriminating test.

A choice to check

This choice is answered on the ticket: numFoundExact is false when more than one sub-query contributes. The distinct-document count is a follow-up.

numFound keeps the largest sub-query count. The alternative, counting distinct matching documents, is a follow-up. On the ticket, the maintainer directed the minimum change: clear numFoundExact when more than one sub-query contributes.

Limits

The tests use fabricated shard responses. One defect from a real timeAllowed exit is not fixed here.

  • A real timeAllowed exit after the first sub-query leaves a shorter list. The merge then throws IndexOutOfBoundsException at the unchecked access (lines 338 to 340). This change leaves that access alone, and the run was not repeated at this head. A size check would fix it.
  • QueryAndResponseCombiner.simpleCombine takes the largest sub-query count as its total (Math.max). This change leaves it alone. The maintainer named it as having the same cause on main.

Changelog: changelog/unreleased/SOLR-18541-combined-query-merge-totals.yml

AI assistance

AI agents assisted with research, implementation, review, and drafting. Nick Shanin directed the work and takes responsibility for this contribution.

…reports

Test the partial skip before folding a shard's totals in, so a skipped
shard contributes nothing to its query's numFound, approximateTotalHits
or exactness. The approximateTotalHits fallback now adds the shard's own
count instead of the running merged total. The hitCountIsExact fold now
keeps the merged flag true only when every contributing shard reports
an exact count, matching QueryComponent.

Adds CombinedQueryComponentMergeTotalsTest covering the skip, the
fallback and the exactness fold, and pinning the current maximum
behavior across sub-queries.
…t, and pin the skip cases

A real timeAllowed exit sends no per-query entry for the sub-query it did not reach, so the skipped shard reports no count at all and the merged count leaves out an unknown number of hits. The hits-only condition could not see that shape. Two tests pin the shard with no entry and the shard with zero hits, and one test name now says the skipped total is excluded.
A shard that stops early may send no count for the sub-query it did not reach, or an empty result with zero hits and an exact flag. A rule that looks at reported hits cannot see either shape, so any skipped shard now makes the merged count a lower bound.
nick-boss-tech added a commit to nick-boss-tech/solr that referenced this pull request Oct 13, 2026
… gate jobs, two settling runs, draft fixes

Verdicts: apache#5116 SATISFIED; apache#5115, apache#5117 and apache#5118 NOT SATISFIED with blocking items named. Round 2 assignment posted.
nick-boss-tech added a commit to nick-boss-tech/solr that referenced this pull request Oct 15, 2026
@nick-boss-tech
nick-boss-tech marked this pull request as ready for review October 16, 2026 07:01
@ercsonusharma

Copy link
Copy Markdown
Contributor

Thanks @nick-boss-tech, these are good fixes. I'd like to come back to this once SOLR-18550 is merged and this is rebased on top, since both changes how numFound/numFoundExact are computed for combined queries. Some piece of code will likely need adjusting after that.

@punAhuja

Copy link
Copy Markdown
Contributor

@nick-boss-tech @ercsonusharma — following up on the sequencing: #5172 (SOLR-18550) is updated per review with the interim numFound semantics and Sonu's numFoundExact handling. Once it merges, the exactness-fold fix here should rebase cleanly on top — both PRs fixed the same inverted fold, so that line drops to one copy. Happy to help with any conflicts in that rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants