Removing Class.forName for implementation bridge helpers by FabianMeiswinkel · Pull Request #3 · xinlian12/azure-sdk-for-java

FabianMeiswinkel · 2022-05-16T22:48:42Z

No description provided.

Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>

…nd CosmosException (Azure#28620) * Improved diagnostics with new models for StoreResponse, StoreResult and CosmosException * Fixed spot bugs related to storeResult.getException() * Updated query plan cache to ConcurrentHashMap with fixed size of 1000 to start with * Added exception response headers and message to direct and gateway errors. Also added code for throwing any java.lang.Error * Added unit tests for StoreReader and ConsistencyWriter * Disabled StoreReader unit test for error since it is causing other tests to fail. Will investigate later * Commented out the broken test * Reverted StoreReaderTest * Removed mockito-inline * Fixed StoreReaderTest static mocking * Fixed ConsistencyWriterTest static mocking * Code review comments and changelog addition * Fixed some test cases

xinlian12 · 2022-05-17T05:13:40Z

        public static void setCosmosClientBuilderAccessor(final CosmosClientBuilderAccessor newAccessor) {
-            if (accessor != null) {
-                throw new IllegalStateException("CosmosClientBuilder accessor already initialized!");
+            if (!accessor.compareAndSet(null, newAccessor)) {


should we call doNothingButEnsureLoadingClass() in setter as well?

No - because setXXX will always be called from the class that we want to ensure to be loaded.

oh yea make sense

xinlian12 · 2022-05-17T05:19:50Z

Change log update?

FabianMeiswinkel · 2022-05-17T08:42:50Z

Change log update?

Added - although this PR is just to merge into your PR :-)

xinlian12

LGTM, thanks

… multiple times (Azure#28905) * ThroughputControl fix * add change log Co-authored-by: annie-mac <annie-mac@yindeng2019.fareast.corp.microsoft.com>

…to users/fabianm/SetAccessorsFix

…bianMeiswinkel/azure-sdk-for-java into users/fabianm/SetAccessorsFix

* health insights sdk for java * rename country to countryOrRegion * update product name to azure-health-insights * update cspell with missing words * update root pom.xml and version_client.txt * update dependency versions * update dependency versions (#2) * skip jacoco * fix spell checks * Asaflevi/feature/healthinsights sdk for java (#3) * update dependency versions * skip jacoco * fix spell checks * skip jacoco * linting * remove redundant plugins * remove test.yaml files * remove redundant package.json file * fix sampes and readmes * spell * fix linting * fix sample repository reference * remove duplicate dependency (mockito) * alignment (readme) * fix readme title * setPlaybackSyncPollerPollInterval * update serviceversion class (emitter 0.5.1) * remove impressions png

…BrokenConnections TestPR - NO REVIEW

…property of AKV SecretClient (Azure#36603) * Allows configuration of disableChallengeResourceVerification property - Adds disableChallengeResourceVerification property to properties objects - Includes new property in mapping methods - Configures SecretClient in Factory when disableChallengeResourceVerification is set - Configures CertificateClient in Factory when disableChallengeResourceVerification is set - Updates/adds new tests - Updates Changelog Resolves Azure#36561 Signed-off-by: Esta Nagy <nagyesta@gmail.com> * Allows configuration of disableChallengeResourceVerification property - Code review fixes #1 - Renames disableChallengeResourceVerification to challengeResourceVerificationEnabled - Adds additional JavaDoc Resolves Azure#36561 Signed-off-by: Esta Nagy <nagyesta@gmail.com> * Allows configuration of disableChallengeResourceVerification property - Fix a missed JavaDoc Signed-off-by: Esta Nagy <nagyesta@gmail.com> * Improve the configuration properties javadoc, and complete the additional-spring-configuration-metadata.json * Allows configuration of disableChallengeResourceVerification property - Code review fixes #3 - Simplifies factory method logic as per code review recommendation Resolves Azure#36561 Signed-off-by: Esta Nagy <nagyesta@gmail.com> --------- Signed-off-by: Esta Nagy <nagyesta@gmail.com> Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com> Co-authored-by: Xiaolu Dai <xiada@microsoft.com>

@xinlian12

…meout (Azure#49276) * [Cosmos Spark] Surface empty change feed pages to avoid end-to-end timeout Spark partition tasks reading change feed (or executing cross-partition queries) against a sparse workload could hit OperationCancelledException ("End-to-end timeout hit when trying to retrieve the next page") at the connector's 65-second per-operation end-to-end timeout. Root cause: with the default emptyPagesAllowed=false, ParallelDocumentQueryExecutionContext and ChangeFeedFetcher swallow empty / 304 pages internally — a single producer-side nextPage() call can keep draining many sub-feedRanges before emitting one non-empty page. For sparse workloads the cumulative time blows the per-operation timeout. Fix: * Spark ItemsPartitionReader (query path) calls setAllowEmptyPages(true) on the CosmosQueryRequestOptions so the SDK's existing emptyPagesAllowed plumbing applies. * New internal-only emptyPagesAllowed flag on CosmosChangeFeedRequestOptionsImpl (default false; behavior unchanged for all other callers) plumbed through Paginator. getChangeFeedQueryResultAsObservable into ChangeFeedFetcher. nextPageInternal. When the flag is true, both 304 branches return Mono.just(r) so empty pages bubble up to the iterator. Surfaced via new package-private bridge accessor CosmosChangeFeedRequestOptionsAccessor.{get,set}AllowEmptyPages. * ChangeFeedFetcher.isFullyDrained no longer short-circuits to true on noChanges responses (it now consults only continuation.isDone()), which removes the load-bearing reEnableShouldFetchMoreForRetry() pattern that was previously needed to undo a base-class decision. * Spark ChangeFeedPartitionReader opts into the new flag via the bridge accessor. * CosmosChangeFeedRequestOptions.withCosmosPagedFluxOptions now also propagates emptyPagesAllowed when the paged-flux pull mechanism supplies a continuation token (the freshly-built impl would otherwise silently lose the flag — comment added flagging the broader drift hazard). Tests: * New ChangeFeedFetcherEmptyPagesTest (5 unit tests): exercises the isFullyDrained behavior change and asserts that nextPageInternal surfaces noChanges responses individually when the flag is true and swallows them via repeatWhenEmpty when the flag is false. * New CosmosChangeFeedRequestOptionsWithPagedFluxOptionsTest (3 unit tests): locks in the flag propagation through withCosmosPagedFluxOptions. * Extended TransientIOErrorsRetryingIteratorSpec with a regression test that drains hundreds of leading empty pages followed by data without hitting the end-to-end timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add PR link to CHANGELOG entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore noChanges short-circuit in isFullyDrained when emptyPagesAllowed=false The previous cleanup of ChangeFeedFetcher.isFullyDrained removed the noChanges short-circuit unconditionally, on the rationale that consulting continuation.isDone() was simpler. That regressed every non-Spark caller of the change-feed API: FeedRangeCompositeContinuationImpl.isDone() returns compositeContinuationTokens.size() == 0, but moveToNextToken() rotates the deque via poll() + add() and never shrinks it. So isDone() is permanently false for normal incremental change-feed iteration. For the default emptyPagesAllowed=false path: 1. The 304 arrives. 2. updateState calls isFullyDrained -> false (because isDone() is false). 3. nextPageInternal's else-branch sees handleChangeFeedNotModified return NO_RETRY (single-partition case, multi-partition cycle-complete, or the >4*(size+1) consecutive-304 defense) and falls through to Mono.just(r). 4. Paginator's generate-loop checks shouldFetchMore() -> true and calls nextPage() again -> infinite poll loop. Customer-visible impact would be: any consumer that drains queryChangeFeed(...).byPage() to completion (e.g. .toIterable().iterator(), .collectList(), .blockLast()) hangs forever once the change feed catches up. flag is true (Spark path), surface every noChanges to the caller and let the consumer decide when to stop iterating. When the flag is false (every other caller, including the SDK's public queryChangeFeed API), preserve the original termination signal. Also addressed reviewer feedback: * Drop unused org.mockito.Mockito import (would break checkstyle's UnusedImports rule). * Replace reflective field assignment in stubRequest() with direct field writes on the public fields RxDocumentServiceRequest.requestContext and .faultInjectionRequestContext. Mockito only intercepts method calls; field writes on a mock work directly. * Add a regression test isFullyDrained_noChangesResponseWithEmptyPagesAllowedFalse_returnsTrue that locks in the termination signal for the default path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address pass-3 review: defense-in-depth on NO_RETRY + DRY cleanup * Defense-in-depth (F1): when emptyPagesAllowed=true the streaming change- feed path takes branch 2 of nextPageInternal. If handleChangeFeedNotModified returns NO_RETRY on a noChanges page (single- partition case, multi-partition full-cycle complete, or the >4*(size+1) consecutive-304 defense in FeedRangeCompositeContinuationImpl), the SDK's built-in termination signal was being silently dropped because isFullyDrained() consults only continuation.isDone() in that mode (which is permanently false for incremental change feed). Now we explicitly call disableShouldFetchMore() to preserve the defense-in-depth termination guarantee even for emptyPagesAllowed=true callers. * DRY (F4): extracted the two near-identical 'surface or swallow via repeatWhenEmpty' blocks in nextPageInternal into a private surfaceOrSwallowNoChangesPage(r) helper. The branches now read as a one-line intent instead of seven near-identical lines. * Comment density (F9): tightened the long isFullyDrained comment to a 2-line tl;dr followed by the detailed rationale. * Test (F2): new nextPage_emptyPagesAllowedTrueWithNoRetryOnNoChanges_ terminatesIteration locks in the defense-in-depth fix - asserts that after a terminal NO_RETRY noChanges page, shouldFetchMore() flips to false so Paginator stops calling nextPage(). * Test (F3): added Mockito.verify(continuation, never()).isDone() to the pass-2 regression test so a future refactor that accidentally drops the noChanges short-circuit and falls through to the (permanently-false) continuation.isDone() check fails loudly instead of silently hanging. Test results: ChangeFeedFetcherEmptyPagesTest 7/7, FetcherTest 5/5, CosmosChangeFeedRequestOptionsWithPagedFluxOptionsTest 3/3, TransientIOErrorsRetryingIteratorSpec 7/7 - all green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address pass-4 review: lock contract on data-page non-termination * Test (F1): assert callIndex==4 after NO_RETRY termination in the pass-3 defense-in-depth test so a future regression that terminates iteration but still over-fetches is caught. * Test (F2): new nextPage_emptyPagesAllowedTrueWithDataPages_doesNotTerminate pins the production contract that the noChanges(r) guard on the disableShouldFetchMore() arm is load-bearing. In production, FeedRangeCompositeContinuationImpl.handleChangeFeedNotModified returns NO_RETRY for EVERY non-noChanges response (the early branch resets state and falls through). Without the noChanges(r) guard, every data page would silently truncate iteration after the first emission. * Comment (F5): added an inline rationale next to the noChanges(r) guard explaining why it must remain - prevents a future engineer from 'simplifying' away the guard without realizing the production truncation hazard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Shrink bridge accessor surface; reclassify CHANGELOG entry * Drop the new CosmosChangeFeedRequestOptionsAccessor.{get,set}AllowEmptyPages wrapper methods. Callers (ChangeFeedQueryImpl, Spark ChangeFeedPartitionReader, the propagation unit test) now use the already-exposed accessor.getImpl(options).{is,set}EmptyPagesAllowed() instead, keeping the bridge accessor interface at its pre-PR shape. * Move the azure-cosmos CHANGELOG entry from 'Bugs Fixed' to 'Other Changes' and reword: this PR adds an internal-only field on CosmosChangeFeedRequestOptionsImpl that pure SDK consumers cannot reach without going through getImpl(). The customer-facing fix lives in the Spark connector CHANGELOGs (which keep their 'Bugs Fixed' entries). Test results: ChangeFeedFetcherEmptyPagesTest 8/8, FetcherTest 5/5, CosmosChangeFeedRequestOptionsWithPagedFluxOptionsTest 3/3, TransientIOErrorsRetryingIteratorSpec 7/7 - all green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address pass-5 review (Kushagra + Annie): drift hazard, test coverage, polish Major (Kushagra): * M1 — Update PR description to reflect the post-commit-34d3da4 accessor shape (getImpl() escape hatch was being misrepresented as a named accessor wrapper). * M2 — Invert the default in CosmosChangeFeedRequestOptions .withCosmosPagedFluxOptions: instead of building a fresh impl from the continuation token and copying back 4 fields, build from the token and inherit ALL non-token-encoded fields via a new CosmosChangeFeedRequestOptionsImpl.inheritNonContinuationFieldsFrom helper. This closes silent drift for endLSN, customSerializer, excludeRegions, readConsistencyStrategy, thresholds, customOptions, operationContextAndListenerTuple, keywordIdentifiers, completeAfterAllCurrentChangesRetrieved, quotaInfoEnabled, isSplitHandlingDisabled, partitionKeyDefinition, and collectionRid. The 4 token-encoded fields (continuationState, feedRangeInternal, mode, startFromInternal) remain authoritative from the parsed token. Single maintenance point for any future field. * M3 — Add timeOut = 10_000 to the 4 nextPage_* tests in ChangeFeedFetcherEmptyPagesTest so a regression that reintroduces unbounded repeatWhenEmpty drain fails fast instead of looking like CI flake (.NET parity). Minor (Kushagra + Annie): * m1+m3+m7 — Combined javadoc on CosmosChangeFeedRequestOptionsImpl .setEmptyPagesAllowed/isEmptyPagesAllowed: explains default, paging semantics impact, and the deliberate 'not surfaced on public API' decision. Replaces the PR-body claim with an in-code source of truth. * m2 = Annie #2 — Re-add the bridge accessor wrappers setAllowEmptyPages/getAllowEmptyPages on CosmosChangeFeedRequestOptionsAccessor mirroring the query-side pattern. Restores grep-discoverability and reduces refactor blast radius. The public CosmosChangeFeedRequestOptions API is unchanged (no public setter); friend-API surface only. * m4 = Annie #1 — Add 2 nextPage_endLsnSet_emptyPagesAllowed_* tests exercising branch 1 of nextPageInternal (the completeAfterAllCurrentChangesRetrieved || endLSN != null path), which is the production path Spark's ChangeFeedPartitionReader hits for bounded snapshot reads. * m5 — Add emulator-group end-to-end test CosmosContainerChangeFeedTest.changeFeedQuery_emptyPagesAllowed_ surfacesNoChangesPagesAndTerminates exercising the real FeedRangeCompositeContinuationImpl >4*(size+1) consecutive-304 defense path that mock-based unit tests can't reach (the impl class is package-private by design). * m6 — Shorten the verbose Spark + azure-cosmos CHANGELOG entries and append a brief 'one iterator callback per empty page' trade-off note for operator observability. * Annie #3 = M2 — broader drift hazard closed via the inheritNonContinuationFieldsFrom approach above. Nits: * n1 — Extract the nextPageInternal flatMap body into a private applyNoChangesDecision(FeedResponse<T>) method. Reduces nesting depth from 4 to 2 and improves readability of the contract. * n3 — Replace reflective Fetcher.isFullyDrained invocation with a direct call. The test lives in com.azure.cosmos.implementation.query, same package as Fetcher, so protected access works without reflection. Tests: * ChangeFeedFetcherEmptyPagesTest: 10 tests (was 8), all green * CosmosChangeFeedRequestOptionsWithPagedFluxOptionsTest: 6 tests (was 3 — adds endLSN, customSerializer, negative pin), all green * FetcherTest: 5/5 (no regression) * TransientIOErrorsRetryingIteratorSpec: 7/7 * CosmosContainerChangeFeedTest.changeFeedQuery_emptyPagesAllowed_*: new emulator-group test (compiles; runs in CI Test Emulator lane) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * De-flake CosmosContainerChangeFeedTest.asyncChangeFeedPrefetching The test failed intermittently on macOS / slow CI runners with 'count > 2' failing at count == 2. Root causes: * .subscribe() is fire-and-forget; Thread.sleep(3000) was the only synchronization, hoping 3s was enough for async pages to arrive * Race between the two subscriptions in FULL_FIDELITY mode: line 367 read continuation.get() synchronously without waiting for the first subscription to populate it (could feed '' to createForProcessingFromContinuation) * Slow runners need more than 3s to receive 3 pages of 100 docs at maxItemCount=10 Fix: * Replace each .subscribe() + Thread.sleep(3000) pattern with a CountDownLatch(N) for 'N pages received' + a generous 30s timeout. Deterministic ordering, no fixed sleep. * For the bounded .take(2, true) block, switch from fire-and-forget .subscribe() to .blockLast(Duration.ofSeconds(30)) so the test waits for the pipeline to complete after exactly 2 pages. * Dispose subscriptions in finally blocks to avoid leaking pages between test iterations. Test intent preserved: count > 2 on the first/resume subscriptions, count == 2 on the bounded take(2, true) subscription. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * change * add one more change * Fix asyncChangeFeedPrefetching FULL_FIDELITY: insert AFTER subscribe Previous de-flake (commit 5683b08) reordered the FULL_FIDELITY insert and the first subscription such that the insert ran BEFORE subscribing. But FULL_FIDELITY uses createForProcessingFromNow, so docs written before the subscription opens are invisible — the first subscription saw zero pages and the new CountDownLatch(3) timed out at 30s. Fix: branch by mode. INCREMENTAL keeps the pre-subscribe insert (since createForProcessingFromBeginning sees pre-existing docs). FULL_FIDELITY inserts AFTER each subscribe (first subscription, resume-from-continuation subscription, and the bounded take(2,true) subscription) so the from-now pipeline actually has writes to consume. Caught by: CosmosContainerChangeFeedTest.asyncChangeFeedPrefetching:385 [first change-feed subscription should produce at least 3 pages within 30 seconds] on Windows TCP Java8 + Java17 emulator runs, FULL_FIDELITY parameter only; INCREMENTAL passed on all platforms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address @xinlian12 review: rename changefeed flag, fix broken test, add spark 4.x changelogs - Renamed change-feed-side flag emptyPagesAllowed -> notModifiedPagesAllowed (and bridge accessor methods setAllowEmptyPages/getAllowEmptyPages -> setAllowNotModifiedPages/getAllowNotModifiedPages) so the name reflects what it actually controls: 304/NotModified pages from sub-partitions. Query-side CosmosQueryRequestOptions.setAllowEmptyPages stays unchanged. Test file renamed via git mv (history preserved). Closes Annie's naming feedback. - Updated ChangeFeedFetcherNotModifiedPagesTest.isFullyDrained_...returnsTrue assertion to match xinlian12's merged simpler isFullyDrained (unconditional noChanges -> true). Was failing CI build 6368298 with 'Expecting value to be false but was true' across all NotFromSource_TestsOnly + EmulatorTCP jobs. - Added missing PR 49276 bugfix entries to azure-cosmos-spark_4-0_2-13 and azure-cosmos-spark_4-1_2-13 CHANGELOGs (both ship the shared azure-cosmos-spark_3 code so the fix lands there). - Stripped IcM 51000001033272 references from test docstrings (per Annie - PR link in CHANGELOG is sufficient internal traceability). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stabilize asyncChangeFeedPrefetching on Windows EmulatorTcp CI build 6368554 failed only on windows2022_EmulatorOnlyIntegrationTestsTcpJava{8,17} with 'first change-feed subscription should produce at least 3 pages within 30 seconds' on both INCREMENTAL and FULL_FIDELITY parameter sets. Root cause: the test's TestNG method timeOut was TIMEOUT (40s), but the deterministic de-flake from a prior session uses 30s firstLatch.await + another 30s for the FF resume phase + bounded take(.blockLast(30s)). On the slow Windows EmulatorTcp runner the sum routinely exceeds the method timeout, and even within each phase the page-arrival cadence sometimes can't deliver 3 pages in 30s. Fix: - timeOut = TIMEOUT * 5 (200s) — matches sibling emulator tests on lines 200, 1121, 1167 that also need the longer budget. - awaitSeconds = 60L — doubles the per-phase wait window so the slow runner has room to deliver pages. - retryAnalyzer = FlakyTestRetryAnalyzer.class — consistent with the changeFeedQueryEndLSNHang test on line 1070; absorbs residual jitter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert asyncChangeFeedPrefetching to original Thread.sleep shape + add retry The deterministic CountDownLatch + .blockLast() de-flake I introduced earlier in the session over-corrected: while it eliminates non-determinism in principle, it traded subtle race jitter for slow-runner sensitivity that became a hard failure on Windows EmulatorTcp Java 8 (build 6368807 still failed all 3 retries even with TIMEOUT * 5 and 60s per-phase awaits). The original Thread.sleep(3000) shape had been passing in CI for months and predates this PR entirely — the test is exercising Reactor's byPage prefetch behavior on the change-feed stream, which is unrelated to the notModifiedPagesAllowed work this PR ships. Reverting to that proven shape is the lowest-risk path; retryAnalyzer = FlakyTestRetryAnalyzer is the only addition (consistent with sibling change-feed tests) to absorb the residual slow-runner jitter that remains in the original. Removed the now-unused CountDownLatch + TimeUnit imports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove stale TODO-style comment on surfaceOrRetryNoChangesPage The 3-line comment was xinlian12's reasoning note explaining the need for the setFeedResponseContinuationToken call directly below it. The token re-stamp already addresses the concern (re-stamps with this.changeFeedState.toString() so the surfaced empty page carries the post-rotation cursor), so the speculative wording reads as a stale TODO. Removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert change-feed half; keep query-path empty-pages fix and CF options drift fix The change-feed half of this PR (surfacing intermediate sub-feedRange 304s to Spark via a new notModifiedPagesAllowed flag) had no IcM repro and caused a regression in TransientIOErrorsRetryingIterator. Reverting all CF-specific surface: - Reverted ChangeFeedFetcher, ChangeFeedQueryImpl, Paginator, ChangeFeedPartitionReader, and the ImplementationBridgeHelpers accessor for the CF flag - Removed notModifiedPagesAllowed field/getter/setter from CosmosChangeFeedRequestOptionsImpl and CosmosChangeFeedRequestOptions - Removed ChangeFeedFetcherNotModifiedPagesTest and 3 flag-specific tests from CosmosChangeFeedRequestOptionsWithPagedFluxOptionsTest; removed the new CF flag test from CosmosContainerChangeFeedTest - Updated 6 Spark CHANGELOGs to query-only wording (allowEmptyPages) Kept the orthogonal drift fix added in pass-5 review: inheritNonContinuationFieldsFrom in CosmosChangeFeedRequestOptionsImpl now propagates endLSN, customSerializer, excludeRegions, readConsistencyStrategy, and other non-token-encoded fields when resuming via byPage(continuation). Pre-PR code only inherited maxPrefetchPageCount and throughputControlGroupName, silently dropping the rest. New 'Bugs Fixed' bullet in azure-cosmos/CHANGELOG.md for this. Net PR is now just the query-path setAllowEmptyPages(true) in ItemsPartitionReader plus the CF options drift fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge customOptions in inheritNonContinuationFieldsFrom to preserve token-mode headers When a CosmosChangeFeedRequestOptionsImpl is rebuilt from a continuation token, the constructor populates mode-derived headers (e.g., CHANGE_FEED_WIRE_FORMAT_VERSION for FULL_FIDELITY). inheritNonContinuationFieldsFrom previously did `this.customOptions = source.customOptions`, which silently dropped those required headers if the source's customOptions did not contain them. Switch to a putIfAbsent merge so token-driven headers win on key collision and source's caller-supplied headers are added otherwise. Added 2 unit tests covering the FULL_FIDELITY header preservation case and the caller-supplied + token-mode header coexistence case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Annie Liang <xin.liang@microsoft.com>

@afterclass

…V2 endpoint (Azure#47759) * Route Execute Stored Procedure requests to Thin Proxy endpoint. * Route Execute Stored Procedure and QueryPlan requests to Thin Proxy endpoint. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Adding query + thin-client tests. * Fixing tests. * Fixing tests. * Fixing tests. * Addressing review comments. * Addressing review comments. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add change feed tests for FeedRange.forFullRange and forLogicalPartition Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add thin client E2E test matrix documentation for QueryPlan review Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix testGetCurrentDateTime flaky assertion, add AAD auth support, RNTBD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor thin client query tests for reliability - Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix container leaks and Direct TCP AAD auth in tests - Fix container leak: get container reference before createContainer() so finally block can always delete. Use safeDeleteContainer() helper. - Fix Direct TCP client: apply AAD credential via applyCredential() for accounts with disableLocalAuth=true. All 89 thin client tests pass (80 query + 3 change feed + 3 point ops + 3 sprocs). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use bulkDelete in AfterClass for seeded docs cleanup Replace sequential deleteItem loop with executeBulkOperations for faster and more reliable cleanup of seeded test documents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback 1. Remove rntbd.instructions.md (not part of PR) 2. Move MAPPER variable to top of PartitionKeyInternalTest 3. Remove COSMOS.REUSE_DATABASE_ID (beforeSuiteReuse, afterSuite guard, pom.xml) 4. Keep System.setProperty in ThinClientQueryE2ETest (required — extends TestSuiteBase not ThinClientTestBase, property must be set before client build) 5. Fix RNTBD IDs to match server-side RntbdConstants.cs (ADO PR 1982503): SupportedQueryFeatures=0x00FF (String), QueryVersion=0x0100 (SmallString) The SmallString type was the root cause of 400 BadRequest — String uses 2-byte length prefix, SmallString uses 1-byte, causing token stream misparse. All 89 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align vector/FTS/hybrid tests with existing patterns - Add euclidean VectorDistance variant to vector search test - Add document ID comparison to FTS and hybrid tests (was count-only) - Add testFullTextScoreRanking: ORDER BY RANK FullTextScore with exact order comparison between Direct and thin client - All assertions now validate both result size AND content parity 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments + fix broken link Code fixes: - PartitionKeyInternalHelper: materialize fieldNames() Iterator in error msg - QueryPlanRetriever: throw IllegalStateException when partitionKeyDef null in thin client mode (was silent fallthrough) - RxDocumentClientImpl: add parentheses around compound && in boolean exprs - ThinClientStoreModel: use != instead of !(==) - ThinClientTestBase: remove stale 'uncomment' comment - ThinClientQueryE2ETest: accept status 0 (transport rejection) for invalid query — proxy returns transport error, not HTTP 400 - THINCLIENT_TEST_MATRIX.md: remove broken link, update methodology to reflect Direct TCP baseline Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments (round 2) - RntbdOperationType: add QueryPlan case to fromId()/fromType() lookups - PartitionKeyInternalHelper: sanitize PII from error message (log node type instead of raw partition key values) - THINCLIENT_TEST_MATRIX.md: remove PR-specific references and known blockers - PartitionedQueryExecutionInfo: remove stale queryRanges from backing ObjectNode after EPK conversion to prevent data inconsistency - QueryPlanRetriever: add TODO for 3 missing query features vs .NET SDK (ListAndSetAggregate, CountIf, HybridSearchSkipOrderByRewrite) 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor queryRanges deserialization — detect format from response, no ObjectNode mutation - PartitionedQueryExecutionInfo: getQueryRanges() now inspects the first range's 'min' field to detect format (string=EPK hex, array=PK-internal) and applies the correct deserialization path automatically. - Remove ObjectNode.remove('queryRanges') — no longer mutate the backing JSON. The format is detected lazily on first access. - 3-arg constructor now takes PartitionKeyDefinition (not pre-computed ranges) so conversion happens inside getQueryRanges(), keeping the DTO self-contained. - QueryPlanRetriever: pass partitionKeyDefinition to constructor instead of pre-converting ranges externally. 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use agnostic QueryRangesFormat hint instead of PartitionKeyDefinition presence Introduce QueryRangesFormat enum (EPK_HEX_STRING, PARTITION_KEY_INTERNAL_ARRAY) as the deserialization hint. The hint guides which path to try first, but getQueryRanges() always validates by inspecting the actual min node type — if the hint doesn't match the wire format, it falls through to the other path. Naming is transport-agnostic (no Proxy/ThinClient/Gateway references). PartitionKeyDefinition is only stored when needed for EPK conversion, not used as a format signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fetchQueryPlanForValidation caller after merge Pass null for the DocumentCollection parameter so the public fetchQueryPlanForValidation entry point compiles against the new fetchQueryPlan signature introduced during the merge with upstream/main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make PartitionedQueryExecutionInfo(ObjectNode, RequestTimeline) ctor public The upstream ReadManyByPartitionKeyQueryPlanValidationTest (added in Azure#48801) constructs PartitionedQueryExecutionInfo directly from a test package. Upstream/main exposed this ctor as public; restore that visibility so the new test compiles after the merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Cosmos] Default thin-client to enabled and add HTTP/2 connectivity-probe gate Adds an EndpointOrchestrator that fans out POST /connectivity-probe to every thin-client regional endpoint after each topology refresh. SDK only routes data-plane traffic to thin-client (Gateway V2) when all regional probes succeed across N consecutive refresh cycles (configurable via COSMOS.THINCLIENT_PROBE_FAILURE_THRESHOLD, default 2); otherwise traffic falls back to Gateway V1 at the next refresh boundary. No mid-flight fallback. Caveats: - Probe wiring is skipped entirely for Direct mode and when HTTP/2 is not configured; controlled by RxDocumentClientImpl.useThinClient. - QueryPlan, metadata reads, and AllVersionsAndDeletes change feed continue to route through Compute Gateway (Gateway V1). - Probe failures are absorbed inside the orchestrator and the trigger is fire-and-forget on the GEM scheduler, so probe issues can never trip CosmosClient initialization or fail a topology refresh. - EndpointOrchestrator implements Closeable and is closed from GlobalEndpointManager.close() so no further probes are issued after client shutdown. THINCLIENT_ENABLED now defaults to true; opt out via COSMOS.THINCLIENT_ENABLED=false or COSMOS.THINCLIENT_PROBE_ENABLED=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR Azure#49437 review comments: probe single-flight, body-drain lifecycle, cancellable in-flight probes - EndpointOrchestrator: fold body-drain into probe Mono via flatMap+then(perProbeTimeout) so a slow/hanging response body cannot leak resources outside the cycle budget (Copilot #1, deep-review #3, jeet HIGH-2 minor). - EndpointOrchestrator: add single-flight CAS (cycleInProgress) plus monotonic cycle id; closed-check inside applyCycleResult drops late results so a post-close cycle cannot mutate health state (deep-review #1+#2, jeet HIGH-2). - EndpointOrchestrator: re-evaluate closed/feature-flag/endpoints at subscription time via Mono.defer so GEM.close() cancellation is honored before any HTTP I/O is issued. - GlobalEndpointManager: retain probe Disposable in AtomicReference; close() now disposes the in-flight probe subscription so probe work cannot outlive the GEM/CosmosClient (Copilot #2, deep-review #2). - CHANGELOG: moved entry to unreleased 4.82.0-beta.1, reworded to honestly describe optimistic startup, N=2 RED-to-fallback hysteresis, and Direct-mode/metadata exclusion (Copilot #3, deep-review #4). Tests: 45 unit tests pass (EndpointOrchestratorTests + ConfigsTests + ThinClientProbeWiringTests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR Azure#49437 second-batch review feedback - LocationCache.getThinClientRegionalEndpoints now walks both read and write region endpoint maps so single-master write-region failures still flip the probe gate. - EndpointOrchestrator.forceUnhealthy(reason) provides a non-HTTP path to flip the gate; GlobalEndpointManager calls it when topology says thin-client is eligible but no regional endpoint resolves. - Symmetric hysteresis: new COSMOS.THINCLIENT_PROBE_RECOVERY_THRESHOLD (default 1) so operators can require N consecutive GREEN cycles before flipping back to proxy. - Extracted RxDocumentClientImpl.useThinClientStoreModel(...) body into package-private static shouldUseThinClientStoreModel for direct unit testability; added ThinClientRoutingGateTests covering 9 routing paths. - EndpointOrchestratorTests.stubResponse now returns Mono.empty() to avoid Unpooled.EMPTY_BUFFER refCnt underflow across multiple probe calls. - Removed unused locals; added recoveryThresholdRequiresMultipleGreenCycles, forceUnhealthy_flipsGateToRedWithoutRunningProbe, forceUnhealthy_onClosedOrchestrator_isNoOp tests. All 57 unit tests in the touched files pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify CHANGELOG: probe recovery threshold is now configurable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR Azure#49437 review (rounds 4-8): reactor-chain probe, FQN cleanup, fix gwV2Cto and ThinClient user-agent assertions - GlobalEndpointManager: convert thin-client probe trigger to a Mono<Void> chained into the topology-refresh reactor pipeline (replaces fire-and-forget subscribe). Removes thinClientProbeDisposable field and its close() handling since cancellation now propagates through the outer subscription. - EndpointProbeClient/EndpointProbeClientTests/ThinClientProbeWiringTests: replace inline FQNs with imports (java.io.Closeable, java.util.List, java.net.ConnectException, com.azure.cosmos.implementation.http.HttpHeaders). - ClientConfigDiagnosticsTest: compute gwV2Cto dynamically from Configs.isThinClientEnabled() so assertions remain valid after the default flip to true. - ConfigsTests: update default-threshold assertions from 2 to 1 to match DEFAULT_THINCLIENT_PROBE_FAILURE_THRESHOLD=1. - UserAgentContainerTest.UserAgentIntegration: expect '|F4' suffix because the ThinClient feature flag (1 << 2) is now included by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI test fallout from default ThinClient enablement UserAgentSuffixTest.validateUserAgentSuffix and CosmosDiagnosticsTest.generateHttp2OptedInUserAgentIfRequired: include UserAgentFeatureFlags.ThinClient in computed |F<hex> suffix when COSMOS.THINCLIENT_ENABLED is true (now default after Gateway V2 default enablement). Mirrors RxDocumentClientImpl.addUserAgentSuffix + UserAgentContainer.setFeatureEnabledFlagsAsSuffix behavior. SinglePartitionDocumentQueryTest.querySinglePartitionDocuments: spy on both gateway-proxy and thin-proxy and assert exactly one invocation. Previous code only spied on the proxy implied by useThinClient() config intent, which races with the probe-healthy gate -- routing AND's intent with isProxyProbeHealthy() so on first cycle the request may go through gateway even when thin-client is configured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe in Http2PingKeepaliveTest The test installs an iptables DROP on thin-client port 10250 to verify that Http2PingHandler closes the broken connection after consecutive PING ACK timeouts and the recovery request uses a new connection on the same regional endpoint. After default Gateway V2 enablement, the connectivity probe also fires HTTP/2 POSTs to port 10250 on every account refresh. With iptables dropping that port, the probe trips proxyHealthy=false, useThinClient StoreModel() returns false, and the data plane request routes through Gateway V1 on port 443 -- which iptables is not dropping. Result: the PING handler never fires, the warm-up and recovery requests use the same gateway channel, and the assertion 'recovery channel must differ from initial' fails (both ended up as 77af2e47 on build 6419227). Set COSMOS.THINCLIENT_PROBE_ENABLED=false in beforeClass so the probe short-circuits to a no-op, EndpointProbeClient.proxyHealthy stays optimistically true, and the data plane request actually flows over port 10250 where the iptables DROP can take effect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe by default for E2E tests in TestSuiteBase Build 6424287 surfaced two new failure patterns: 1. CosmosNotFoundTests.performBulkOnDeletedContainerWithGatewayV2 (45 failures) - asserts substatus 1003 from the thin-client routing path, but observed 0 because the data plane was routed to Gateway V1 instead of the proxy. 2. PerPartitionCircuitBreakerE2ETests.*Gateway (26 failures) - TestSuiteBase.assertThinClientEndpointUsed could not find any request whose endpoint contained ':10250/', i.e. nothing actually went to the thin-client proxy. Both patterns trace to the same source: the new connectivity probe is enabled by default, the proxy-side /connectivity-probe endpoint is not deployed in every CI test account yet, and the default failure threshold is 1. So after the first probe cycle the SDK marks the proxy unhealthy and routes data plane traffic to Gateway V1, which breaks tests that explicitly assert thin-client routing. Disable the probe by default in TestSuiteBase's static initializer (only when the property is not already set), so all E2E tests inherit deterministic, configuration-driven routing. Tests that exercise the probe itself (EndpointProbeClientTests, ThinClientProbeWiringTests) set the property explicitly in @BeforeMethod and are not affected. Also drop the now-redundant per-class override in Http2PingKeepaliveTest - the base class disables it, and the test's @afterclass clear would otherwise re-enable the probe for any subsequent E2E test sharing the JVM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Disable thin-client probe by default for E2E tests in TestSuiteBase" This reverts commit ad9e3df. * Add per-class thinclient probe disable in CosmosNotFoundTests and PerPartitionCircuitBreakerE2ETests Companion to the prior revert. The revert undid the global TestSuiteBase probe disable (which masked production behaviour). This commit adds the necessary per-class disable to the two test classes whose assertions explicitly require thinclient routing: CosmosNotFoundTests (thinclient group) and PerPartitionCircuitBreakerE2ETests (fi-thinclient-multi-master group). Both clear the property in their @afterclass. Http2PingKeepaliveTest already has its own disable (restored by the revert). Production callers continue to get the connectivity probe ON by default with the production failure threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add unit tests for QueryPlan and stored-procedure thin-client routing Covers 5 new scenarios in ThinClientRoutingGateTests: - ExecuteStoredProcedure on a StoredProcedure resource routes to thin client - Non-execute StoredProcedure ops (Create) route to Gateway V1 - OperationType.QueryPlan routes to thin client - QueryPlan returns false when probe is unhealthy - ExecuteStoredProcedure returns false when probe is unhealthy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove endpoint-probe content from QueryPlan PR branch Reverts the merge of jeet1995/thin-client-probe-flow that was brought into this PR earlier, while keeping the branch up-to-date with upstream/main. Net diff vs upstream/main is now only the QueryPlan and StoredProcedure Gateway V2 routing changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Thread DocumentCollection into readMany query-plan validation readManyByPartitionKeys validates any caller-supplied custom query by fetching a query plan and asserting it is single-partition and non-hybrid. Until now that validation called fetchQueryPlanForValidation with no DocumentCollection, so the plan request was pinned to Gateway V1 (useGatewayMode = (partitionKeyDefinition == null)). Thread the container's DocumentCollection from RxDocumentClientImpl.validateCustomQueryForReadManyByPartitionKeys -> DocumentQueryExecutionContextFactory.fetchQueryPlanForValidation so QueryPlanRetriever has the PartitionKeyDefinition needed to convert PartitionKeyInternal-formatted queryRanges from the thin client (Gateway V2) proxy into the EPK-hex Range<String> entries the query pipeline consumes. With this wiring, the validation query plan goes to the thin client when the client is configured for it, and remains on Gateway V1 otherwise. No behavior change on the non-thin-client path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client QueryPlan error returning statusCode 0 instead of 400 The thin-client/Gateway-V2 proxy returns a non-2xx response with a raw, non-JSON, NUL-padded error body for invalid-syntax queries. In RxGatewayStoreModel.validateOrThrow, new CosmosError(body) attempted to parse that body as JSON and threw IllegalArgumentException, which escaped the method before the existing status-carrying throw could run. Upstream then wrapped it as statusCode 0. Wrap the CosmosError(body) construction in a narrow try/catch (IllegalArgumentException) and fall back to the non-parsing CosmosError(errorCode, message) constructor with a sanitized body. The existing throw now fires with the correct status (400) and the proxy error text is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden ThinClientQueryE2ETest assertions (F1-F5) F1: strict 400 + unconditional thin-client endpoint check on invalid query, locking the statusCode-0->400 fix. F2: ordered-vs-sorted-set document-ID comparison (ORDER BY sequence-compared, others set-compared). F3: ID-set equality + no-duplicate check across drained continuation pages. F4: numeric-tolerance (1e-6) comparison for scalar and GROUP BY aggregates to avoid float-formatting false mismatches. F5: validated vector/full-text/hybrid queries match Direct vs thin-client end-to-end through the proxy. Validated live via -Pthinclient: 84 tests, 0 failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace thin-client test matrix with reverse-engineered E2E test spec Rewrite THINCLIENT_TEST_MATRIX.md as a reviewable test-design specification reverse-engineered from the committed ThinClientQueryE2ETest code (84 tests). Documents the differential-testing oracle (Direct :443 baseline vs thin client :10250 SUT), the data-model fixture, every assertion contract (endpoint provenance, ordered-vs-unordered ID equality, scalar/GROUP BY tolerance), the full 84-test matrix, the F1-F5 hardened special cases (continuation draining, invalid-query 400, vector/FTS/hybrid ranking, readMany validation path), advertised-feature coverage, and known gaps (CountIf/DCount/MultipleOrderBy) for reviewer sign-off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply Aditya Sarpotdar review feedback to thin-client E2E tests Executes actionable review feedback on ThinClientQueryE2ETest plus two harness fixes surfaced during live validation against thin-client-multi-region-ci. ThinClientQueryE2ETest: - Strict ordering: ORDER BY results validated with isStrictlyOrdered across the board (stricter-by-default, per reviewer guidance) instead of set parity. - Add testMultipleOrderBy() with a composite-index container to cover the MultipleOrderBy query feature. - Add testDCount() using the canonical Cosmos DCount idiom (COUNT over a DISTINCT VALUE subquery); SQL-standard COUNT(DISTINCT ...) is not valid Cosmos SQL grammar. - Reword multi-EPK-range comment/javadoc to reflect emulator/backend reality (multiple metadata ranges served by a single backend partition; SDK routing and query pipeline still exercised). Harness fixes: - TestSuiteBase: add wait-and-poll utility waitForCollectionToBeAvailableToRead (predicate on NotFound/substatus 1013) to deflake "Collection is not yet available for read" on freshly created containers; update call sites in OrderbyDocumentQueryTest, NonStreamingOrderByQueryVectorSearchTest, QueryValidationTests, ReadFeedCollectionsTest. - SinglePartitionDocumentQueryTest: make the processMessage Mockito assertion mode-aware (thin-client routes query plan + query => times(2)). Validated live against thin-client-multi-region-ci: 86/86 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: opt-in thin-client QueryPlan routing + kill-switch, stop advertising CountIf, remove PR-reference comments * Address review feedback: broaden gateway error fallback to ClassCastException, drop response ObjectNode mutation, fail-fast on empty queryRanges, align readMany routing test with opt-in * Make thin-client QueryPlan no-EPK-headers contract explicit in wrapInHttpRequest * Fix missing StandardCharsets import in RxGatewayStoreModelTest * Test: add QueryOracle-derived LIKE/scalar-expression thin-client parity tests; drain only first page in OperationPolicies readAllItems to avoid full-container enumeration timeout * Test: verify a cached proxy-generated query plan still executes correctly through the thin client * Test: cross-partition tests use a dedicated multi-physical-partition fixture and assert >1 partition key range contacted; enforce ordered full-row comparison when id is not projected; clearer naming * Add CHANGELOG entry for QueryPlan request routing to Gateway V2 * Call out Execute Stored Procedure support in CHANGELOG entry * Fix flaky region/timing-sensitive fault-injection E2E tests CustomerWorkflowPartitionLevelCircuitBreakerTest and CustomerWorkflowHighE2ETimeoutTest: bump the ThresholdBasedAvailabilityStrategy threshold from 100ms to 300ms so the cold primary-region connection can dispatch the request and record the injected fault before the availability-strategy hedge to a warm secondary region wins the race and cancels it (which previously yielded 0 fault hits). PerPartitionCircuitBreakerE2ETests: before asserting short-circuit routing, wait until the first preferred region is actually marked Unavailable, removing the race between the failure-counter estimate and the asynchronous routing exclusion. Scoped to single-faulty-region configs (expectedRegionCountWithFailures == 1) and run once so fault-in-all-regions configs (which can never short-circuit) do not hit the method timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove test AAD auth toggle; move thin-client test matrix to docs/ and align with ThinClientQueryE2ETest * Add cross-partition aggregate / GROUP BY parity tests for thin-client query Adds 7 thin-client E2E tests covering cross-partition aggregate merge (COUNT/SUM/AVG/MIN/MAX) and GROUP BY, plus two helpers that assert Direct vs thin-client value parity and verify the query genuinely fanned out (distinctPartitionKeyRangesContacted > 1), exercising the cross-partition MERGE path the single-partition aggregate tests miss. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix EPK over-span; add QueryPlan parity tests For a partial (prefix) hierarchical partition key, the thin-client / Gateway V2 path previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter, so the proxy resolved the request to the owning physical partition and returned every co-located document (e.g. 18 instead of 6). ThinClientStoreModel.wrapInHttpRequest now sets READ_FEED_KEY_TYPE= EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from() so RntbdRequestHeaders.addStartAndEndKeys serializes the prefix EPK sub-range as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. Adds ThinClientQueryE2ETest coverage: QueryPlan parity across sources and a hierarchical prefix half-open range test asserting thin-client matches Direct TCP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enforce QueryPlan thin-client routing invariant and harden container-readiness probe for thin-client-default-on - Remove the QueryPlan exemption in assertThinClientEndpointUsed: when ThinClient is opted in with HTTP/2 enabled, QueryPlan calls MUST also route to Gateway V2 (:10250), matching production default DEFAULT_THINCLIENT_QUERY_PLAN_ENABLED=true. - Restore public static visibility on ThinClientTestBase.assertThinClientEndpointUsed / assertGatewayEndpointUsed so cross-package static imports (CosmosMultiHashTest) compile. - Add OWNER_RESOURCE_NOT_EXISTS (subStatus 1003) to the retryable NOTFOUND allowlist in TestSuiteBase.isRetryableCollectionReadinessFailure. With thin-client default-on the beforeSuite readability probe (a QueryPlan) routes through GW V2; a freshly-created container not yet propagated to a secondary region returns 404/1003, which must be treated as a transient readiness failure instead of cascade-skipping the suite. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add kill-switch flip validation for QueryPlan thin-client routing Validate the COSMOS.THINCLIENT_QUERY_PLAN_ENABLED sysprop/env kill-switch in both directions: - Unit: new case in ReadManyByPartitionKeyQueryPlanRoutingTest asserts QueryPlan is forced to Gateway V1 (useGatewayMode=true) when the flag is disabled, even with thin-client eligible. - E2E: make TestSuiteBase.assertThinClientEndpointUsed flag-aware so QueryPlan endpoint assertions flip based on the live-read flag while data operations continue to route to the thin-client endpoint. Both scenarios verified green (unit 3/3; E2E 112/112, oracle:true for flag ON and flag OFF). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore ambient COSMOS.THINCLIENT_ENABLED in ThinClientTestBase teardown ThinClientTestBase teardown unconditionally cleared the COSMOS.THINCLIENT_ENABLED system property. Since Configs.isThinClientEnabled() reads the property lazily per client build, a ThinClientTestBase subclass running before a property-dependent class inherited from main (e.g. CosmosMultiHashTest, ThinClientQueryE2ETest) in the same JVM wiped the ambient -DCOSMOS.THINCLIENT_ENABLED=true flag supplied by the thin-client CI lane. Later clients then built with thin client disabled and routed to :443, breaking the :10250 thin-client endpoint assertions. Save the prior value on enable and restore it on teardown (clear only when it was originally absent), mirroring CosmosConsistencyOverrideValidationTest's restoreSystemProperty pattern. Helpers changed from static to instance so the saved value is per-instance, avoiding races across TestNG factory instances. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove redundant COSMOS.THINCLIENT_ENABLED mutation from ThinClientTestBase; revert workflow tests to main ThinClientTestBase no longer sets/clears COSMOS.THINCLIENT_ENABLED per class. The 'thinclient' Maven lane already sets THINCLIENT_ENABLED=true and HTTP2_ENABLED=true JVM-wide via failsafe systemPropertyVariables, and all thin classes run only in that lane (groups={thinclient}). The prior per-class clearProperty in @afterclass wiped the ambient -D for later byte-identical-to-main classes in the same JVM, mis-routing requests to :443. Relying on the lane flag (matching ThinClientQueryE2ETest) removes that footgun. Also revert CustomerWorkflowHighE2ETimeoutTest and CustomerWorkflowPartitionLevelCircuitBreakerTest to upstream/main (availability-strategy threshold stays 100ms), since this PR requires no changes there. Verified locally (AZURE_TEST_MODE=LIVE, -Pthinclient,query,consistency-overrides): 234 passed, 0 failures/skips; QueryPlan routes to :10250. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Abhijeet Mohanty <copilot@noreply.local>

…probe mechanism. (Azure#49437) * Route Execute Stored Procedure requests to Thin Proxy endpoint. * Route Execute Stored Procedure and QueryPlan requests to Thin Proxy endpoint. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Adding query + thin-client tests. * Fixing tests. * Fixing tests. * Fixing tests. * Addressing review comments. * Addressing review comments. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add change feed tests for FeedRange.forFullRange and forLogicalPartition Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add thin client E2E test matrix documentation for QueryPlan review Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix testGetCurrentDateTime flaky assertion, add AAD auth support, RNTBD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor thin client query tests for reliability - Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix container leaks and Direct TCP AAD auth in tests - Fix container leak: get container reference before createContainer() so finally block can always delete. Use safeDeleteContainer() helper. - Fix Direct TCP client: apply AAD credential via applyCredential() for accounts with disableLocalAuth=true. All 89 thin client tests pass (80 query + 3 change feed + 3 point ops + 3 sprocs). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use bulkDelete in AfterClass for seeded docs cleanup Replace sequential deleteItem loop with executeBulkOperations for faster and more reliable cleanup of seeded test documents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback 1. Remove rntbd.instructions.md (not part of PR) 2. Move MAPPER variable to top of PartitionKeyInternalTest 3. Remove COSMOS.REUSE_DATABASE_ID (beforeSuiteReuse, afterSuite guard, pom.xml) 4. Keep System.setProperty in ThinClientQueryE2ETest (required — extends TestSuiteBase not ThinClientTestBase, property must be set before client build) 5. Fix RNTBD IDs to match server-side RntbdConstants.cs (ADO PR 1982503): SupportedQueryFeatures=0x00FF (String), QueryVersion=0x0100 (SmallString) The SmallString type was the root cause of 400 BadRequest — String uses 2-byte length prefix, SmallString uses 1-byte, causing token stream misparse. All 89 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align vector/FTS/hybrid tests with existing patterns - Add euclidean VectorDistance variant to vector search test - Add document ID comparison to FTS and hybrid tests (was count-only) - Add testFullTextScoreRanking: ORDER BY RANK FullTextScore with exact order comparison between Direct and thin client - All assertions now validate both result size AND content parity 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments + fix broken link Code fixes: - PartitionKeyInternalHelper: materialize fieldNames() Iterator in error msg - QueryPlanRetriever: throw IllegalStateException when partitionKeyDef null in thin client mode (was silent fallthrough) - RxDocumentClientImpl: add parentheses around compound && in boolean exprs - ThinClientStoreModel: use != instead of !(==) - ThinClientTestBase: remove stale 'uncomment' comment - ThinClientQueryE2ETest: accept status 0 (transport rejection) for invalid query — proxy returns transport error, not HTTP 400 - THINCLIENT_TEST_MATRIX.md: remove broken link, update methodology to reflect Direct TCP baseline Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments (round 2) - RntbdOperationType: add QueryPlan case to fromId()/fromType() lookups - PartitionKeyInternalHelper: sanitize PII from error message (log node type instead of raw partition key values) - THINCLIENT_TEST_MATRIX.md: remove PR-specific references and known blockers - PartitionedQueryExecutionInfo: remove stale queryRanges from backing ObjectNode after EPK conversion to prevent data inconsistency - QueryPlanRetriever: add TODO for 3 missing query features vs .NET SDK (ListAndSetAggregate, CountIf, HybridSearchSkipOrderByRewrite) 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor queryRanges deserialization — detect format from response, no ObjectNode mutation - PartitionedQueryExecutionInfo: getQueryRanges() now inspects the first range's 'min' field to detect format (string=EPK hex, array=PK-internal) and applies the correct deserialization path automatically. - Remove ObjectNode.remove('queryRanges') — no longer mutate the backing JSON. The format is detected lazily on first access. - 3-arg constructor now takes PartitionKeyDefinition (not pre-computed ranges) so conversion happens inside getQueryRanges(), keeping the DTO self-contained. - QueryPlanRetriever: pass partitionKeyDefinition to constructor instead of pre-converting ranges externally. 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use agnostic QueryRangesFormat hint instead of PartitionKeyDefinition presence Introduce QueryRangesFormat enum (EPK_HEX_STRING, PARTITION_KEY_INTERNAL_ARRAY) as the deserialization hint. The hint guides which path to try first, but getQueryRanges() always validates by inspecting the actual min node type — if the hint doesn't match the wire format, it falls through to the other path. Naming is transport-agnostic (no Proxy/ThinClient/Gateway references). PartitionKeyDefinition is only stored when needed for EPK conversion, not used as a format signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fetchQueryPlanForValidation caller after merge Pass null for the DocumentCollection parameter so the public fetchQueryPlanForValidation entry point compiles against the new fetchQueryPlan signature introduced during the merge with upstream/main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make PartitionedQueryExecutionInfo(ObjectNode, RequestTimeline) ctor public The upstream ReadManyByPartitionKeyQueryPlanValidationTest (added in #48801) constructs PartitionedQueryExecutionInfo directly from a test package. Upstream/main exposed this ctor as public; restore that visibility so the new test compiles after the merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(cosmos): HTTP/2 PING handler must observe child-stream read activity, not just parent-pipeline frames Follow-up to #49095. Root cause: Http2PingHandler is installed on the parent (TCP) channel and only updated lastReadActivityNanos from parent-pipeline channelRead -- which sees PING ACK / SETTINGS / GOAWAY but NOT HEADERS / DATA frames. With reactor-netty's Http2MultiplexHandler in the parent pipeline, all application H2 traffic is demuxed onto per-stream child channels, so a connection serving thousands of QPS still looked idle to the PING handler, causing unnecessary PINGs and measurable P95 overhead under load. Fix: - New package-private AttributeKey<AtomicLong> LAST_CHILD_STREAM_READ_ACTIVITY_NANOS seeded on the parent channel in handlerAdded. - Http2PingCloseRewrapHandler (already installed in every child stream's pipeline) overrides channelReadComplete to stamp the parent attribute via accumulateAndGet(now, Math::max) -- once per read cycle, monotonic. - Http2PingHandler.maybeSendPing now uses effectiveLastReadActivityNanos = max(field, child attr) for both the idle-threshold check AND the outstanding-PING early-return: if child-stream reads arrived after an outstanding PING, clear pingOutstandingSinceNanos and reset consecutiveFailures (defense against middleboxes that drop PING-ACK but pass application H2 frames). - Names use lastReadActivity / read activity -- both signals are inbound reads, never writes. Tests: two new unit tests in Http2PingHandlerTest covering the suppression path and the outstanding-PING reset path; both bypass real time via reflection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * perf: install Http2PingCloseRewrapHandler in doOnConnected, not doOnRequest The previous .doOnRequest() chained onto the HTTP/2 HttpClient builder wrapped every outgoing request's Mono in a Reactor operator. That paid per-request costs even when the rewrap handler was already installed: * MonoPeekTerminal wrap + per-request subscriber allocation (InternalMonoOperator.subscribe +2.60% in alloc profile) * per-request BiConsumer lambda invocation (Http2Pool\$\...run +1.16% in alloc profile) * per-request channel pipeline String-key walk via pipeline.get(...) Long-run async-profiler diff vs v4.80.0 on D2 attributed the PR's -7.6% QPS / +14.5% mean / +166% tail-max regression primarily to a +10.2% allocation rate -- with the operators above dominating the delta. Move the install into the existing .doOnConnected() block, which already runs once per HTTP/2 child stream (the customHeaderCleaner install right above it relies on the same contract). The rewrap handler is @Sharable, so installing once per stream is correct and amortizes the install + pipeline-walk away from the hot path. The ch.parent() != null guard is kept so the install only targets child streams, never the parent TCP channel (which uses Http2ParentChannelExceptionHandler instead). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Cosmos] Default thin-client to enabled and add HTTP/2 connectivity-probe gate Adds an EndpointOrchestrator that fans out POST /connectivity-probe to every thin-client regional endpoint after each topology refresh. SDK only routes data-plane traffic to thin-client (Gateway V2) when all regional probes succeed across N consecutive refresh cycles (configurable via COSMOS.THINCLIENT_PROBE_FAILURE_THRESHOLD, default 2); otherwise traffic falls back to Gateway V1 at the next refresh boundary. No mid-flight fallback. Caveats: - Probe wiring is skipped entirely for Direct mode and when HTTP/2 is not configured; controlled by RxDocumentClientImpl.useThinClient. - QueryPlan, metadata reads, and AllVersionsAndDeletes change feed continue to route through Compute Gateway (Gateway V1). - Probe failures are absorbed inside the orchestrator and the trigger is fire-and-forget on the GEM scheduler, so probe issues can never trip CosmosClient initialization or fail a topology refresh. - EndpointOrchestrator implements Closeable and is closed from GlobalEndpointManager.close() so no further probes are issued after client shutdown. THINCLIENT_ENABLED now defaults to true; opt out via COSMOS.THINCLIENT_ENABLED=false or COSMOS.THINCLIENT_PROBE_ENABLED=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review comments: probe single-flight, body-drain lifecycle, cancellable in-flight probes - EndpointOrchestrator: fold body-drain into probe Mono via flatMap+then(perProbeTimeout) so a slow/hanging response body cannot leak resources outside the cycle budget (Copilot #1, deep-review #3, jeet HIGH-2 minor). - EndpointOrchestrator: add single-flight CAS (cycleInProgress) plus monotonic cycle id; closed-check inside applyCycleResult drops late results so a post-close cycle cannot mutate health state (deep-review #1+#2, jeet HIGH-2). - EndpointOrchestrator: re-evaluate closed/feature-flag/endpoints at subscription time via Mono.defer so GEM.close() cancellation is honored before any HTTP I/O is issued. - GlobalEndpointManager: retain probe Disposable in AtomicReference; close() now disposes the in-flight probe subscription so probe work cannot outlive the GEM/CosmosClient (Copilot #2, deep-review #2). - CHANGELOG: moved entry to unreleased 4.82.0-beta.1, reworded to honestly describe optimistic startup, N=2 RED-to-fallback hysteresis, and Direct-mode/metadata exclusion (Copilot #3, deep-review #4). Tests: 45 unit tests pass (EndpointOrchestratorTests + ConfigsTests + ThinClientProbeWiringTests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 second-batch review feedback - LocationCache.getThinClientRegionalEndpoints now walks both read and write region endpoint maps so single-master write-region failures still flip the probe gate. - EndpointOrchestrator.forceUnhealthy(reason) provides a non-HTTP path to flip the gate; GlobalEndpointManager calls it when topology says thin-client is eligible but no regional endpoint resolves. - Symmetric hysteresis: new COSMOS.THINCLIENT_PROBE_RECOVERY_THRESHOLD (default 1) so operators can require N consecutive GREEN cycles before flipping back to proxy. - Extracted RxDocumentClientImpl.useThinClientStoreModel(...) body into package-private static shouldUseThinClientStoreModel for direct unit testability; added ThinClientRoutingGateTests covering 9 routing paths. - EndpointOrchestratorTests.stubResponse now returns Mono.empty() to avoid Unpooled.EMPTY_BUFFER refCnt underflow across multiple probe calls. - Removed unused locals; added recoveryThresholdRequiresMultipleGreenCycles, forceUnhealthy_flipsGateToRedWithoutRunningProbe, forceUnhealthy_onClosedOrchestrator_isNoOp tests. All 57 unit tests in the touched files pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify CHANGELOG: probe recovery threshold is now configurable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review (rounds 4-8): reactor-chain probe, FQN cleanup, fix gwV2Cto and ThinClient user-agent assertions - GlobalEndpointManager: convert thin-client probe trigger to a Mono<Void> chained into the topology-refresh reactor pipeline (replaces fire-and-forget subscribe). Removes thinClientProbeDisposable field and its close() handling since cancellation now propagates through the outer subscription. - EndpointProbeClient/EndpointProbeClientTests/ThinClientProbeWiringTests: replace inline FQNs with imports (java.io.Closeable, java.util.List, java.net.ConnectException, com.azure.cosmos.implementation.http.HttpHeaders). - ClientConfigDiagnosticsTest: compute gwV2Cto dynamically from Configs.isThinClientEnabled() so assertions remain valid after the default flip to true. - ConfigsTests: update default-threshold assertions from 2 to 1 to match DEFAULT_THINCLIENT_PROBE_FAILURE_THRESHOLD=1. - UserAgentContainerTest.UserAgentIntegration: expect '|F4' suffix because the ThinClient feature flag (1 << 2) is now included by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI test fallout from default ThinClient enablement UserAgentSuffixTest.validateUserAgentSuffix and CosmosDiagnosticsTest.generateHttp2OptedInUserAgentIfRequired: include UserAgentFeatureFlags.ThinClient in computed |F<hex> suffix when COSMOS.THINCLIENT_ENABLED is true (now default after Gateway V2 default enablement). Mirrors RxDocumentClientImpl.addUserAgentSuffix + UserAgentContainer.setFeatureEnabledFlagsAsSuffix behavior. SinglePartitionDocumentQueryTest.querySinglePartitionDocuments: spy on both gateway-proxy and thin-proxy and assert exactly one invocation. Previous code only spied on the proxy implied by useThinClient() config intent, which races with the probe-healthy gate -- routing AND's intent with isProxyProbeHealthy() so on first cycle the request may go through gateway even when thin-client is configured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe in Http2PingKeepaliveTest The test installs an iptables DROP on thin-client port 10250 to verify that Http2PingHandler closes the broken connection after consecutive PING ACK timeouts and the recovery request uses a new connection on the same regional endpoint. After default Gateway V2 enablement, the connectivity probe also fires HTTP/2 POSTs to port 10250 on every account refresh. With iptables dropping that port, the probe trips proxyHealthy=false, useThinClient StoreModel() returns false, and the data plane request routes through Gateway V1 on port 443 -- which iptables is not dropping. Result: the PING handler never fires, the warm-up and recovery requests use the same gateway channel, and the assertion 'recovery channel must differ from initial' fails (both ended up as 77af2e47 on build 6419227). Set COSMOS.THINCLIENT_PROBE_ENABLED=false in beforeClass so the probe short-circuits to a no-op, EndpointProbeClient.proxyHealthy stays optimistically true, and the data plane request actually flows over port 10250 where the iptables DROP can take effect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe by default for E2E tests in TestSuiteBase Build 6424287 surfaced two new failure patterns: 1. CosmosNotFoundTests.performBulkOnDeletedContainerWithGatewayV2 (45 failures) - asserts substatus 1003 from the thin-client routing path, but observed 0 because the data plane was routed to Gateway V1 instead of the proxy. 2. PerPartitionCircuitBreakerE2ETests.*Gateway (26 failures) - TestSuiteBase.assertThinClientEndpointUsed could not find any request whose endpoint contained ':10250/', i.e. nothing actually went to the thin-client proxy. Both patterns trace to the same source: the new connectivity probe is enabled by default, the proxy-side /connectivity-probe endpoint is not deployed in every CI test account yet, and the default failure threshold is 1. So after the first probe cycle the SDK marks the proxy unhealthy and routes data plane traffic to Gateway V1, which breaks tests that explicitly assert thin-client routing. Disable the probe by default in TestSuiteBase's static initializer (only when the property is not already set), so all E2E tests inherit deterministic, configuration-driven routing. Tests that exercise the probe itself (EndpointProbeClientTests, ThinClientProbeWiringTests) set the property explicitly in @BeforeMethod and are not affected. Also drop the now-redundant per-class override in Http2PingKeepaliveTest - the base class disables it, and the test's @AfterClass clear would otherwise re-enable the probe for any subsequent E2E test sharing the JVM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Disable thin-client probe by default for E2E tests in TestSuiteBase" This reverts commit ad9e3dff1db97df0d53a6f92aaec709676caff19. * Add per-class thinclient probe disable in CosmosNotFoundTests and PerPartitionCircuitBreakerE2ETests Companion to the prior revert. The revert undid the global TestSuiteBase probe disable (which masked production behaviour). This commit adds the necessary per-class disable to the two test classes whose assertions explicitly require thinclient routing: CosmosNotFoundTests (thinclient group) and PerPartitionCircuitBreakerE2ETests (fi-thinclient-multi-master group). Both clear the property in their @AfterClass. Http2PingKeepaliveTest already has its own disable (restored by the revert). Production callers continue to get the connectivity probe ON by default with the production failure threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add unit tests for QueryPlan and stored-procedure thin-client routing Covers 5 new scenarios in ThinClientRoutingGateTests: - ExecuteStoredProcedure on a StoredProcedure resource routes to thin client - Non-execute StoredProcedure ops (Create) route to Gateway V1 - OperationType.QueryPlan routes to thin client - QueryPlan returns false when probe is unhealthy - ExecuteStoredProcedure returns false when probe is unhealthy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove endpoint-probe content from QueryPlan PR branch Reverts the merge of jeet1995/thin-client-probe-flow that was brought into this PR earlier, while keeping the branch up-to-date with upstream/main. Net diff vs upstream/main is now only the QueryPlan and StoredProcedure Gateway V2 routing changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * review: address PR #49437 items 3 and 4 Item 3: remove unused COSMOS.USE_AAD_AUTH system property from the thinclient profile in azure-cosmos-tests/pom.xml. The Maven property cosmos.use.aad.auth is not defined anywhere in the repo, so the substitution produced a literal that Boolean.parseBoolean reads as false; TestSuiteBase already falls back to the COSMOS_USE_AAD_AUTH env var. Item 4: in Http2PingKeepaliveTest, gate the THINCLIENT_PROBE_ENABLED=false override behind !THIN_CLIENT_ENABLED so the probe is disabled only in the pure Compute Gateway (Gateway V2) branch on port 443, where thin-client routing is off entirely and probe POSTs to port 10250 are pure noise. In the thin-client branch (port 10250) the probe is intentionally left enabled so the same production code path that gates thin-client routing is exercised. Mirror the conditional in afterClass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 Review 2 feedback - QueryPlanRetriever: clarify thin-client / Gateway V1 routing comment; explain that Gateway V1 is pinned only when PartitionKeyDefinition is unavailable to convert proxy queryRanges to EPK hex ranges. - RxDocumentClientImpl + DocumentQueryExecutionContextFactory: plumb the resolved DocumentCollection into validateCustomQueryForReadManyByPartitionKeys -> fetchQueryPlanForValidation so the validation query-plan request is thin-client-eligible instead of forcibly pinned to Gateway V1. - EndpointProbeClient: extract DiagnosticsSnapshot and ProbeResult into top-level types (EndpointProbeDiagnosticsSnapshot, EndpointProbeResult); remove @SuppressWarnings("unused"); use the failure reason in RED-cycle logs and toString(); verbose NPE message in the constructor naming the required dependency and wiring entry point. - CHANGELOG: move thin-client entry under Features Added; one-line summary with HTTP/2 + gatewayMode requirement and PR link. - CosmosNotFoundTests: add forceHttp1IfGatewayMode helper applied to every fast-group client construction so Gateway-mode clients in the fast group use HTTP/1.1 and are guaranteed not to route through the proxy. Keep the COSMOS.THINCLIENT_PROBE_ENABLED=false set/clear for the thinclient group's defensive teardown. - PerPartitionCircuitBreakerE2ETests: drop the THINCLIENT_PROBE_ENABLED property mutation now that probe behavior is asserted only by tests that opt in explicitly. * Fix CI compile: import ConnectionPolicy in CosmosNotFoundTests * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Thread DocumentCollection into readMany query-plan validation readManyByPartitionKeys validates any caller-supplied custom query by fetching a query plan and asserting it is single-partition and non-hybrid. Until now that validation called fetchQueryPlanForValidation with no DocumentCollection, so the plan request was pinned to Gateway V1 (useGatewayMode = (partitionKeyDefinition == null)). Thread the container's DocumentCollection from RxDocumentClientImpl.validateCustomQueryForReadManyByPartitionKeys -> DocumentQueryExecutionContextFactory.fetchQueryPlanForValidation so QueryPlanRetriever has the PartitionKeyDefinition needed to convert PartitionKeyInternal-formatted queryRanges from the thin client (Gateway V2) proxy into the EPK-hex Range<String> entries the query pipeline consumes. With this wiring, the validation query plan goes to the thin client when the client is configured for it, and remains on Gateway V1 otherwise. No behavior change on the non-thin-client path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(thin-client): disable probe in CI fault-injection tests + honor kill switch in forceUnhealthy Stage 1990 of build 6432767 had 83 failures in ClientRetryPolicyE2ETestsWithGatewayV2#serviceUnavailableWithGatewayV2 because the probe gate added by PR #49437 flips routing from :10250 to :443 when CI test accounts lack /connectivity-probe (failureThreshold=1 trips proxyHealthy=false on the first failed probe and the assertion at TestSuiteBase#assertThinClientEndpointUsed then sees :443 and fails). Test fix (per CosmosNotFoundTests precedent, commit 3b392e3acf5): - ClientRetryPolicyE2ETestsWithGatewayV2: disable probe in @BeforeClass / clear in @AfterClass. - ThinClientTestBase: same lifecycle in enableThinClientForTest / clearThinClientForTest so all ThinClient*E2ETest subclasses inherit the guard. Source fix: honor COSMOS.THINCLIENT_PROBE_ENABLED kill switch inside EndpointProbeClient.forceUnhealthy(), closing the item-5 invariant gap (previously only runProbeCycle() checked the flag, so a mismatch-driven forceUnhealthy could mutate proxyHealthy even with the probe disabled). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(thin-client): kill switch bypasses probe-health gate When COSMOS.THINCLIENT_PROBE_ENABLED=false, no probe cycles run, so the probe-health signal is stale/meaningless. Treat probe as healthy in that case so the gate is driven solely by COSMOS.THINCLIENT_ENABLED (already folded into useThinClient at construction). This fixes regressions in CI test classes (GatewayReadConsistencyStrategyE2ETest, FaultInjectionWithAvailabilityStrategyTestsBase, ThinClientQueryE2ETest) where probe to /connectivity-probe fails on test accounts and flips routing to :443 instead of :10250. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(thin-client): disable probe in CI fault-injection / failover / consistency tests Adds COSMOS.THINCLIENT_PROBE_ENABLED=false in @BeforeClass (and clears in @AfterClass) for PerPartitionCircuitBreakerE2ETests, PerPartitionAutomaticFailoverE2ETests, FaultInjectionServerErrorRuleOnGatewayV2Tests, and GatewayReadConsistencyStrategyE2ETest. These tests run against CI accounts whose /connectivity-probe endpoint is not deployed. Without this, the first failed probe trips proxyHealthy=false and routing falls back to Gateway V1 (port 443), causing assertThinClientEndpointUsed (which expects port 10250) to fail. Locally validated PPAF (16 tests, 0 failures), GRCS (3 tests, 0 failures), FI (3 tests, 0 failures). PPCB still in progress at commit time; pushing to let local + CI run in parallel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client QueryPlan error returning statusCode 0 instead of 400 The thin-client/Gateway-V2 proxy returns a non-2xx response with a raw, non-JSON, NUL-padded error body for invalid-syntax queries. In RxGatewayStoreModel.validateOrThrow, new CosmosError(body) attempted to parse that body as JSON and threw IllegalArgumentException, which escaped the method before the existing status-carrying throw could run. Upstream then wrapped it as statusCode 0. Wrap the CosmosError(body) construction in a narrow try/catch (IllegalArgumentException) and fall back to the non-parsing CosmosError(errorCode, message) constructor with a sanitized body. The existing throw now fires with the correct status (400) and the proxy error text is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden ThinClientQueryE2ETest assertions (F1-F5) F1: strict 400 + unconditional thin-client endpoint check on invalid query, locking the statusCode-0->400 fix. F2: ordered-vs-sorted-set document-ID comparison (ORDER BY sequence-compared, others set-compared). F3: ID-set equality + no-duplicate check across drained continuation pages. F4: numeric-tolerance (1e-6) comparison for scalar and GROUP BY aggregates to avoid float-formatting false mismatches. F5: validated vector/full-text/hybrid queries match Direct vs thin-client end-to-end through the proxy. Validated live via -Pthinclient: 84 tests, 0 failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace thin-client test matrix with reverse-engineered E2E test spec Rewrite THINCLIENT_TEST_MATRIX.md as a reviewable test-design specification reverse-engineered from the committed ThinClientQueryE2ETest code (84 tests). Documents the differential-testing oracle (Direct :443 baseline vs thin client :10250 SUT), the data-model fixture, every assertion contract (endpoint provenance, ordered-vs-unordered ID equality, scalar/GROUP BY tolerance), the full 84-test matrix, the F1-F5 hardened special cases (continuation draining, invalid-query 400, vector/FTS/hybrid ranking, readMany validation path), advertised-feature coverage, and known gaps (CountIf/DCount/MultipleOrderBy) for reviewer sign-off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply Aditya Sarpotdar review feedback to thin-client E2E tests Executes actionable review feedback on ThinClientQueryE2ETest plus two harness fixes surfaced during live validation against thin-client-multi-region-ci. ThinClientQueryE2ETest: - Strict ordering: ORDER BY results validated with isStrictlyOrdered across the board (stricter-by-default, per reviewer guidance) instead of set parity. - Add testMultipleOrderBy() with a composite-index container to cover the MultipleOrderBy query feature. - Add testDCount() using the canonical Cosmos DCount idiom (COUNT over a DISTINCT VALUE subquery); SQL-standard COUNT(DISTINCT ...) is not valid Cosmos SQL grammar. - Reword multi-EPK-range comment/javadoc to reflect emulator/backend reality (multiple metadata ranges served by a single backend partition; SDK routing and query pipeline still exercised). Harness fixes: - TestSuiteBase: add wait-and-poll utility waitForCollectionToBeAvailableToRead (predicate on NotFound/substatus 1013) to deflake "Collection is not yet available for read" on freshly created containers; update call sites in OrderbyDocumentQueryTest, NonStreamingOrderByQueryVectorSearchTest, QueryValidationTests, ReadFeedCollectionsTest. - SinglePartitionDocumentQueryTest: make the processMessage Mockito assertion mode-aware (thin-client routes query plan + query => times(2)). Validated live against thin-client-multi-region-ci: 86/86 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Redesign thin-client probe to per-region probe-and-cache model Convert the global circuit-breaker (failure/recovery hysteresis re-probing all regions every refresh) into a per-region one-shot probe-and-cache: probe only the delta of unproven regions, cache successes across account refreshes, and never re-probe a proven region. The routing gate stays conservative (Gateway V1 until every known region is proven, Gateway V2 once all succeed). - Configs: drop failure/recovery threshold knobs; add THINCLIENT_PROBE_MAX_RETRIES (default 3) and isThinClientExplicitlyEnabled() bypass. - EndpointProbeClient: per-region cache, delta probing, retry-N, all-known-succeeded gate, forced-unhealthy latch; per-probe timeout from thin-client connection timeout. - EndpointProbeDiagnosticsSnapshot: expose knownRegionCount/succeededRegionCount. - RxDocumentClientImpl: gate probe wiring/routing on !isThinClientExplicitlyEnabled. - Tests: rewrite EndpointProbeClientTests; update ConfigsTests, ThinClientProbeWiringTests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: opt-in thin-client QueryPlan routing + kill-switch, stop advertising CountIf, remove PR-reference comments * Address review feedback: broaden gateway error fallback to ClassCastException, drop response ObjectNode mutation, fail-fast on empty queryRanges, align readMany routing test with opt-in * Make thin-client QueryPlan no-EPK-headers contract explicit in wrapInHttpRequest * Fix missing StandardCharsets import in RxGatewayStoreModelTest * Test: add QueryOracle-derived LIKE/scalar-expression thin-client parity tests; drain only first page in OperationPolicies readAllItems to avoid full-container enumeration timeout * Test: verify a cached proxy-generated query plan still executes correctly through the thin client * Test: cross-partition tests use a dedicated multi-physical-partition fixture and assert >1 partition key range contacted; enforce ordered full-row comparison when id is not projected; clearer naming * Add CHANGELOG entry for QueryPlan request routing to Gateway V2 * Call out Execute Stored Procedure support in CHANGELOG entry * Fix flaky region/timing-sensitive fault-injection E2E tests CustomerWorkflowPartitionLevelCircuitBreakerTest and CustomerWorkflowHighE2ETimeoutTest: bump the ThresholdBasedAvailabilityStrategy threshold from 100ms to 300ms so the cold primary-region connection can dispatch the request and record the injected fault before the availability-strategy hedge to a warm secondary region wins the race and cancels it (which previously yielded 0 fault hits). PerPartitionCircuitBreakerE2ETests: before asserting short-circuit routing, wait until the first preferred region is actually marked Unavailable, removing the race between the failure-counter estimate and the asynchronous routing exclusion. Scoped to single-faulty-region configs (expectedRegionCountWithFailures == 1) and run once so fault-in-all-regions configs (which can never short-circuit) do not hit the method timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove test AAD auth toggle; move thin-client test matrix to docs/ and align with ThinClientQueryE2ETest * Thin-client review: tri-state enablement/probe-decision + EndpointProbeClient cleanup - Configs: COSMOS.THINCLIENT_ENABLED is tri-state (null=unset default-on, TRUE=opt-in, FALSE=opt-out); consumers interpret null. Remove probe kill switch (COSMOS.THINCLIENT_PROBE_ENABLED) and probe-path config (path is fixed /connectivity-probe). - ThinClientConnectivityConfig (new): single authority for enablement + the shouldUseThinClientStoreModel routing gate, taking the nullable probe decision so null never collapses into a boolean clause at the call site. - GlobalEndpointManager.getProxyProbeDecision(): tri-state Boolean (null = no decision: probe not wired). - EndpointProbeClient: hard-code PROBE_PATH; remove cycleId correlation and unused EndpointProbeResult.reason; crispen runProbeCycle (single currentGate no-op Mono, removeIf delta). - Tests: repoint routing-gate tests to the 5-arg tri-state signature (+null/opt-in cases); remove THINCLIENT_PROBE_ENABLED usage from E2E tests; drop removed-config tests. * Consolidate probe state into single source-of-truth map; remove force-unhealthy latch and per-cycle copies - Replace volatile knownEndpoints Set + probeSucceeded CHM with one ConcurrentHashMap<URI,Boolean> (probeEndpointToProbeSuccessState): keys = current topology, value = proven/not-yet-proven. Gate is non-empty AND all TRUE. - Remove the force-unhealthy latch: an empty/null topology now empties the map via retainAll so the gate falls to false naturally (route to Gateway V1). - runProbeCycle reconciles and probes by iterating the provided Collection<URI> directly (reference only, no LinkedHashSet copies) and drops the delta/empty short-circuit fast paths; the Flux filters not-yet-proven endpoints inline. - applyCycleResult uses replace() so a region pruned by a concurrent reconcile is never resurrected. Unit suite green: 2579 tests, 0 failures. * Add cross-partition aggregate / GROUP BY parity tests for thin-client query Adds 7 thin-client E2E tests covering cross-partition aggregate merge (COUNT/SUM/AVG/MIN/MAX) and GROUP BY, plus two helpers that assert Direct vs thin-client value parity and verify the query genuinely fanned out (distinctPartitionKeyRangesContacted > 1), exercising the cross-partition MERGE path the single-partition aggregate tests miss. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix EPK over-span; add QueryPlan parity tests For a partial (prefix) hierarchical partition key, the thin-client / Gateway V2 path previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter, so the proxy resolved the request to the owning physical partition and returned every co-located document (e.g. 18 instead of 6). ThinClientStoreModel.wrapInHttpRequest now sets READ_FEED_KEY_TYPE= EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from() so RntbdRequestHeaders.addStartAndEndKeys serializes the prefix EPK sub-range as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. Adds ThinClientQueryE2ETest coverage: QueryPlan parity across sources and a hierarchical prefix half-open range test asserting thin-client matches Direct TCP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix partition key EPK over-span For a partial (prefix) hierarchical (MULTI_HASH) partition key, the thin-client (Gateway V2) store model previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter. The proxy therefore resolved the request to the owning physical partition and returned every co-located document (an over-span) instead of only those matching the prefix. ThinClientStoreModel.wrapInHttpRequest now detects a prefix MULTI_HASH key and sets READ_FEED_KEY_TYPE=EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from(), so RntbdRequestHeaders serializes the prefix EPK sub-range [hash(prefix), hash(prefix) + "FF") as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. QueryPlan requests and full (non-prefix) keys are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thin-client query E2E parity regression tests Ports ThinClientQueryE2ETest and its ThinClientTestBase helper. The suite self-baselines every query against a Direct (RNTBD) client and the thin-client (:10250 HTTP/2) path, asserting identical results. This directly guards the MULTI_HASH prefix partition-key EPK over-span fix: prefix (single-component) hierarchical-partition-key queries must return only the narrow matching set, matching Direct, rather than over-spanning to all co-located documents. Both files are runtime self-enabling (enableThinClientForTest() + builder-driven HTTP/2 via setEnabled(true)); no module property or TestSuiteBase changes needed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reuse prefix EPK range on thin-client parallel prefix-query path The earlier fix (499f928) only scoped the prefix MULTI_HASH read when the request carried a non-null partition key. The parallel prefix-query path intentionally nulls the partial hierarchical partition key and instead carries the narrow prefix effective-partition-key range on the request's feedRange, so that guard never fired there and the thin-client read still over-spanned to the whole physical partition. Mark such requests (prefixPartitionKeyQuery) in ParallelDocumentQueryExecutionContextBase and, in ThinClientStoreModel, reuse the already-computed FeedRangeEpkImpl range as the doc-level EPK filter (START_EPK/END_EPK + StartEpkHash/EndEpkHash) instead of recomputing getEPKRangeForPrefixPartitionKey from a partition key we no longer have. This mirrors the Direct/GatewayV1 decision points and honors DRY/SRP. Also relax assertThinClientEndpointUsed so an all-QueryPlan diagnostics context passes: in thin-client mode QueryPlan calls resolve via the classic gateway and are the only requests permitted on a non-thin-client endpoint; any non-QueryPlan (data) request on a non-thin-client endpoint still fails the assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Consolidate thin-client prefix EPK RNTBD headers, propagate prefix flag on clone, add readAllItems/readMany prefix HPK tests ThinClientStoreModel.wrapInHttpRequest now sets all five EPK-specific RNTBD headers (ReadFeedKeyType, StartEpk/EndEpk, StartEpkHash/EndEpkHash) in a single block directly on the RNTBD request, replacing the split request-header-map + post-construction approach. Wire encoding is unchanged (hex-string UTF-8 bytes for StartEpk/EndEpk, decoded hash bytes for the hash headers). RxDocumentServiceRequest.clone() now copies prefixPartitionKeyQuery so hedged/availability-strategy/retry clones keep the prefix signal and do not over-span. ThinClientQueryE2ETest adds prefix-HPK regression coverage for readAllItems (ReadFeed) and readManyByPartitionKeys, asserting Direct-vs-thin-client parity and per-tenant scoping (no over-span). * Simplify prefix-query flag assignment in ParallelDocumentQueryExecutionContextBase Assign isPrefixPartitionKeyQuery directly from isPartialPartitionKeyQuery(...) and guard only the PARTITION_KEY-setting branch with !isPrefixPartitionKeyQuery, removing the if/else. Behavior is unchanged: a partial (prefix) key still leaves partitionKeyInternal null so the prefix FeedRangeEpkImpl survives createDocumentServiceRequestWithFeedRange. * Add CHANGELOG entry for thin-client prefix hierarchical partition key over-span fix * Minimize prefix-query flag change vs main in ParallelDocumentQueryExecutionContextBase Keep main's single 'if (!isPartialPartitionKeyQuery)' structure; only capture the predicate in isPrefixPartitionKeyQuery and pass it to request.setPrefixPartitionKeyQuery. Removes the if/else. * Order thin-client prefix detection to prefer the cached feed-range path In ThinClientStoreModel.wrapInHttpRequest, check the cached prefix feed-range path (isPrefixPartitionKeyQuery + FeedRangeEpkImpl) first and fall back to recomputing from the partition key only for the PK-bearing prefix case. Behavior is unchanged (the two cases are mutually exclusive). * Address Copilot review: validate all requests in thin-client endpoint assertion TestSuiteBase.assertThinClientEndpointUsed now validates every request instead of early-returning on the first thin-client match, so a mixed scenario (some data requests via the classic gateway) fails; QueryPlan requests remain exempt and null endpoints are handled. ThinClientTestBase.assertThinClientEndpointUsed delegates to the shared TestSuiteBase implementation, removing duplicated logic that could NPE on a null endpoint and reject valid QueryPlan-via-gateway scenarios. * Simplify thin-client prefix EPK handling to reuse HTTP EPK headers Replace the explicit MULTI_HASH prefix-detection and pre-serialization range computation with a header-presence branch: when the request already carries START_EPK/END_EPK HTTP headers (populated upstream by FeedRangeEpkImpl for prefix hierarchical partition keys and explicit EPK-range reads), set only the additive StartEpkHash/EndEpkHash RNTBD tokens. The ReadFeedKeyType/StartEpk/ EndEpk string tokens are copied verbatim from those HTTP headers by RntbdRequestHeaders#addStartAndEndKeys during RntbdRequest.from(), so they no longer need to be set here. Removes now-unused imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove dead prefix-query flag and expand HPK thin-client test coverage Following the simplification of the prefix EPK fix to reuse the existing StartEpk/EndEpk HTTP headers (a56db9ad44c), the RxDocumentServiceRequest prefixPartitionKeyQuery flag is no longer read anywhere. Remove the field, its accessors, the clone copy, and the caller assignment in ParallelDocumentQueryExecutionContextBase. Test changes: - Retrofit CosmosMultiHashTest into the thinclient TestNG group so the MULTI_HASH prefix over-span coverage also runs against GatewayV2 (proxy :10250) over HTTP/2, not just the emulator gateway. - ThinClientQueryE2ETest: correct the prefix-scope Javadoc to describe the actual header-copy behavior and add a deterministic prefix-partition-key readAllItems regression. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert ParallelDocumentQueryExecutionContextBase to main The prefix-query flag on RxDocumentServiceRequest was removed in 03a7c672b95 after the fix was simplified to reuse the StartEpk/EndEpk HTTP headers. The only functional edit in this file (the setter call) went away with it, leaving behind unnecessary refactor noise. Restore the file to its upstream/main state so the hotfix touches no query execution code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore thin-client-probe symbols dropped during merge of #47759 The merge of AzCosmos_GatewayV2_QueryPlanSupport (#47759) dropped probe's thin-client-probe supporting code while keeping the method bodies that reference it, breaking compilation: - GlobalEndpointManager: restore imports (http.HttpClient, java.util.Set) and the thinClientProbeClient AtomicReference field. - LocationCache: restore getThinClientRegionalEndpoints() and the collectThinClientEndpointsAllOrNothing() helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cosmos): allowlist subStatus 1003 in thinclient readiness-probe retry The @BeforeSuite container-readiness probe issues a QueryPlan against a freshly created container. With thin-client default-on, a fresh container in a multi-region account can transiently return 404/subStatus 1003 (OWNER_RESOURCE_NOT_EXISTS) during regional metadata propagation. The classifier did not allowlist 1003, so the probe treated it as non-retryable and cascade-skipped the entire thinclient suite. Add OWNER_RESOURCE_NOT_EXISTS to the NOTFOUND retryable allowlist in isRetryableCollectionReadinessFailure so the probe retries through propagation and the suite runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cosmos-benchmark): parameterize SDK version + add RunId/EndpointFlavor cold-start knobs Instrument azure-cosmos-benchmark for the cold-start Create-latency matrix driving the thin-client probe (PR #49437): - pom: parameterize azure-cosmos + azure-cosmos-encryption versions (-Dazure-cosmos.version=...) so GA 4.79/4.80/4.81 and the probe build can be benchmarked from one harness. - Common tags (RunId, SdkVersion, ConnectionMode, EndpointFlavor, Phase) applied to every metric so a shared Grafana dashboard can slice runs. - endpointFlavor knob (ComputeGateway|ThinClient) maps to COSMOS.THINCLIENT_ENABLED at client build to drill both the Gateway V1 and Gateway V2 (thin-client) HTTP/2 paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce QueryPlan thin-client routing invariant and harden container-readiness probe for thin-client-default-on - Remove the QueryPlan exemption in assertThinClientEndpointUsed: when ThinClient is opted in with HTTP/2 enabled, QueryPlan calls MUST also route to Gateway V2 (:10250), matching production default DEFAULT_THINCLIENT_QUERY_PLAN_ENABLED=true. - Restore public static visibility on ThinClientTestBase.assertThinClientEndpointUsed / assertGatewayEndpointUsed so cross-package static imports (CosmosMultiHashTest) compile. - Add OWNER_RESOURCE_NOT_EXISTS (subStatus 1003) to the retryable NOTFOUND allowlist in TestSuiteBase.isRetryableCollectionReadinessFailure. With thin-client default-on the beforeSuite readability probe (a QueryPlan) routes through GW V2; a freshly-created container not yet propagated to a secondary region returns 404/1003, which must be treated as a transient readiness failure instead of cascade-skipping the suite. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add kill-switch flip validation for QueryPlan thin-client routing Validate the COSMOS.THINCLIENT_QUERY_PLAN_ENABLED sysprop/env kill-switch in both directions: - Unit: new case in ReadManyByPartitionKeyQueryPlanRoutingTest asserts QueryPlan is forced to Gateway V1 (useGatewayMode=true) when the flag is disabled, even with thin-client eligible. - E2E: make TestSuiteBase.assertThinClientEndpointUsed flag-aware so QueryPlan endpoint assertions flip based on the live-read flag while data operations continue to route to the thin-client endpoint. Both scenarios verified green (unit 3/3; E2E 112/112, oracle:true for flag ON and flag OFF). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore isAnyRegionShortCircuitedForPartition helper dropped during #47759 merge The #47759 merge conflict resolution kept HEAD's short-circuit-wait loop (which calls isAnyRegionShortCircuitedForPartition) but dropped the method definition. Recovered from probe tip 42c2c7d733f and re-inserted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove azure-cosmos-benchmark and HTTP/2 ping-handler changes (moved to #49725) These changes belong in PR #49725, not the endpoint-probe PR #49437. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Run thin-client probe cycle on every account refresh (delta-gated) Wire runThinClientProbeCycleMono() into the account-refresh funnel in GlobalEndpointManager so the connectivity probe runs on every refresh, delta-gated to new/unproven regions in EndpointProbeClient. Previously the probe driver had no production callers, leaving the default-on thin-client routing gate permanently false. Add a GEM delta-refresh unit test proving a subsequent refresh probes only newly-added regions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thin-client endpoint-probe (implicit path) E2E test lane + GEM probe wiring Wire runThinClientProbeCycleMono() into GlobalEndpointManager.refreshLocationPrivateAsync so the connectivity probe runs on every account refresh: as a prefix when a fresh databaseAccount is present, and after the foreground database-account read (.map -> .flatMap). Add an implicit-path test lane that leaves COSMOS.THINCLIENT_ENABLED unset (thin-client default-on, routing gated on the probe verdict with Gateway V1 fallback) while keeping HTTP2_ENABLED true: - new 'thinclient-endpoint-probe' maven profile (azure-cosmos-tests/pom.xml) - new Cosmos_Live_Test_ThinClient_EndpointProbe CI stage (tests.yml) - live-thinclient-endpoint-probe-platform-matrix.json - thinclient-endpoint-probe-testng.xml suite - register thinclientEndpointProbe group in TestSuiteBase before/after suite - E2E probe tests: point-operation, query, change-feed, stored-procedure + shared base Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align query/* and test files with Azure/main to reduce PR diff Reset QueryPlanRetriever, PartitionedQueryExecutionInfo and 9 test files to Azure/main. These carried Gateway V2 QueryPlan work (partitionKeyDefinition guard, CountIf, queryRanges removal) and parity-test adaptations that overlap upstream #47759, which Azure/main already owns. Scoping this PR to the thin-client connectivity-probe flow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove isThinClientEnabled() helper; require positive probe verdict to route thin-client Collapse all call sites onto the tri-state reader Configs.isThinClientEnabledExplicitly() and remove the redefined isThinClientEnabled() helper, which had flipped the unset default from false to true. Thin-client routing now requires an explicit opt-in or an affirmative connectivity-probe verdict; an unset flag no longer routes to Gateway V2 by default (null/FALSE probe verdict -> Gateway V1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Collapse thin-client enablement to single nullable isThinClientEnabled() Rename Configs.isThinClientEnabledExplicitly() to isThinClientEnabled() (nullable Boolean) and remove the hasUserExplicitlyEnabledThinClient() and ThinClientConnectivityConfig.isExplicitThinClientOptIn() helpers. The routing gate shouldUseThinClientStoreModel now takes the tri-state Boolean directly: a non-null value is a hard contract (TRUE routes to Gateway V2, FALSE pins to V1), and null falls back to the connectivity-probe verdict. This eliminates the scattered Boolean.equals opt-in checks. Default remains null (not default-on). Updated ThinClientRoutingGateTests implicit-path cases to pass null for the tri-state arg (probe-gated) instead of false. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Consolidate thin-client probe E2E tests into shared classes via dual TestNG group Tag the existing ThinClient*E2ETest classes (and ThinClientTestBase) with both the `thinclient` and `thinclientEndpointProbe` groups so the same test bodies run in either CI lane against a different ambient enablement path: - thinclient lane: -DCOSMOS.THINCLIENT_ENABLED=true (explicit opt-in) - thinclient-endpoint-probe lane: flag unset -> endpoint connectivity probe drives routing Delete the 5 duplicate ThinClientEndpointProbe* copies; the probe suite XML discovers by group across com.azure.cosmos.*, so it auto-picks up the retagged classes with zero duplication. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thinclientEndpointProbe group to shared suite setup/teardown The consolidated thin-client E2E tests are dual-tagged with the thinclientEndpointProbe group, but the shared @BeforeSuite/@AfterSuite in TestSuiteBase (which creates and deletes the SHARED_* collections) was still gated only on the thinclient group. Under the probe lane's group filter the suite setup never ran, leaving the shared collections null and causing an NPE in cleanUpContainer during @BeforeClass. Adding the group makes both lanes green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix ClientConfigDiagnosticsTest stale gwV2Cto assertions Update the gw connCfg assertions to expect gwV2Cto:PT5S instead of gwV2Cto:n/a, matching the intentional diagnostics change where the thin-client (gateway V2) connect timeout is emitted by default when COSMOS.THINCLIENT_ENABLED is unset. This also resolves the cascading sessionRetryOptionsInDiagnostics failure caused by full() leaking a system property when its assertion threw before cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Update UserAgent tests to expect |F4 ThinClient feature flag ThinClient is default-enabled when COSMOS.THINCLIENT_ENABLED is unset on this branch, so the UserAgent suffix now carries the |F4 feature flag. Update the stale assertions to mirror RxDocumentClientImpl.addUserAgentSuffix dynamically (ThinClient default-on + Http2/Http2PingHealth) instead of hardcoding the suffix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align endpoint probe cache to .NET add-only semantics Refactor EndpointProbeClient to an add-only proven-healthy endpoint cache, matching the .NET implementation (PR azure-cosmos-dotnet-v3#5970). Once an endpoint is proven ThinClient-routable it is never pruned; the routing gate is evaluated against a volatile snapshot of the current topology so a transiently vanished proven region does not gate current routing and is not re-probed when it re-appears. Probe cycles only probe the delta of new, not-yet-proven endpoints on each account refresh. Add unit tests covering vanish/re-appear and unsupported-new-region scenarios (region added without support blocks the gate; removing it restores routing). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix CosmosDiagnosticsTest for default-on ThinClient |F4 user-agent suffix Fold ThinClient into generateHttp2OptedInUserAgentIfRequired so the helper dynamically mirrors production feature-flag suffixing, and remove a double-wrap of directClientUserAgent (already suffixed at construction). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Gate thin-client endpoint assertion on explicit opt-in only On the implicit/probe path (COSMOS.THINCLIENT_ENABLED unset), thin-client routing depends on the endpoint probe greenlighting all current regions. Under fault injection the probe is not guaranteed to converge, so requests may legitimately fall back to Gateway V1 (:443). Assert thin-client routing only when thin-client is explicitly opted in (THINCLIENT_ENABLED=true), the sole config where routing is deterministic because the probe is bypassed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review comments: tri-state THINCLIENT_ENABLED parsing, probe-client close, fire-and-forget probe cycle, single-flight tests - Configs: parse COSMOS.THINCLIENT_ENABLED via explicit true/false whitelist (parseTriStateThinClientEnabled); any other value logs a warning and is treated as unset (null -> probe-gated) instead of Boolean.parseBoolean silently collapsing every non-"true" string to a hard opt-out. - GlobalEndpointManager: close() now cancels the in-flight probe-cycle subscription and closes the EndpointProbeClient so a stale cycle drops its result. Probe cycle is fired fire-and-forget on force-refresh and topology refresh so init()/cross-region retry never block on the probe; routing stays on Gateway V1 until the probe proves the proxy endpoints. - Tests: add ConfigsTests.thinClientEnabledInvalidValueTreatedAsUnset and two EndpointProbeClientTests covering single-flight overlap skip and closed-mid-cycle result-drop (gate stays conservative). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refactoring. * Fix SinglePartitionDocumentQueryTest stale thin-client routing predicate useThinClient() now reflects config-eligibility only; the test must gate on actual per-request routing (read locations + probe/opt-in) to match where the query lands. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address PR review: consolidate probe-wiring gate on canThinClientBeUsed() - Gate probe wiring in RxDocumentClientImpl on canThinClientBeUsed() so an explicit THINCLIENT_ENABLED=true still wires the probe (routing bypasses it), covering true->null runtime transitions. Removed dead canThinClientBeImplicitlyEnabled() from ThinClientConnectivityConfig and updated javadoc/comments/pom p…

…low-up to Azure#49437) (Azure#49796) * Route Execute Stored Procedure requests to Thin Proxy endpoint. * Route Execute Stored Procedure and QueryPlan requests to Thin Proxy endpoint. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC + GW Mode) cases. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Obtain List<Range<String>> from List<PartitionKeyInternal>. * Adding query + thin-client tests. * Fixing tests. * Fixing tests. * Fixing tests. * Addressing review comments. * Addressing review comments. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Refactor thin-client E2E tests based on operation type. * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add change feed tests for FeedRange.forFullRange and forLogicalPartition Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add thin client E2E test matrix documentation for QueryPlan review Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add SupportedQueryFeatures and QueryVersion RNTBD request headers for QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix testGetCurrentDateTime flaky assertion, add AAD auth support, RNTBD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor thin client query tests for reliability - Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix container leaks and Direct TCP AAD auth in tests - Fix container leak: get container reference before createContainer() so finally block can always delete. Use safeDeleteContainer() helper. - Fix Direct TCP client: apply AAD credential via applyCredential() for accounts with disableLocalAuth=true. All 89 thin client tests pass (80 query + 3 change feed + 3 point ops + 3 sprocs). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use bulkDelete in AfterClass for seeded docs cleanup Replace sequential deleteItem loop with executeBulkOperations for faster and more reliable cleanup of seeded test documents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback 1. Remove rntbd.instructions.md (not part of PR) 2. Move MAPPER variable to top of PartitionKeyInternalTest 3. Remove COSMOS.REUSE_DATABASE_ID (beforeSuiteReuse, afterSuite guard, pom.xml) 4. Keep System.setProperty in ThinClientQueryE2ETest (required — extends TestSuiteBase not ThinClientTestBase, property must be set before client build) 5. Fix RNTBD IDs to match server-side RntbdConstants.cs (ADO PR 1982503): SupportedQueryFeatures=0x00FF (String), QueryVersion=0x0100 (SmallString) The SmallString type was the root cause of 400 BadRequest — String uses 2-byte length prefix, SmallString uses 1-byte, causing token stream misparse. All 89 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align vector/FTS/hybrid tests with existing patterns - Add euclidean VectorDistance variant to vector search test - Add document ID comparison to FTS and hybrid tests (was count-only) - Add testFullTextScoreRanking: ORDER BY RANK FullTextScore with exact order comparison between Direct and thin client - All assertions now validate both result size AND content parity 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments + fix broken link Code fixes: - PartitionKeyInternalHelper: materialize fieldNames() Iterator in error msg - QueryPlanRetriever: throw IllegalStateException when partitionKeyDef null in thin client mode (was silent fallthrough) - RxDocumentClientImpl: add parentheses around compound && in boolean exprs - ThinClientStoreModel: use != instead of !(==) - ThinClientTestBase: remove stale 'uncomment' comment - ThinClientQueryE2ETest: accept status 0 (transport rejection) for invalid query — proxy returns transport error, not HTTP 400 - THINCLIENT_TEST_MATRIX.md: remove broken link, update methodology to reflect Direct TCP baseline Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review agent comments (round 2) - RntbdOperationType: add QueryPlan case to fromId()/fromType() lookups - PartitionKeyInternalHelper: sanitize PII from error message (log node type instead of raw partition key values) - THINCLIENT_TEST_MATRIX.md: remove PR-specific references and known blockers - PartitionedQueryExecutionInfo: remove stale queryRanges from backing ObjectNode after EPK conversion to prevent data inconsistency - QueryPlanRetriever: add TODO for 3 missing query features vs .NET SDK (ListAndSetAggregate, CountIf, HybridSearchSkipOrderByRewrite) 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor queryRanges deserialization — detect format from response, no ObjectNode mutation - PartitionedQueryExecutionInfo: getQueryRanges() now inspects the first range's 'min' field to detect format (string=EPK hex, array=PK-internal) and applies the correct deserialization path automatically. - Remove ObjectNode.remove('queryRanges') — no longer mutate the backing JSON. The format is detected lazily on first access. - 3-arg constructor now takes PartitionKeyDefinition (not pre-computed ranges) so conversion happens inside getQueryRanges(), keeping the DTO self-contained. - QueryPlanRetriever: pass partitionKeyDefinition to constructor instead of pre-converting ranges externally. 90/90 thin client tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use agnostic QueryRangesFormat hint instead of PartitionKeyDefinition presence Introduce QueryRangesFormat enum (EPK_HEX_STRING, PARTITION_KEY_INTERNAL_ARRAY) as the deserialization hint. The hint guides which path to try first, but getQueryRanges() always validates by inspecting the actual min node type — if the hint doesn't match the wire format, it falls through to the other path. Naming is transport-agnostic (no Proxy/ThinClient/Gateway references). PartitionKeyDefinition is only stored when needed for EPK conversion, not used as a format signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fetchQueryPlanForValidation caller after merge Pass null for the DocumentCollection parameter so the public fetchQueryPlanForValidation entry point compiles against the new fetchQueryPlan signature introduced during the merge with upstream/main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make PartitionedQueryExecutionInfo(ObjectNode, RequestTimeline) ctor public The upstream ReadManyByPartitionKeyQueryPlanValidationTest (added in #48801) constructs PartitionedQueryExecutionInfo directly from a test package. Upstream/main exposed this ctor as public; restore that visibility so the new test compiles after the merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(cosmos): HTTP/2 PING handler must observe child-stream read activity, not just parent-pipeline frames Follow-up to #49095. Root cause: Http2PingHandler is installed on the parent (TCP) channel and only updated lastReadActivityNanos from parent-pipeline channelRead -- which sees PING ACK / SETTINGS / GOAWAY but NOT HEADERS / DATA frames. With reactor-netty's Http2MultiplexHandler in the parent pipeline, all application H2 traffic is demuxed onto per-stream child channels, so a connection serving thousands of QPS still looked idle to the PING handler, causing unnecessary PINGs and measurable P95 overhead under load. Fix: - New package-private AttributeKey<AtomicLong> LAST_CHILD_STREAM_READ_ACTIVITY_NANOS seeded on the parent channel in handlerAdded. - Http2PingCloseRewrapHandler (already installed in every child stream's pipeline) overrides channelReadComplete to stamp the parent attribute via accumulateAndGet(now, Math::max) -- once per read cycle, monotonic. - Http2PingHandler.maybeSendPing now uses effectiveLastReadActivityNanos = max(field, child attr) for both the idle-threshold check AND the outstanding-PING early-return: if child-stream reads arrived after an outstanding PING, clear pingOutstandingSinceNanos and reset consecutiveFailures (defense against middleboxes that drop PING-ACK but pass application H2 frames). - Names use lastReadActivity / read activity -- both signals are inbound reads, never writes. Tests: two new unit tests in Http2PingHandlerTest covering the suppression path and the outstanding-PING reset path; both bypass real time via reflection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * perf: install Http2PingCloseRewrapHandler in doOnConnected, not doOnRequest The previous .doOnRequest() chained onto the HTTP/2 HttpClient builder wrapped every outgoing request's Mono in a Reactor operator. That paid per-request costs even when the rewrap handler was already installed: * MonoPeekTerminal wrap + per-request subscriber allocation (InternalMonoOperator.subscribe +2.60% in alloc profile) * per-request BiConsumer lambda invocation (Http2Pool\$\...run +1.16% in alloc profile) * per-request channel pipeline String-key walk via pipeline.get(...) Long-run async-profiler diff vs v4.80.0 on D2 attributed the PR's -7.6% QPS / +14.5% mean / +166% tail-max regression primarily to a +10.2% allocation rate -- with the operators above dominating the delta. Move the install into the existing .doOnConnected() block, which already runs once per HTTP/2 child stream (the customHeaderCleaner install right above it relies on the same contract). The rewrap handler is @Sharable, so installing once per stream is correct and amortizes the install + pipeline-walk away from the hot path. The ch.parent() != null guard is kept so the install only targets child streams, never the parent TCP channel (which uses Http2ParentChannelExceptionHandler instead). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [Cosmos] Default thin-client to enabled and add HTTP/2 connectivity-probe gate Adds an EndpointOrchestrator that fans out POST /connectivity-probe to every thin-client regional endpoint after each topology refresh. SDK only routes data-plane traffic to thin-client (Gateway V2) when all regional probes succeed across N consecutive refresh cycles (configurable via COSMOS.THINCLIENT_PROBE_FAILURE_THRESHOLD, default 2); otherwise traffic falls back to Gateway V1 at the next refresh boundary. No mid-flight fallback. Caveats: - Probe wiring is skipped entirely for Direct mode and when HTTP/2 is not configured; controlled by RxDocumentClientImpl.useThinClient. - QueryPlan, metadata reads, and AllVersionsAndDeletes change feed continue to route through Compute Gateway (Gateway V1). - Probe failures are absorbed inside the orchestrator and the trigger is fire-and-forget on the GEM scheduler, so probe issues can never trip CosmosClient initialization or fail a topology refresh. - EndpointOrchestrator implements Closeable and is closed from GlobalEndpointManager.close() so no further probes are issued after client shutdown. THINCLIENT_ENABLED now defaults to true; opt out via COSMOS.THINCLIENT_ENABLED=false or COSMOS.THINCLIENT_PROBE_ENABLED=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review comments: probe single-flight, body-drain lifecycle, cancellable in-flight probes - EndpointOrchestrator: fold body-drain into probe Mono via flatMap+then(perProbeTimeout) so a slow/hanging response body cannot leak resources outside the cycle budget (Copilot #1, deep-review #3, jeet HIGH-2 minor). - EndpointOrchestrator: add single-flight CAS (cycleInProgress) plus monotonic cycle id; closed-check inside applyCycleResult drops late results so a post-close cycle cannot mutate health state (deep-review #1+#2, jeet HIGH-2). - EndpointOrchestrator: re-evaluate closed/feature-flag/endpoints at subscription time via Mono.defer so GEM.close() cancellation is honored before any HTTP I/O is issued. - GlobalEndpointManager: retain probe Disposable in AtomicReference; close() now disposes the in-flight probe subscription so probe work cannot outlive the GEM/CosmosClient (Copilot #2, deep-review #2). - CHANGELOG: moved entry to unreleased 4.82.0-beta.1, reworded to honestly describe optimistic startup, N=2 RED-to-fallback hysteresis, and Direct-mode/metadata exclusion (Copilot #3, deep-review #4). Tests: 45 unit tests pass (EndpointOrchestratorTests + ConfigsTests + ThinClientProbeWiringTests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 second-batch review feedback - LocationCache.getThinClientRegionalEndpoints now walks both read and write region endpoint maps so single-master write-region failures still flip the probe gate. - EndpointOrchestrator.forceUnhealthy(reason) provides a non-HTTP path to flip the gate; GlobalEndpointManager calls it when topology says thin-client is eligible but no regional endpoint resolves. - Symmetric hysteresis: new COSMOS.THINCLIENT_PROBE_RECOVERY_THRESHOLD (default 1) so operators can require N consecutive GREEN cycles before flipping back to proxy. - Extracted RxDocumentClientImpl.useThinClientStoreModel(...) body into package-private static shouldUseThinClientStoreModel for direct unit testability; added ThinClientRoutingGateTests covering 9 routing paths. - EndpointOrchestratorTests.stubResponse now returns Mono.empty() to avoid Unpooled.EMPTY_BUFFER refCnt underflow across multiple probe calls. - Removed unused locals; added recoveryThresholdRequiresMultipleGreenCycles, forceUnhealthy_flipsGateToRedWithoutRunningProbe, forceUnhealthy_onClosedOrchestrator_isNoOp tests. All 57 unit tests in the touched files pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify CHANGELOG: probe recovery threshold is now configurable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review (rounds 4-8): reactor-chain probe, FQN cleanup, fix gwV2Cto and ThinClient user-agent assertions - GlobalEndpointManager: convert thin-client probe trigger to a Mono<Void> chained into the topology-refresh reactor pipeline (replaces fire-and-forget subscribe). Removes thinClientProbeDisposable field and its close() handling since cancellation now propagates through the outer subscription. - EndpointProbeClient/EndpointProbeClientTests/ThinClientProbeWiringTests: replace inline FQNs with imports (java.io.Closeable, java.util.List, java.net.ConnectException, com.azure.cosmos.implementation.http.HttpHeaders). - ClientConfigDiagnosticsTest: compute gwV2Cto dynamically from Configs.isThinClientEnabled() so assertions remain valid after the default flip to true. - ConfigsTests: update default-threshold assertions from 2 to 1 to match DEFAULT_THINCLIENT_PROBE_FAILURE_THRESHOLD=1. - UserAgentContainerTest.UserAgentIntegration: expect '|F4' suffix because the ThinClient feature flag (1 << 2) is now included by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI test fallout from default ThinClient enablement UserAgentSuffixTest.validateUserAgentSuffix and CosmosDiagnosticsTest.generateHttp2OptedInUserAgentIfRequired: include UserAgentFeatureFlags.ThinClient in computed |F<hex> suffix when COSMOS.THINCLIENT_ENABLED is true (now default after Gateway V2 default enablement). Mirrors RxDocumentClientImpl.addUserAgentSuffix + UserAgentContainer.setFeatureEnabledFlagsAsSuffix behavior. SinglePartitionDocumentQueryTest.querySinglePartitionDocuments: spy on both gateway-proxy and thin-proxy and assert exactly one invocation. Previous code only spied on the proxy implied by useThinClient() config intent, which races with the probe-healthy gate -- routing AND's intent with isProxyProbeHealthy() so on first cycle the request may go through gateway even when thin-client is configured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe in Http2PingKeepaliveTest The test installs an iptables DROP on thin-client port 10250 to verify that Http2PingHandler closes the broken connection after consecutive PING ACK timeouts and the recovery request uses a new connection on the same regional endpoint. After default Gateway V2 enablement, the connectivity probe also fires HTTP/2 POSTs to port 10250 on every account refresh. With iptables dropping that port, the probe trips proxyHealthy=false, useThinClient StoreModel() returns false, and the data plane request routes through Gateway V1 on port 443 -- which iptables is not dropping. Result: the PING handler never fires, the warm-up and recovery requests use the same gateway channel, and the assertion 'recovery channel must differ from initial' fails (both ended up as 77af2e47 on build 6419227). Set COSMOS.THINCLIENT_PROBE_ENABLED=false in beforeClass so the probe short-circuits to a no-op, EndpointProbeClient.proxyHealthy stays optimistically true, and the data plane request actually flows over port 10250 where the iptables DROP can take effect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Disable thin-client probe by default for E2E tests in TestSuiteBase Build 6424287 surfaced two new failure patterns: 1. CosmosNotFoundTests.performBulkOnDeletedContainerWithGatewayV2 (45 failures) - asserts substatus 1003 from the thin-client routing path, but observed 0 because the data plane was routed to Gateway V1 instead of the proxy. 2. PerPartitionCircuitBreakerE2ETests.*Gateway (26 failures) - TestSuiteBase.assertThinClientEndpointUsed could not find any request whose endpoint contained ':10250/', i.e. nothing actually went to the thin-client proxy. Both patterns trace to the same source: the new connectivity probe is enabled by default, the proxy-side /connectivity-probe endpoint is not deployed in every CI test account yet, and the default failure threshold is 1. So after the first probe cycle the SDK marks the proxy unhealthy and routes data plane traffic to Gateway V1, which breaks tests that explicitly assert thin-client routing. Disable the probe by default in TestSuiteBase's static initializer (only when the property is not already set), so all E2E tests inherit deterministic, configuration-driven routing. Tests that exercise the probe itself (EndpointProbeClientTests, ThinClientProbeWiringTests) set the property explicitly in @BeforeMethod and are not affected. Also drop the now-redundant per-class override in Http2PingKeepaliveTest - the base class disables it, and the test's @AfterClass clear would otherwise re-enable the probe for any subsequent E2E test sharing the JVM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Disable thin-client probe by default for E2E tests in TestSuiteBase" This reverts commit ad9e3dff1db97df0d53a6f92aaec709676caff19. * Add per-class thinclient probe disable in CosmosNotFoundTests and PerPartitionCircuitBreakerE2ETests Companion to the prior revert. The revert undid the global TestSuiteBase probe disable (which masked production behaviour). This commit adds the necessary per-class disable to the two test classes whose assertions explicitly require thinclient routing: CosmosNotFoundTests (thinclient group) and PerPartitionCircuitBreakerE2ETests (fi-thinclient-multi-master group). Both clear the property in their @AfterClass. Http2PingKeepaliveTest already has its own disable (restored by the revert). Production callers continue to get the connectivity probe ON by default with the production failure threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add unit tests for QueryPlan and stored-procedure thin-client routing Covers 5 new scenarios in ThinClientRoutingGateTests: - ExecuteStoredProcedure on a StoredProcedure resource routes to thin client - Non-execute StoredProcedure ops (Create) route to Gateway V1 - OperationType.QueryPlan routes to thin client - QueryPlan returns false when probe is unhealthy - ExecuteStoredProcedure returns false when probe is unhealthy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove endpoint-probe content from QueryPlan PR branch Reverts the merge of jeet1995/thin-client-probe-flow that was brought into this PR earlier, while keeping the branch up-to-date with upstream/main. Net diff vs upstream/main is now only the QueryPlan and StoredProcedure Gateway V2 routing changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Propagate hybrid query diagnostics Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Advertise CountIf query feature Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address query plan review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Share thin client test property setup Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clean up query range conversion state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * review: address PR #49437 items 3 and 4 Item 3: remove unused COSMOS.USE_AAD_AUTH system property from the thinclient profile in azure-cosmos-tests/pom.xml. The Maven property cosmos.use.aad.auth is not defined anywhere in the repo, so the substitution produced a literal that Boolean.parseBoolean reads as false; TestSuiteBase already falls back to the COSMOS_USE_AAD_AUTH env var. Item 4: in Http2PingKeepaliveTest, gate the THINCLIENT_PROBE_ENABLED=false override behind !THIN_CLIENT_ENABLED so the probe is disabled only in the pure Compute Gateway (Gateway V2) branch on port 443, where thin-client routing is off entirely and probe POSTs to port 10250 are pure noise. In the thin-client branch (port 10250) the probe is intentionally left enabled so the same production code path that gates thin-client routing is exercised. Mirror the conditional in afterClass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #49437 Review 2 feedback - QueryPlanRetriever: clarify thin-client / Gateway V1 routing comment; explain that Gateway V1 is pinned only when PartitionKeyDefinition is unavailable to convert proxy queryRanges to EPK hex ranges. - RxDocumentClientImpl + DocumentQueryExecutionContextFactory: plumb the resolved DocumentCollection into validateCustomQueryForReadManyByPartitionKeys -> fetchQueryPlanForValidation so the validation query-plan request is thin-client-eligible instead of forcibly pinned to Gateway V1. - EndpointProbeClient: extract DiagnosticsSnapshot and ProbeResult into top-level types (EndpointProbeDiagnosticsSnapshot, EndpointProbeResult); remove @SuppressWarnings("unused"); use the failure reason in RED-cycle logs and toString(); verbose NPE message in the constructor naming the required dependency and wiring entry point. - CHANGELOG: move thin-client entry under Features Added; one-line summary with HTTP/2 + gatewayMode requirement and PR link. - CosmosNotFoundTests: add forceHttp1IfGatewayMode helper applied to every fast-group client construction so Gateway-mode clients in the fast group use HTTP/1.1 and are guaranteed not to route through the proxy. Keep the COSMOS.THINCLIENT_PROBE_ENABLED=false set/clear for the thinclient group's defensive teardown. - PerPartitionCircuitBreakerE2ETests: drop the THINCLIENT_PROBE_ENABLED property mutation now that probe behavior is asserted only by tests that opt in explicitly. * Fix CI compile: import ConnectionPolicy in CosmosNotFoundTests * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Thread DocumentCollection into readMany query-plan validation readManyByPartitionKeys validates any caller-supplied custom query by fetching a query plan and asserting it is single-partition and non-hybrid. Until now that validation called fetchQueryPlanForValidation with no DocumentCollection, so the plan request was pinned to Gateway V1 (useGatewayMode = (partitionKeyDefinition == null)). Thread the container's DocumentCollection from RxDocumentClientImpl.validateCustomQueryForReadManyByPartitionKeys -> DocumentQueryExecutionContextFactory.fetchQueryPlanForValidation so QueryPlanRetriever has the PartitionKeyDefinition needed to convert PartitionKeyInternal-formatted queryRanges from the thin client (Gateway V2) proxy into the EPK-hex Range<String> entries the query pipeline consumes. With this wiring, the validation query plan goes to the thin client when the client is configured for it, and remains on Gateway V1 otherwise. No behavior change on the non-thin-client path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cover ReadManyByPartitionKey QueryPlan bifurcation with unit + E2E tests - Add ReadManyByPartitionKeyQueryPlanRoutingTest unit tests that pin the useGatewayMode gate in QueryPlanRetriever: gateway mode when DocumentCollection is null and partitioned mode when a PartitionKeyDefinition is present. - Add three readManyByPartitionKeys E2E tests to ThinClientQueryE2ETest that exercise the validation QueryPlan path through Direct TCP (baseline) and Gateway V2 (thin client), covering no-custom-query, projection+filter, and parameterized variants. Each thin-client diagnostics page is asserted to use the :10250 endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(thin-client): disable probe in CI fault-injection tests + honor kill switch in forceUnhealthy Stage 1990 of build 6432767 had 83 failures in ClientRetryPolicyE2ETestsWithGatewayV2#serviceUnavailableWithGatewayV2 because the probe gate added by PR #49437 flips routing from :10250 to :443 when CI test accounts lack /connectivity-probe (failureThreshold=1 trips proxyHealthy=false on the first failed probe and the assertion at TestSuiteBase#assertThinClientEndpointUsed then sees :443 and fails). Test fix (per CosmosNotFoundTests precedent, commit 3b392e3acf5): - ClientRetryPolicyE2ETestsWithGatewayV2: disable probe in @BeforeClass / clear in @AfterClass. - ThinClientTestBase: same lifecycle in enableThinClientForTest / clearThinClientForTest so all ThinClient*E2ETest subclasses inherit the guard. Source fix: honor COSMOS.THINCLIENT_PROBE_ENABLED kill switch inside EndpointProbeClient.forceUnhealthy(), closing the item-5 invariant gap (previously only runProbeCycle() checked the flag, so a mismatch-driven forceUnhealthy could mutate proxyHealthy even with the probe disabled). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(thin-client): kill switch bypasses probe-health gate When COSMOS.THINCLIENT_PROBE_ENABLED=false, no probe cycles run, so the probe-health signal is stale/meaningless. Treat probe as healthy in that case so the gate is driven solely by COSMOS.THINCLIENT_ENABLED (already folded into useThinClient at construction). This fixes regressions in CI test classes (GatewayReadConsistencyStrategyE2ETest, FaultInjectionWithAvailabilityStrategyTestsBase, ThinClientQueryE2ETest) where probe to /connectivity-probe fails on test accounts and flips routing to :443 instead of :10250. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(thin-client): disable probe in CI fault-injection / failover / consistency tests Adds COSMOS.THINCLIENT_PROBE_ENABLED=false in @BeforeClass (and clears in @AfterClass) for PerPartitionCircuitBreakerE2ETests, PerPartitionAutomaticFailoverE2ETests, FaultInjectionServerErrorRuleOnGatewayV2Tests, and GatewayReadConsistencyStrategyE2ETest. These tests run against CI accounts whose /connectivity-probe endpoint is not deployed. Without this, the first failed probe trips proxyHealthy=false and routing falls back to Gateway V1 (port 443), causing assertThinClientEndpointUsed (which expects port 10250) to fail. Locally validated PPAF (16 tests, 0 failures), GRCS (3 tests, 0 failures), FI (3 tests, 0 failures). PPCB still in progress at commit time; pushing to let local + CI run in parallel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip RNTBD QueryPlan frames when capturing V2 feed requests QueryPlan requests intentionally carry no RCS/CL headers (matches the V1 HTTP behavior). When the V2 thin-client routes the QueryPlan precursor through the same :10250 endpoint as the data query, the spy must skip the QueryPlan frame so the assertion checks the actual data-query frame. This mirrors the IS_QUERY_PLAN_REQUEST filter on the V1 path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client QueryPlan error returning statusCode 0 instead of 400 The thin-client/Gateway-V2 proxy returns a non-2xx response with a raw, non-JSON, NUL-padded error body for invalid-syntax queries. In RxGatewayStoreModel.validateOrThrow, new CosmosError(body) attempted to parse that body as JSON and threw IllegalArgumentException, which escaped the method before the existing status-carrying throw could run. Upstream then wrapped it as statusCode 0. Wrap the CosmosError(body) construction in a narrow try/catch (IllegalArgumentException) and fall back to the non-parsing CosmosError(errorCode, message) constructor with a sanitized body. The existing throw now fires with the correct status (400) and the proxy error text is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden ThinClientQueryE2ETest assertions (F1-F5) F1: strict 400 + unconditional thin-client endpoint check on invalid query, locking the statusCode-0->400 fix. F2: ordered-vs-sorted-set document-ID comparison (ORDER BY sequence-compared, others set-compared). F3: ID-set equality + no-duplicate check across drained continuation pages. F4: numeric-tolerance (1e-6) comparison for scalar and GROUP BY aggregates to avoid float-formatting false mismatches. F5: validated vector/full-text/hybrid queries match Direct vs thin-client end-to-end through the proxy. Validated live via -Pthinclient: 84 tests, 0 failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace thin-client test matrix with reverse-engineered E2E test spec Rewrite THINCLIENT_TEST_MATRIX.md as a reviewable test-design specification reverse-engineered from the committed ThinClientQueryE2ETest code (84 tests). Documents the differential-testing oracle (Direct :443 baseline vs thin client :10250 SUT), the data-model fixture, every assertion contract (endpoint provenance, ordered-vs-unordered ID equality, scalar/GROUP BY tolerance), the full 84-test matrix, the F1-F5 hardened special cases (continuation draining, invalid-query 400, vector/FTS/hybrid ranking, readMany validation path), advertised-feature coverage, and known gaps (CountIf/DCount/MultipleOrderBy) for reviewer sign-off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply Aditya Sarpotdar review feedback to thin-client E2E tests Executes actionable review feedback on ThinClientQueryE2ETest plus two harness fixes surfaced during live validation against thin-client-multi-region-ci. ThinClientQueryE2ETest: - Strict ordering: ORDER BY results validated with isStrictlyOrdered across the board (stricter-by-default, per reviewer guidance) instead of set parity. - Add testMultipleOrderBy() with a composite-index container to cover the MultipleOrderBy query feature. - Add testDCount() using the canonical Cosmos DCount idiom (COUNT over a DISTINCT VALUE subquery); SQL-standard COUNT(DISTINCT ...) is not valid Cosmos SQL grammar. - Reword multi-EPK-range comment/javadoc to reflect emulator/backend reality (multiple metadata ranges served by a single backend partition; SDK routing and query pipeline still exercised). Harness fixes: - TestSuiteBase: add wait-and-poll utility waitForCollectionToBeAvailableToRead (predicate on NotFound/substatus 1013) to deflake "Collection is not yet available for read" on freshly created containers; update call sites in OrderbyDocumentQueryTest, NonStreamingOrderByQueryVectorSearchTest, QueryValidationTests, ReadFeedCollectionsTest. - SinglePartitionDocumentQueryTest: make the processMessage Mockito assertion mode-aware (thin-client routes query plan + query => times(2)). Validated live against thin-client-multi-region-ci: 86/86 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Redesign thin-client probe to per-region probe-and-cache model Convert the global circuit-breaker (failure/recovery hysteresis re-probing all regions every refresh) into a per-region one-shot probe-and-cache: probe only the delta of unproven regions, cache successes across account refreshes, and never re-probe a proven region. The routing gate stays conservative (Gateway V1 until every known region is proven, Gateway V2 once all succeed). - Configs: drop failure/recovery threshold knobs; add THINCLIENT_PROBE_MAX_RETRIES (default 3) and isThinClientExplicitlyEnabled() bypass. - EndpointProbeClient: per-region cache, delta probing, retry-N, all-known-succeeded gate, forced-unhealthy latch; per-probe timeout from thin-client connection timeout. - EndpointProbeDiagnosticsSnapshot: expose knownRegionCount/succeededRegionCount. - RxDocumentClientImpl: gate probe wiring/routing on !isThinClientExplicitlyEnabled. - Tests: rewrite EndpointProbeClientTests; update ConfigsTests, ThinClientProbeWiringTests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: opt-in thin-client QueryPlan routing + kill-switch, stop advertising CountIf, remove PR-reference comments * Address review feedback: broaden gateway error fallback to ClassCastException, drop response ObjectNode mutation, fail-fast on empty queryRanges, align readMany routing test with opt-in * Make thin-client QueryPlan no-EPK-headers contract explicit in wrapInHttpRequest * Fix missing StandardCharsets import in RxGatewayStoreModelTest * Test: add QueryOracle-derived LIKE/scalar-expression thin-client parity tests; drain only first page in OperationPolicies readAllItems to avoid full-container enumeration timeout * Test: verify a cached proxy-generated query plan still executes correctly through the thin client * Test: cross-partition tests use a dedicated multi-physical-partition fixture and assert >1 partition key range contacted; enforce ordered full-row comparison when id is not projected; clearer naming * Add CHANGELOG entry for QueryPlan request routing to Gateway V2 * Call out Execute Stored Procedure support in CHANGELOG entry * Fix flaky region/timing-sensitive fault-injection E2E tests CustomerWorkflowPartitionLevelCircuitBreakerTest and CustomerWorkflowHighE2ETimeoutTest: bump the ThresholdBasedAvailabilityStrategy threshold from 100ms to 300ms so the cold primary-region connection can dispatch the request and record the injected fault before the availability-strategy hedge to a warm secondary region wins the race and cancels it (which previously yielded 0 fault hits). PerPartitionCircuitBreakerE2ETests: before asserting short-circuit routing, wait until the first preferred region is actually marked Unavailable, removing the race between the failure-counter estimate and the asynchronous routing exclusion. Scoped to single-faulty-region configs (expectedRegionCountWithFailures == 1) and run once so fault-in-all-regions configs (which can never short-circuit) do not hit the method timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove test AAD auth toggle; move thin-client test matrix to docs/ and align with ThinClientQueryE2ETest * Thin-client review: tri-state enablement/probe-decision + EndpointProbeClient cleanup - Configs: COSMOS.THINCLIENT_ENABLED is tri-state (null=unset default-on, TRUE=opt-in, FALSE=opt-out); consumers interpret null. Remove probe kill switch (COSMOS.THINCLIENT_PROBE_ENABLED) and probe-path config (path is fixed /connectivity-probe). - ThinClientConnectivityConfig (new): single authority for enablement + the shouldUseThinClientStoreModel routing gate, taking the nullable probe decision so null never collapses into a boolean clause at the call site. - GlobalEndpointManager.getProxyProbeDecision(): tri-state Boolean (null = no decision: probe not wired). - EndpointProbeClient: hard-code PROBE_PATH; remove cycleId correlation and unused EndpointProbeResult.reason; crispen runProbeCycle (single currentGate no-op Mono, removeIf delta). - Tests: repoint routing-gate tests to the 5-arg tri-state signature (+null/opt-in cases); remove THINCLIENT_PROBE_ENABLED usage from E2E tests; drop removed-config tests. * Consolidate probe state into single source-of-truth map; remove force-unhealthy latch and per-cycle copies - Replace volatile knownEndpoints Set + probeSucceeded CHM with one ConcurrentHashMap<URI,Boolean> (probeEndpointToProbeSuccessState): keys = current topology, value = proven/not-yet-proven. Gate is non-empty AND all TRUE. - Remove the force-unhealthy latch: an empty/null topology now empties the map via retainAll so the gate falls to false naturally (route to Gateway V1). - runProbeCycle reconciles and probes by iterating the provided Collection<URI> directly (reference only, no LinkedHashSet copies) and drops the delta/empty short-circuit fast paths; the Flux filters not-yet-proven endpoints inline. - applyCycleResult uses replace() so a region pruned by a concurrent reconcile is never resurrected. Unit suite green: 2579 tests, 0 failures. * Add cross-partition aggregate / GROUP BY parity tests for thin-client query Adds 7 thin-client E2E tests covering cross-partition aggregate merge (COUNT/SUM/AVG/MIN/MAX) and GROUP BY, plus two helpers that assert Direct vs thin-client value parity and verify the query genuinely fanned out (distinctPartitionKeyRangesContacted > 1), exercising the cross-partition MERGE path the single-partition aggregate tests miss. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix EPK over-span; add QueryPlan parity tests For a partial (prefix) hierarchical partition key, the thin-client / Gateway V2 path previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter, so the proxy resolved the request to the owning physical partition and returned every co-located document (e.g. 18 instead of 6). ThinClientStoreModel.wrapInHttpRequest now sets READ_FEED_KEY_TYPE= EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from() so RntbdRequestHeaders.addStartAndEndKeys serializes the prefix EPK sub-range as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. Adds ThinClientQueryE2ETest coverage: QueryPlan parity across sources and a hierarchical prefix half-open range test asserting thin-client matches Direct TCP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix thin-client MULTI_HASH prefix partition key EPK over-span For a partial (prefix) hierarchical (MULTI_HASH) partition key, the thin-client (Gateway V2) store model previously sent only StartEpkHash/EndEpkHash routing headers and no doc-level EPK filter. The proxy therefore resolved the request to the owning physical partition and returned every co-located document (an over-span) instead of only those matching the prefix. ThinClientStoreModel.wrapInHttpRequest now detects a prefix MULTI_HASH key and sets READ_FEED_KEY_TYPE=EffectivePartitionKeyRange, START_EPK and END_EPK on the request headers before RntbdRequest.from(), so RntbdRequestHeaders serializes the prefix EPK sub-range [hash(prefix), hash(prefix) + "FF") as the backend doc-level filter (hex string as UTF-8 bytes), mirroring the Direct/RNTBD FeedRangeEpkImpl path. StartEpkHash/EndEpkHash routing headers are retained. QueryPlan requests and full (non-prefix) keys are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thin-client query E2E parity regression tests Ports ThinClientQueryE2ETest and its ThinClientTestBase helper. The suite self-baselines every query against a Direct (RNTBD) client and the thin-client (:10250 HTTP/2) path, asserting identical results. This directly guards the MULTI_HASH prefix partition-key EPK over-span fix: prefix (single-component) hierarchical-partition-key queries must return only the narrow matching set, matching Direct, rather than over-spanning to all co-located documents. Both files are runtime self-enabling (enableThinClientForTest() + builder-driven HTTP/2 via setEnabled(true)); no module property or TestSuiteBase changes needed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reuse prefix EPK range on thin-client parallel prefix-query path The earlier fix (499f928) only scoped the prefix MULTI_HASH read when the request carried a non-null partition key. The parallel prefix-query path intentionally nulls the partial hierarchical partition key and instead carries the narrow prefix effective-partition-key range on the request's feedRange, so that guard never fired there and the thin-client read still over-spanned to the whole physical partition. Mark such requests (prefixPartitionKeyQuery) in ParallelDocumentQueryExecutionContextBase and, in ThinClientStoreModel, reuse the already-computed FeedRangeEpkImpl range as the doc-level EPK filter (START_EPK/END_EPK + StartEpkHash/EndEpkHash) instead of recomputing getEPKRangeForPrefixPartitionKey from a partition key we no longer have. This mirrors the Direct/GatewayV1 decision points and honors DRY/SRP. Also relax assertThinClientEndpointUsed so an all-QueryPlan diagnostics context passes: in thin-client mode QueryPlan calls resolve via the classic gateway and are the only requests permitted on a non-thin-client endpoint; any non-QueryPlan (data) request on a non-thin-client endpoint still fails the assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Consolidate thin-client prefix EPK RNTBD headers, propagate prefix flag on clone, add readAllItems/readMany prefix HPK tests ThinClientStoreModel.wrapInHttpRequest now sets all five EPK-specific RNTBD headers (ReadFeedKeyType, StartEpk/EndEpk, StartEpkHash/EndEpkHash) in a single block directly on the RNTBD request, replacing the split request-header-map + post-construction approach. Wire encoding is unchanged (hex-string UTF-8 bytes for StartEpk/EndEpk, decoded hash bytes for the hash headers). RxDocumentServiceRequest.clone() now copies prefixPartitionKeyQuery so hedged/availability-strategy/retry clones keep the prefix signal and do not over-span. ThinClientQueryE2ETest adds prefix-HPK regression coverage for readAllItems (ReadFeed) and readManyByPartitionKeys, asserting Direct-vs-thin-client parity and per-tenant scoping (no over-span). * Simplify prefix-query flag assignment in ParallelDocumentQueryExecutionContextBase Assign isPrefixPartitionKeyQuery directly from isPartialPartitionKeyQuery(...) and guard only the PARTITION_KEY-setting branch with !isPrefixPartitionKeyQuery, removing the if/else. Behavior is unchanged: a partial (prefix) key still leaves partitionKeyInternal null so the prefix FeedRangeEpkImpl survives createDocumentServiceRequestWithFeedRange. * Add CHANGELOG entry for thin-client prefix hierarchical partition key over-span fix * Minimize prefix-query flag change vs main in ParallelDocumentQueryExecutionContextBase Keep main's single 'if (!isPartialPartitionKeyQuery)' structure; only capture the predicate in isPrefixPartitionKeyQuery and pass it to request.setPrefixPartitionKeyQuery. Removes the if/else. * Order thin-client prefix detection to prefer the cached feed-range path In ThinClientStoreModel.wrapInHttpRequest, check the cached prefix feed-range path (isPrefixPartitionKeyQuery + FeedRangeEpkImpl) first and fall back to recomputing from the partition key only for the PK-bearing prefix case. Behavior is unchanged (the two cases are mutually exclusive). * Address Copilot review: validate all requests in thin-client endpoint assertion TestSuiteBase.assertThinClientEndpointUsed now validates every request instead of early-returning on the first thin-client match, so a mixed scenario (some data requests via the classic gateway) fails; QueryPlan requests remain exempt and null endpoints are handled. ThinClientTestBase.assertThinClientEndpointUsed delegates to the shared TestSuiteBase implementation, removing duplicated logic that could NPE on a null endpoint and reject valid QueryPlan-via-gateway scenarios. * Simplify thin-client prefix EPK handling to reuse HTTP EPK headers Replace the explicit MULTI_HASH prefix-detection and pre-serialization range computation with a header-presence branch: when the request already carries START_EPK/END_EPK HTTP headers (populated upstream by FeedRangeEpkImpl for prefix hierarchical partition keys and explicit EPK-range reads), set only the additive StartEpkHash/EndEpkHash RNTBD tokens. The ReadFeedKeyType/StartEpk/ EndEpk string tokens are copied verbatim from those HTTP headers by RntbdRequestHeaders#addStartAndEndKeys during RntbdRequest.from(), so they no longer need to be set here. Removes now-unused imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove dead prefix-query flag and expand HPK thin-client test coverage Following the simplification of the prefix EPK fix to reuse the existing StartEpk/EndEpk HTTP headers (a56db9ad44c), the RxDocumentServiceRequest prefixPartitionKeyQuery flag is no longer read anywhere. Remove the field, its accessors, the clone copy, and the caller assignment in ParallelDocumentQueryExecutionContextBase. Test changes: - Retrofit CosmosMultiHashTest into the thinclient TestNG group so the MULTI_HASH prefix over-span coverage also runs against GatewayV2 (proxy :10250) over HTTP/2, not just the emulator gateway. - ThinClientQueryE2ETest: correct the prefix-scope Javadoc to describe the actual header-copy behavior and add a deterministic prefix-partition-key readAllItems regression. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert ParallelDocumentQueryExecutionContextBase to main The prefix-query flag on RxDocumentServiceRequest was removed in 03a7c672b95 after the fix was simplified to reuse the StartEpk/EndEpk HTTP headers. The only functional edit in this file (the setter call) went away with it, leaving behind unnecessary refactor noise. Restore the file to its upstream/main state so the hotfix touches no query execution code. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore thin-client-probe symbols dropped during merge of #47759 The merge of AzCosmos_GatewayV2_QueryPlanSupport (#47759) dropped probe's thin-client-probe supporting code while keeping the method bodies that reference it, breaking compilation: - GlobalEndpointManager: restore imports (http.HttpClient, java.util.Set) and the thinClientProbeClient AtomicReference field. - LocationCache: restore getThinClientRegionalEndpoints() and the collectThinClientEndpointsAllOrNothing() helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cosmos): allowlist subStatus 1003 in thinclient readiness-probe retry The @BeforeSuite container-readiness probe issues a QueryPlan against a freshly created container. With thin-client default-on, a fresh container in a multi-region account can transiently return 404/subStatus 1003 (OWNER_RESOURCE_NOT_EXISTS) during regional metadata propagation. The classifier did not allowlist 1003, so the probe treated it as non-retryable and cascade-skipped the entire thinclient suite. Add OWNER_RESOURCE_NOT_EXISTS to the NOTFOUND retryable allowlist in isRetryableCollectionReadinessFailure so the probe retries through propagation and the suite runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cosmos-benchmark): parameterize SDK version + add RunId/EndpointFlavor cold-start knobs Instrument azure-cosmos-benchmark for the cold-start Create-latency matrix driving the thin-client probe (PR #49437): - pom: parameterize azure-cosmos + azure-cosmos-encryption versions (-Dazure-cosmos.version=...) so GA 4.79/4.80/4.81 and the probe build can be benchmarked from one harness. - Common tags (RunId, SdkVersion, ConnectionMode, EndpointFlavor, Phase) applied to every metric so a shared Grafana dashboard can slice runs. - endpointFlavor knob (ComputeGateway|ThinClient) maps to COSMOS.THINCLIENT_ENABLED at client build to drill both the Gateway V1 and Gateway V2 (thin-client) HTTP/2 paths. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Enforce QueryPlan thin-client routing invariant and harden container-readiness probe for thin-client-default-on - Remove the QueryPlan exemption in assertThinClientEndpointUsed: when ThinClient is opted in with HTTP/2 enabled, QueryPlan calls MUST also route to Gateway V2 (:10250), matching production default DEFAULT_THINCLIENT_QUERY_PLAN_ENABLED=true. - Restore public static visibility on ThinClientTestBase.assertThinClientEndpointUsed / assertGatewayEndpointUsed so cross-package static imports (CosmosMultiHashTest) compile. - Add OWNER_RESOURCE_NOT_EXISTS (subStatus 1003) to the retryable NOTFOUND allowlist in TestSuiteBase.isRetryableCollectionReadinessFailure. With thin-client default-on the beforeSuite readability probe (a QueryPlan) routes through GW V2; a freshly-created container not yet propagated to a secondary region returns 404/1003, which must be treated as a transient readiness failure instead of cascade-skipping the suite. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add kill-switch flip validation for QueryPlan thin-client routing Validate the COSMOS.THINCLIENT_QUERY_PLAN_ENABLED sysprop/env kill-switch in both directions: - Unit: new case in ReadManyByPartitionKeyQueryPlanRoutingTest asserts QueryPlan is forced to Gateway V1 (useGatewayMode=true) when the flag is disabled, even with thin-client eligible. - E2E: make TestSuiteBase.assertThinClientEndpointUsed flag-aware so QueryPlan endpoint assertions flip based on the live-read flag while data operations continue to route to the thin-client endpoint. Both scenarios verified green (unit 3/3; E2E 112/112, oracle:true for flag ON and flag OFF). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore isAnyRegionShortCircuitedForPartition helper dropped during #47759 merge The #47759 merge conflict resolution kept HEAD's short-circuit-wait loop (which calls isAnyRegionShortCircuitedForPartition) but dropped the method definition. Recovered from probe tip 42c2c7d733f and re-inserted. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove azure-cosmos-benchmark and HTTP/2 ping-handler changes (moved to #49725) These changes belong in PR #49725, not the endpoint-probe PR #49437. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Run thin-client probe cycle on every account refresh (delta-gated) Wire runThinClientProbeCycleMono() into the account-refresh funnel in GlobalEndpointManager so the connectivity probe runs on every refresh, delta-gated to new/unproven regions in EndpointProbeClient. Previously the probe driver had no production callers, leaving the default-on thin-client routing gate permanently false. Add a GEM delta-refresh unit test proving a subsequent refresh probes only newly-added regions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thin-client endpoint-probe (implicit path) E2E test lane + GEM probe wiring Wire runThinClientProbeCycleMono() into GlobalEndpointManager.refreshLocationPrivateAsync so the connectivity probe runs on every account refresh: as a prefix when a fresh databaseAccount is present, and after the foreground database-account read (.map -> .flatMap). Add an implicit-path test lane that leaves COSMOS.THINCLIENT_ENABLED unset (thin-client default-on, routing gated on the probe verdict with Gateway V1 fallback) while keeping HTTP2_ENABLED true: - new 'thinclient-endpoint-probe' maven profile (azure-cosmos-tests/pom.xml) - new Cosmos_Live_Test_ThinClient_EndpointProbe CI stage (tests.yml) - live-thinclient-endpoint-probe-platform-matrix.json - thinclient-endpoint-probe-testng.xml suite - register thinclientEndpointProbe group in TestSuiteBase before/after suite - E2E probe tests: point-operation, query, change-feed, stored-procedure + shared base Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align query/* and test files with Azure/main to reduce PR diff Reset QueryPlanRetriever, PartitionedQueryExecutionInfo and 9 test files to Azure/main. These carried Gateway V2 QueryPlan work (partitionKeyDefinition guard, CountIf, queryRanges removal) and parity-test adaptations that overlap upstream #47759, which Azure/main already owns. Scoping this PR to the thin-client connectivity-probe flow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Remove isThinClientEnabled() helper; require positive probe verdict to route thin-client Collapse all call sites onto the tri-state reader Configs.isThinClientEnabledExplicitly() and remove the redefined isThinClientEnabled() helper, which had flipped the unset default from false to true. Thin-client routing now requires an explicit opt-in or an affirmative connectivity-probe verdict; an unset flag no longer routes to Gateway V2 by default (null/FALSE probe verdict -> Gateway V1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Collapse thin-client enablement to single nullable isThinClientEnabled() Rename Configs.isThinClientEnabledExplicitly() to isThinClientEnabled() (nullable Boolean) and remove the hasUserExplicitlyEnabledThinClient() and ThinClientConnectivityConfig.isExplicitThinClientOptIn() helpers. The routing gate shouldUseThinClientStoreModel now takes the tri-state Boolean directly: a non-null value is a hard contract (TRUE routes to Gateway V2, FALSE pins to V1), and null falls back to the connectivity-probe verdict. This eliminates the scattered Boolean.equals opt-in checks. Default remains null (not default-on). Updated ThinClientRoutingGateTests implicit-path cases to pass null for the tri-state arg (probe-gated) instead of false. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Consolidate thin-client probe E2E tests into shared classes via dual TestNG group Tag the existing ThinClient*E2ETest classes (and ThinClientTestBase) with both the `thinclient` and `thinclientEndpointProbe` groups so the same test bodies run in either CI lane against a different ambient enablement path: - thinclient lane: -DCOSMOS.THINCLIENT_ENABLED=true (explicit opt-in) - thinclient-endpoint-probe lane: flag unset -> endpoint connectivity probe drives routing Delete the 5 duplicate ThinClientEndpointProbe* copies; the probe suite XML discovers by group across com.azure.cosmos.*, so it auto-picks up the retagged classes with zero duplication. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add thinclientEndpointProbe group to shared suite setup/teardown The consolidated thin-client E2E tests are dual-tagged with the thinclientEndpointProbe group, but the shared @BeforeSuite/@AfterSuite in TestSuiteBase (which creates and deletes the SHARED_* collections) was still gated only on the thinclient group. Under the probe lane's group filter the suite setup never ran, leaving the shared collections null and causing an NPE in cleanUpContainer during @BeforeClass. Adding the group makes both lanes green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix ClientConfigDiagnosticsTest stale gwV2Cto assertions Update the gw connCfg assertions to expect gwV2Cto:PT5S instead of gwV2Cto:n/a, matching the intentional diagnostics change where the thin-client (gateway V2) connect timeout is emitted by default when COSMOS.THINCLIENT_ENABLED is unset. This also resolves the cascading sessionRetryOptionsInDiagnostics failure caused by full() leaking a system property when its assertion threw before cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Update UserAgent tests to expect |F4 ThinClient feature flag ThinClient is default-enabled when COSMOS.THINCLIENT_ENABLED is unset on this branch, so the UserAgent suffix now carries the |F4 feature flag. Update the stale assertions to mirror RxDocumentClientImpl.addUserAgentSuffix dynamically (ThinClient default-on + Http2/Http2PingHealth) instead of hardcoding the suffix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Align endpoint probe cache to .NET add-only semantics Refactor EndpointProbeClient to an add-only proven-healthy endpoint cache, matching the .NET implementation (PR azure-cosmos-dotnet-v3#5970). Once an endpoint is proven ThinClient-routable it is never pruned; the routing gate is evaluated against a volatile snapshot of the current topology so a transiently vanished proven region does not gate current routing and is not re-probed when it re-appears. Probe cycles only probe the delta of new, not-yet-proven endpoints on each account refresh. Add unit tests covering vanish/re-appear and unsupported-new-region scenarios (region added without support blocks the gate; removing it restores routing). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix CosmosDiagnosticsTest for default-on ThinClient |F4 user-agent suffix Fold ThinClient into generateHttp2OptedInUserAgentIfRequired so the helper dynamically mirrors production feature-flag suffixing, and remove a double-wrap of directClientUserAgent (already suffixed at construction). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Gate thin-client endpoint assertion on explicit opt-in only On the implicit/probe path (COSMOS.THINCLIENT_ENABLED unset), thin-client routing depends on the endpoint probe greenlighting all current regions. Under fault injection the probe is not guaranteed to converge, so requests may legitimately fall back to Gateway V1 (:443). Assert thin-client routing only when thin-client is explicitly opted in (THINCLIENT_ENABLED=true), the sole config where routing is deterministic because the probe is bypassed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address PR #49437 review comments: tri-state THINCLIENT_ENABLED parsing, probe-client close, fire-and-forget probe cycle, single-flight tests - Configs: parse COSMOS.THINCLIENT_ENABLED via explicit true/false whitelist (parseTriStateThinClientEnabled); any other value logs a warning and is treated as unset (null -> probe-gated) instead of Boolean.parseBoolean silently collapsing every non-"true" string to a hard opt-out. - GlobalEndpointManager: close() now cancels the in-flight probe-cycle subscription and closes the EndpointProbeClient so a stale cycle drops its result. Probe cycle is fired fire-and-forget on force-refresh and topology refresh so init()/cross-region retry never block on the probe; routing stays on Gateway V1 until the probe proves the proxy endpoints. - Tests: add ConfigsTests.thinClientEnabledInvalidValueTreatedAsUnset and two EndpointProbeClientTests covering single-flight overlap skip and closed-mid-cycle result-drop (gate stays conservative). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refactoring. * Fix SinglePartitionDocumentQueryTest stale thin-client routing predicate useThinClient() now reflects config-eligibility only; the test must gate on actual per-request routing (read locations + probe/opt-in) to match where the query lands. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address PR review: consolidate probe-wiring gate on canThinClientBeUsed() - Gate probe wiring in RxDocumentClientImpl on canThinClientBeUsed() so an explicit THINCLIENT_ENABLED=true still wires the probe (routing bypasses it), covering true->null runtime transitions. Removed dead canThinClientBeImplicitlyEnabled() from ThinClientConnectivityConfig and updated javadoc/comments/pom …

Removing Class.forName for implementation bridge helpers

455778e

FabianMeiswinkel requested a review from xinlian12 as a code owner May 16, 2022 22:48

mssfang and others added 2 commits May 16, 2022 17:43

merge patch changes to main (Azure#28898)

60cf956

Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>

xinlian12 reviewed May 17, 2022

View reviewed changes

Added changelog

fdadd68

Add readme file for spring native configuration module (Azure#28891)

c5745b0

xinlian12 approved these changes May 17, 2022

View reviewed changes

xinlian12 and others added 11 commits May 17, 2022 19:10

ThroughputControlFix - New client item created when enable same group…

ca71030

… multiple times (Azure#28905) * ThroughputControl fix * add change log Co-authored-by: annie-mac <annie-mac@yindeng2019.fareast.corp.microsoft.com>

Update unreleased_ dependency that was missed. (Azure#28909)

1d439c7

Merge branch 'main' into users/fabianm/SetAccessorsFix

a449573

Fix formrecognizer perf tests (Azure#28913)

7414e9f

Merge branch 'main' of https://github.com/Azure/azure-sdk-for-java in…

d6ad30e

…to users/fabianm/SetAccessorsFix

Merge branch 'main' of https://github.com/Azure/azure-sdk-for-java in…

081954e

…to users/fabianm/SetAccessorsFix

Merge branch 'users/fabianm/SetAccessorsFix' of https://github.com/Fa…

d642cdf

…bianMeiswinkel/azure-sdk-for-java into users/fabianm/SetAccessorsFix

Update spotbugs-exclude.xml

4973e59

Update spotbugs-exclude.xml

afd5dde

Adding API comments

91fc278

Update CHANGELOG.md

ffb6676

xinlian12 pushed a commit that referenced this pull request Apr 19, 2023

Merge pull request #3 from xinlian12/ProactiveConnectionManagementFor…

8859de8

…BrokenConnections TestPR - NO REVIEW

github-actions Bot mentioned this pull request Mar 19, 2026

Add sdkReviewAgent agentic workflow for automated PR review #22

Merged

Navigation Menu

Removing Class.forName for implementation bridge helpers - #3

Removing Class.forName for implementation bridge helpers#3
FabianMeiswinkel wants to merge 16 commits into
xinlian12:improvementForCosmosExceptionHelperfrom
FabianMeiswinkel:users/fabianm/SetAccessorsFix

FabianMeiswinkel commented May 16, 2022

Uh oh!

xinlian12 May 17, 2022

Uh oh!

FabianMeiswinkel May 17, 2022

Uh oh!

xinlian12 May 17, 2022

Uh oh!

xinlian12 commented May 17, 2022

Uh oh!

FabianMeiswinkel commented May 17, 2022

Uh oh!

xinlian12 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

FabianMeiswinkel commented May 16, 2022

Uh oh!

xinlian12 May 17, 2022

Choose a reason for hiding this comment

Uh oh!

FabianMeiswinkel May 17, 2022

Choose a reason for hiding this comment

Uh oh!

xinlian12 May 17, 2022

Choose a reason for hiding this comment

Uh oh!

xinlian12 commented May 17, 2022

Uh oh!

FabianMeiswinkel commented May 17, 2022

Uh oh!

xinlian12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants