diff --git a/README.md b/README.md index 56a7de8e..851b2be5 100644 --- a/README.md +++ b/README.md @@ -255,7 +255,7 @@ See [docs/pipelines.md](docs/pipelines.md) for the step-author walkthrough. | `http.auth` | `Credential` sealed hierarchy (`KeyCredential`, `NamedKeyCredential`, `BearerToken`), `BearerTokenProvider`, `AuthScheme`, `AuthMetadata`, RFC 7235 challenge parser, `BasicChallengeHandler`, `DigestChallengeHandler`, `CompositeChallengeHandler`. | | `http.sse` | `ServerSentEventReader` (WHATWG spec), `ServerSentEvent`, `ServerSentEventListener`, `BufferedSource.readServerSentEvents()`. | | `http.paging` | `PagedIterable`, `PagedResponse`, `PagingOptions` with `byPage()` and `stream()` accessors. | -| `pagination` | `Paginator` (with a `maxPages` safety cap) over cursor / page-number / token / link-header `PaginationStrategy` implementations, plus `Page` / `SimplePage`. | +| `pagination` | `Paginator` (with a `maxPages` safety cap) over cursor / page-number / link-header `PaginationStrategy` implementations, plus `Page` / `SimplePage`. | | `pipeline` | Recovery-aware primitives: `RequestPipeline`, `ResponsePipeline`, `ExecutionPipeline` over a sealed `ResponseOutcome`, with steps (`pipeline.step`, `pipeline.step.retry`) like `RetryStep`, `ResponseRecoveryStep`, `IdempotencyKeyStep`, `ClientIdentityStep`. | | `serde` | `Serde`, `Serializer`, `Deserializer` abstractions and `Tristate` (absent / null / present). | | `io` | `Source`, `Sink`, `Buffer`, `BufferedSource`, `BufferedSink`, `IoProvider`, `Io`, `TeeSink`. | @@ -265,6 +265,9 @@ See [docs/pipelines.md](docs/pipelines.md) for the step-author walkthrough. | `util` | `Clock`, `Uuids` (non-blocking v4), `DateTimeRfc1123`, `RetryUtils`, `ProxyOptions`, `Futures`. | | `generics` | `Builder` — the generic builder interface every SDK builder implements. | +Token-style APIs (`next_page_token`, `pageToken`, …) are served by `CursorPaginationStrategy`: +construct it with the desired query-param name, e.g. `CursorPaginationStrategy(items, extractor, "page_token")`. + ## Building ```bash diff --git a/docs/architecture.md b/docs/architecture.md index 32c8a5e3..760f0067 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -346,9 +346,12 @@ Two complementary surfaces for walking multi-page responses. |-----------------------------------------------------------------|-----------------------------------------------------------------------| | `Paginator` | Lazily iterates pages by re-issuing requests through an `HttpClient`; carries a `maxPages` safety cap | | `PaginationStrategy` | Computes the next-page request (or stops) from the current page | -| `CursorPaginationStrategy` / `PageNumberPaginationStrategy` / `TokenPaginationStrategy` / `LinkHeaderPaginationStrategy` | The four shipped strategies | +| `CursorPaginationStrategy` / `PageNumberPaginationStrategy` / `LinkHeaderPaginationStrategy` | The shipped strategies | | `PagedIterable` | First/next-page fetcher abstraction over `PagedResponse`, with its own `maxPages` cap | +Token-style APIs (`next_page_token`, `pageToken`, …) are handled by `CursorPaginationStrategy` +constructed with the query-param name set (e.g. `"page_token"`), so no separate token strategy is needed. + ### Serialization **Package**: `org.dexpace.sdk.core.serde` diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index d7ae2188..cd3874b8 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -372,8 +372,8 @@ defaults (per Square: `FAIL_ON_UNKNOWN_PROPERTIES=false`, `WRITE_DATES_AS_TIMEST ### WU-9: Pagination primitives -**Status: shipped.** `Page`, `Paginator`, `PaginationStrategy`, and the four strategies -(`Cursor` / `PageNumber` / `LinkHeader` / `Token`) are in `sdk-core/.../pagination`, alongside +**Status: shipped.** `Page`, `Paginator`, `PaginationStrategy`, and the three strategies +(`Cursor` / `PageNumber` / `LinkHeader`) are in `sdk-core/.../pagination`, alongside helper types `SimplePage` and `RequestRebuilder`. `Paginator` gained a `maxPages` safety cap (default `Long.MAX_VALUE`) beyond the original sketch, to bound runaway iteration against servers that never advance their cursor. @@ -390,7 +390,6 @@ link-header strategies without over-engineering. Sync first; async adapter follo - `CursorPaginationStrategy(cursorPath, itemsPath, parser)` — read `next_cursor` from body - `PageNumberPaginationStrategy(pageParam, itemsPath, parser)` — increment page number - `LinkHeaderPaginationStrategy(itemsPath, parser)` — RFC 5988 `Link: ; rel="next"` - - `TokenPaginationStrategy(tokenPath, tokenParam, itemsPath, parser)` — token in body, sent as query param - `sdk-core/src/main/kotlin/org/dexpace/sdk/core/pagination/PaginatorTests.kt` (test) — table-driven tests against MockWebServer fixtures. **Acceptance criteria:** diff --git a/sdk-core/api/sdk-core.api b/sdk-core/api/sdk-core.api index de3a5d14..3f2301e0 100644 --- a/sdk-core/api/sdk-core.api +++ b/sdk-core/api/sdk-core.api @@ -1937,13 +1937,6 @@ public final class org/dexpace/sdk/core/pagination/Paginator { public final fun streamAll ()Ljava/util/stream/Stream; } -public final class org/dexpace/sdk/core/pagination/TokenPaginationStrategy : org/dexpace/sdk/core/pagination/PaginationStrategy { - public fun (Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;)V - public fun (Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;Ljava/lang/String;)V - public synthetic fun (Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;Ljava/lang/String;ILkotlin/jvm/internal/DefaultConstructorMarker;)V - public fun parse (Lorg/dexpace/sdk/core/http/response/Response;Lorg/dexpace/sdk/core/http/request/Request;)Lorg/dexpace/sdk/core/pagination/Page; -} - public final class org/dexpace/sdk/core/pipeline/ExecutionPipeline { public fun (Lorg/dexpace/sdk/core/client/HttpClient;)V public fun (Lorg/dexpace/sdk/core/client/HttpClient;Lorg/dexpace/sdk/core/pipeline/RequestPipeline;)V diff --git a/sdk-core/src/main/kotlin/org/dexpace/sdk/core/pagination/TokenPaginationStrategy.kt b/sdk-core/src/main/kotlin/org/dexpace/sdk/core/pagination/TokenPaginationStrategy.kt deleted file mode 100644 index c256be23..00000000 --- a/sdk-core/src/main/kotlin/org/dexpace/sdk/core/pagination/TokenPaginationStrategy.kt +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Copyright (c) 2026 dexpace and Omar Aljarrah - * - * Licensed under the MIT License. See LICENSE in the project root. - * SPDX-License-Identifier: MIT - */ - -package org.dexpace.sdk.core.pagination - -import org.dexpace.sdk.core.http.request.Request -import org.dexpace.sdk.core.http.response.Response - -/** - * Token [PaginationStrategy]. Logically identical to [CursorPaginationStrategy] but with - * a different naming convention — many APIs (Google Cloud, Slack, GitHub GraphQL legacy) - * use the word *token* (`next_page_token`, `pageToken`) rather than *cursor*. - * - * Wire shape: - * - * - Page-N request: `GET /things?page_token=`. - * - Page-N response: JSON body containing both the items and the next token. - * - End of stream: response carries an empty / absent next token. - * - * Use this strategy when the token shape and naming differs from a typical "cursor" - * scheme — primarily for readability at call sites and for the default [tokenQueryParam] - * of `"page_token"`. - * - * @param T Element type extracted from the response. - * @property itemsExtractor Reads the list of items from the response. Called once per - * page; must drain the response body synchronously. - * @property tokenExtractor Reads the next-page token from the response, or returns - * `null` if there are no more pages. - * @property tokenQueryParam Query parameter name used to send the token (default - * `"page_token"`). - */ -public class TokenPaginationStrategy - @JvmOverloads - constructor( - private val itemsExtractor: (Response) -> List, - private val tokenExtractor: (Response) -> String?, - private val tokenQueryParam: String = "page_token", - ) : PaginationStrategy { - override fun parse( - response: Response, - initialRequest: Request, - ): Page { - val items: List = itemsExtractor(response) - val nextToken: String? = tokenExtractor(response) - val hasNext: Boolean = !nextToken.isNullOrEmpty() - val nextRequest: Request? = - if (hasNext) { - RequestRebuilder.withQueryParam(initialRequest, tokenQueryParam, nextToken) - } else { - null - } - return SimplePage(items = items, hasNext = hasNext, nextRequest = nextRequest) - } - } diff --git a/sdk-core/src/test/kotlin/org/dexpace/sdk/core/pagination/CursorPaginationTest.kt b/sdk-core/src/test/kotlin/org/dexpace/sdk/core/pagination/CursorPaginationTest.kt index dee1902b..0222da02 100644 --- a/sdk-core/src/test/kotlin/org/dexpace/sdk/core/pagination/CursorPaginationTest.kt +++ b/sdk-core/src/test/kotlin/org/dexpace/sdk/core/pagination/CursorPaginationTest.kt @@ -11,6 +11,7 @@ import org.dexpace.sdk.core.http.request.Method import org.dexpace.sdk.core.http.request.Request import org.dexpace.sdk.core.http.response.Response import java.util.IdentityHashMap +import java.util.stream.Collectors import kotlin.test.BeforeTest import kotlin.test.Test import kotlin.test.assertEquals @@ -127,4 +128,73 @@ class CursorPaginationTest { val paginator = Paginator(client, authRequest, strategy) assertEquals(listOf("a", "b"), paginator.iterateAll().toList()) } + + @Test + fun `streamAll yields the same items as iterateAll`() { + val client = StubHttpClient() + client.on("https://api.example.com/items") { req -> + textResponse(req, "items=1,2\ncursor=c1") + } + client.on("https://api.example.com/items?cursor=c1") { req -> + textResponse(req, "items=3,4\ncursor=") + } + + val (items, cursor) = buildCachedExtractors() + val strategy = CursorPaginationStrategy(items, cursor) + val paginator = Paginator(client, initialRequest(), strategy) + val streamed: List = paginator.streamAll().collect(Collectors.toList()) + assertEquals(listOf("1", "2", "3", "4"), streamed) + } + + @Test + fun `cursor with special characters is URL encoded in next request`() { + // Opaque cursors may contain `=` `+` `/` characters (base64) — the rebuilder must + // URL-encode them so the server sees the original value unmangled. + val rawCursor = "a+b/c=" + val encoded = "a%2Bb%2Fc%3D" + val client = StubHttpClient() + client.on("https://api.example.com/items") { req -> + textResponse(req, "items=one\ncursor=$rawCursor") + } + client.on("https://api.example.com/items?cursor=$encoded") { req -> + textResponse(req, "items=two\ncursor=") + } + + val (items, cursor) = buildCachedExtractors() + val strategy = CursorPaginationStrategy(items, cursor) + val paginator = Paginator(client, initialRequest(), strategy) + assertEquals(listOf("one", "two"), paginator.iterateAll().toList()) + assertEquals( + listOf( + "https://api.example.com/items", + "https://api.example.com/items?cursor=$encoded", + ), + client.receivedUrls, + ) + } + + @Test + fun `custom query-param name is used for the next-page cursor`() { + // Token-style APIs (next_page_token, pageToken, …) are served by setting + // cursorQueryParam; the next request must carry the cursor under that name. + val client = StubHttpClient() + client.on("https://api.example.com/items") { req -> + textResponse(req, "items=one\ncursor=tok1") + } + client.on("https://api.example.com/items?page_token=tok1") { req -> + textResponse(req, "items=two\ncursor=") + } + + val (items, cursor) = buildCachedExtractors() + val strategy = CursorPaginationStrategy(items, cursor, cursorQueryParam = "page_token") + val paginator = Paginator(client, initialRequest(), strategy) + assertEquals(listOf("one", "two"), paginator.iterateAll().toList()) + assertEquals( + listOf( + "https://api.example.com/items", + "https://api.example.com/items?page_token=tok1", + ), + client.receivedUrls, + ) + } } diff --git a/sdk-core/src/test/kotlin/org/dexpace/sdk/core/pagination/TokenPaginationTest.kt b/sdk-core/src/test/kotlin/org/dexpace/sdk/core/pagination/TokenPaginationTest.kt deleted file mode 100644 index c7e3db82..00000000 --- a/sdk-core/src/test/kotlin/org/dexpace/sdk/core/pagination/TokenPaginationTest.kt +++ /dev/null @@ -1,138 +0,0 @@ -/* - * Copyright (c) 2026 dexpace and Omar Aljarrah - * - * Licensed under the MIT License. See LICENSE in the project root. - * SPDX-License-Identifier: MIT - */ - -package org.dexpace.sdk.core.pagination - -import org.dexpace.sdk.core.http.request.Method -import org.dexpace.sdk.core.http.request.Request -import org.dexpace.sdk.core.http.response.Response -import java.util.IdentityHashMap -import java.util.stream.Collectors -import kotlin.test.BeforeTest -import kotlin.test.Test -import kotlin.test.assertEquals - -class TokenPaginationTest { - @BeforeTest - fun setup() { - installIoProvider() - } - - private fun initialRequest(): Request = - Request.builder() - .url("https://api.example.com/list") - .method(Method.GET) - .build() - - /** - * Parses `items=\ntoken=` once; returns (items, nextToken). - */ - private fun parsePayload(resp: Response): Pair, String?> { - val body = resp.body!!.source().use { it.readUtf8() } - val itemsLine = body.lineSequence().firstOrNull { it.startsWith("items=") } ?: "items=" - val tokenLine = body.lineSequence().firstOrNull { it.startsWith("token=") } ?: "token=" - val itemsRaw = itemsLine.removePrefix("items=") - val tokenRaw = tokenLine.removePrefix("token=") - val items = if (itemsRaw.isEmpty()) emptyList() else itemsRaw.split(",") - val token: String? = tokenRaw.ifEmpty { null } - return Pair(items, token) - } - - private fun cachedExtractors(): Pair<(Response) -> List, (Response) -> String?> { - val cache: MutableMap, String?>> = IdentityHashMap() - val items: (Response) -> List = { r -> - cache.getOrPut(r) { parsePayload(r) }.first - } - val token: (Response) -> String? = { r -> - cache.getOrPut(r) { parsePayload(r) }.second - } - return Pair(items, token) - } - - @Test - fun `token pagination walks pages and stops on empty token`() { - val client = StubHttpClient() - client.on("https://api.example.com/list") { req -> - textResponse(req, "items=alpha,beta\ntoken=t1") - } - client.on("https://api.example.com/list?page_token=t1") { req -> - textResponse(req, "items=gamma\ntoken=t2") - } - client.on("https://api.example.com/list?page_token=t2") { req -> - textResponse(req, "items=delta,epsilon\ntoken=") - } - - val (items, token) = cachedExtractors() - val strategy = TokenPaginationStrategy(items, token) - val paginator = Paginator(client, initialRequest(), strategy) - val collected = paginator.iterateAll().toList() - assertEquals(listOf("alpha", "beta", "gamma", "delta", "epsilon"), collected) - assertEquals(3, client.callCount) - } - - @Test - fun `custom token query param name works`() { - val client = StubHttpClient() - client.on("https://api.example.com/list") { req -> - textResponse(req, "items=x\ntoken=ab") - } - client.on("https://api.example.com/list?nextToken=ab") { req -> - textResponse(req, "items=y\ntoken=") - } - - val (items, token) = cachedExtractors() - val strategy = - TokenPaginationStrategy(items, token, tokenQueryParam = "nextToken") - val paginator = Paginator(client, initialRequest(), strategy) - assertEquals(listOf("x", "y"), paginator.iterateAll().toList()) - assertEquals(2, client.callCount) - } - - @Test - fun `streamAll yields the same items as iterateAll`() { - val client = StubHttpClient() - client.on("https://api.example.com/list") { req -> - textResponse(req, "items=1,2\ntoken=t1") - } - client.on("https://api.example.com/list?page_token=t1") { req -> - textResponse(req, "items=3,4\ntoken=") - } - - val (items, token) = cachedExtractors() - val strategy = TokenPaginationStrategy(items, token) - val paginator = Paginator(client, initialRequest(), strategy) - val streamed: List = paginator.streamAll().collect(Collectors.toList()) - assertEquals(listOf("1", "2", "3", "4"), streamed) - } - - @Test - fun `token with special characters is URL encoded in next request`() { - // Tokens may contain `=` `+` `/` characters (base64) — the rebuilder must URL-encode - // them so the server sees the original token unmangled. - val rawToken = "a+b/c=" - val encoded = "a%2Bb%2Fc%3D" - val client = StubHttpClient() - client.on("https://api.example.com/list") { req -> - textResponse(req, "items=one\ntoken=$rawToken") - } - client.on("https://api.example.com/list?page_token=$encoded") { req -> - textResponse(req, "items=two\ntoken=") - } - - val (items, token) = cachedExtractors() - val strategy = TokenPaginationStrategy(items, token) - val paginator = Paginator(client, initialRequest(), strategy) - assertEquals(listOf("one", "two"), paginator.iterateAll().toList()) - assertEquals( - listOf( - "https://api.example.com/list", - "https://api.example.com/list?page_token=$encoded", - ), - client.receivedUrls, - ) - } -}