Skip to content

Rest client catch up - #411

Merged
RalphSteinhagen merged 4 commits into
mainfrom
rest_client_catch_up
Sep 11, 2026
Merged

RalphSteinhagen merged 4 commits into
mainfrom
rest_client_catch_up

Conversation

@drslebedev

Copy link
Copy Markdown
Contributor

Summary

Adds indexed batch catch-up for REST subscriptions, allowing clients to retrieve buffered messages with fewer requests after falling behind.

Related to #405.

Changes

  • Add LongPollingBatch support to the server and native/Emscripten clients:

    • LongPollingIdx=42&LongPollingBatch=10: wait for messages 42–51 and return them together.
    • LongPollingIdx=42&LongPollingBatch=AllAvailable: return message 42 and all newer buffered messages, waiting for 42 if necessary.
  • Start from the oldest retained message when the requested batch index is too old. Fixed-count batches still return exactly the requested count.

  • Preserve each message’s index, topic, service name, and payload in multipart responses.

  • Report skipped samples alongside valid data and continue polling after malformed batches.

  • Fix subscription cleanup so clients can unsubscribe and subscribe again.

  • Catch native GET/SET and redirect query-parsing errors instead of allowing exceptions to escape.

  • Suppress third-party libsodium compiler warnings for GCC/Clang.

  • Document batch behavior and the response format.

Requests without LongPollingBatch still return one message. Proxy caching policy is unchanged; general automatic recovery remains separate work.

Test coverage

Adds or extends tests for multipart encoding/decoding, batch ranges and buffer retention, invalid requests, subscription lifecycle, and query-parsing failures. Integration coverage includes both native
polling modes and Emscripten batch delivery, gap warnings, and malformed-batch recovery.

Signed-off-by: drslebedev <dr.s.lebedev@gmail.com>
Signed-off-by: drslebedev <dr.s.lebedev@gmail.com>
Guard lazy query parsing and add a regression test for malformed redirects.

Signed-off-by: drslebedev <dr.s.lebedev@gmail.com>
- Support fixed-count and AllAvailable batches from a requested index.
- Preserve per-message metadata and report gaps in both REST clients.
- Continue polling after malformed batches.
- Fix subscription cleanup to allow unsubscribe and resubscribe.
- Add tests and document batch behavior.

Signed-off-by: drslebedev <dr.s.lebedev@gmail.com>
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@RalphSteinhagen RalphSteinhagen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the server, both clients, tests and docs; built locally with clang20 Debug and ran the long-polling/batch tests, all green. The clang20 CI failure is the unrelated OAuth Worker test (Keycloak login simulation).

Batch parameters are bounded, invalid requests are rejected before any side effect, and the multipart format is injection-safe. 👍

The following points below are non-blocking and fine as follow-ups:

  1. Caching: AllAvailable responses are not immutable for a given URL, unlike indexed single-message responses. Add cache-control: no-store to batch responses so a caching proxy can never serve a stale, shorter batch.
  2. Range guard in respondToLongPollBatch: firstIndex - entry.firstIndex is unsigned and the deque access is unchecked. Current dispatch guarantees the range, but handleNotification already tests readiness with >=, so clamping to [entry.firstIndex, entry.nextIndex()) is cheap insurance against an out-of-bounds read.
  3. Error notifications in batches arrive as data. A per-part error header would preserve Message::error; the single-message path has the same limitation.
  4. Recovery after an undecodable batch resumes with Next and drops what is still buffered. Retrying once from last+1 before falling back would lose less.
  5. Memory: the batch body is built as a string and then copied into an IoBuffer, up to 100 messages twice, with no byte cap. Encoding into the IoBuffer directly and capping AllAvailable by bytes bounds this.

If you need to touch the code anyway, I'd suggest to at least tackle the first point because this has an impact on our local testing and use-case.

@RalphSteinhagen
RalphSteinhagen merged commit 2487c6f into main Sep 11, 2026
8 of 11 checks passed
@RalphSteinhagen
RalphSteinhagen deleted the rest_client_catch_up branch September 11, 2026 14:39
@github-project-automation github-project-automation Bot moved this from 🆕 Ideas (∞) to ✅ QA-Accepted/Merged (∞) in Digitizer Reimplementation Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ QA-Accepted/Merged (∞)

Development

Successfully merging this pull request may close these issues.

[5SP,6SP] REST long polling: report backlog so clients can catch up

2 participants