Skip to content

Apply a batch per report with ?partial=true, naming the ones refused - #12

Merged
renatex314 merged 1 commit into
masterfrom
feat/partial-batch
Oct 5, 2026
Merged

renatex314 merged 1 commit into
masterfrom
feat/partial-batch

Conversation

@renatex314

@renatex314 renatex314 commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

Point 1 of #10, as agreed in the discussion there: one report the server refuses no longer has to cost the rest of its batch. Opt-in, so existing ^0.9.0 clients see no change. Independent of #11; the two merge cleanly, and the merged result passes everything (see Testing). Intended for 0.10.0 with #11.

Problem

A batch is all-or-nothing. One refused report (an unknown value, a full capacity dimension, oversized props, half a position) fails the request with a 400 naming only that report, and nothing in the batch lands. The Node client sends up to 1000 reports per request, so one bad device costs up to 999 good ones.

It's worse in the Reporter: it puts a failed batch back in its queue, and a report refused on its merits is refused every time. So one bad device makes every later flush fail and blocks every device queued with it.

Server: POST /positions?partial=true

  • Each report is applied on its own. The good ones land, and the refused ones come back in rejected, in batch order:
    {"accepted":2,"stale":0,"devices":2,
     "rejected":[{"index":1,"id":"truck-9","code":"bad_request","error":"device \"truck-9\": unknown value \"idel\" ..."}]}
    code and error are exactly what the 400 would have carried. id is null for a GeoJSON Feature that carried no id.
  • accepted + stale + patched + unknown + rejected = batch size.
  • rejected is present, possibly empty, whenever the flag is set. That's how a client tells a server that honoured it from one that predates it.
  • Still request-level, because none of them can be pinned on one report: a body that doesn't parse, an oversized batch, an overloaded server, and ?partial= set to anything other than true/false.

Without the flag: a batch is still all-or-nothing, and a batch with one bad report gets the same 400, message and code as before. The existing tests that pin that behaviour pass unchanged.

Change in both modes: reports are now checked before any value is resolved, in this order: position/geometry/id, then props, then values. Resolving a value on a capacity dimension takes a slot permanently, and a report refused for its props used to take one anyway. Covered by a_refused_device_takes_no_capacity_slot.

One visible side effect: when a batch has several bad reports of different kinds, the 400 may name a different one than before. Example: a bad value at item 1 and oversized props at item 2 used to name item 1, and now name item 2. (The commit message says the first-named error is unchanged too; that's wrong in this case.)

Node client

  • partial: true option: available on report, patch, reportGeoJSON and reporter. The default stays false.
  • Results: they carry rejected, with index renumbered from each chunk to the caller's whole list.
  • Acknowledgement check: checkAck counts rejected.
  • Reporter:
    • A refused report is dropped instead of requeued, and counted in stats.rejected.
    • It's passed to onError as a NetClusterError with body: { code: 'rejected', rejected }.
    • The flush itself succeeds, because the rest of the batch landed.
  • Upgrade order: against an older server the flag is ignored and a bad chunk is refused whole, as before. So the client can be upgraded before the servers.
  • Docs: index.d.ts has the new types (WriteOptions, RejectedReport), and both READMEs describe the behaviour.

Testing

  • Server: new tests/partial.rs (8 tests), covering:
    • Without the flag, the same first error and code, and nothing applied.
    • With it, good reports land and bad ones are named in batch order.
    • Accounting alongside stale/patched/unknown.
    • A full capacity rejecting only the overflowing device.
    • A refused report taking no slot.
    • GeoJSON: null geometry, missing id, swapped coordinates, unreadable properties and an unknown value.
    • A batch with nothing good.
    • The flag being strict and always answered.
    • Each case also checks index invariants with verify().
  • Client:
    • A unit test for renumbering across chunks and the acknowledgement check.
    • Integration tests against the real binary for report, patch, reportGeoJSON, and a Reporter that drops the refused report and doesn't resend it.
  • CI steps run locally, in rust:1.98-slim-bookworm:
    • cargo fmt --check and cargo clippy --all-targets --all-features with -D warnings: clean.
    • cargo test --release: 207 pass.
    • Node npm test: 19 unit, 44 integration and 29 CLI tests pass.
  • Merged with Let a dimension hold no value instead of refusing it or filing it at 0 #11: formatting and clippy are clean, 210 Rust tests pass, and the Node tests pass.

🤖 Generated with Claude Code

A batch has been all-or-nothing: one report the server refuses -- an
unknown value, a full capacity dimension, oversized props, half a
position -- fails the request with a 400 naming only that one, and
nothing in the batch lands. The Node client groups up to 1000 reports
per request, so one bad device costs up to 999 good ones; the Reporter
then puts the whole batch back and fails on it again at every flush,
because a report refused on its merits is refused every time.

`POST /positions?partial=true` applies each report on its own. The good
ones land, and the refused ones come back in `rejected`, in batch order,
as `{index, id, code, error}` -- the same code and message the 400 would
have carried. `accepted + stale + patched + unknown + rejected` equals
the batch size, and `rejected` is present, possibly empty, whenever the
flag is set, which is how a client tells a server that honoured it from
one that predates it. A body that does not parse, an oversized batch
and an overloaded server still fail the whole request: none of those
can be pinned on one report.

Opt-in, as asked in #10: two services write to the same collection with
`netcluster-client ^0.9.0`, whose acknowledgement check counts every
report it sent. A partial answer to a client that did not ask for one
would read to it as a short acknowledgement. Without the flag the
response is byte for byte what it was, including which error a batch
with several bad reports names first.

The checks now run before any value is resolved, in both modes:
position, geometry and id, then props, then values. Resolving a value on
a capacity dimension takes a slot for good, and a report refused for its
props used to take one anyway.

The Node client takes `partial` on report, patch, reportGeoJSON and
reporter, renumbers `index` from each chunk to the caller's whole list,
and counts `rejected` in its acknowledgement check. The Reporter drops a
refused report instead of requeueing it, counts it in stats.rejected,
and passes it to onError as { code: 'rejected', rejected }; the flush
itself succeeds, since the rest of the batch landed. Against a server
older than this the flag is ignored and a bad chunk is refused whole, so
the client can be upgraded first.

Refs #10

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@renatex314
renatex314 merged commit 3eacbe4 into master Oct 5, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant