Repository navigation
Apply a batch per report with ?partial=true, naming the ones refused - #12
Merged
Merged
Conversation
A batch has been all-or-nothing: one report the server refuses -- an
unknown value, a full capacity dimension, oversized props, half a
position -- fails the request with a 400 naming only that one, and
nothing in the batch lands. The Node client groups up to 1000 reports
per request, so one bad device costs up to 999 good ones; the Reporter
then puts the whole batch back and fails on it again at every flush,
because a report refused on its merits is refused every time.
`POST /positions?partial=true` applies each report on its own. The good
ones land, and the refused ones come back in `rejected`, in batch order,
as `{index, id, code, error}` -- the same code and message the 400 would
have carried. `accepted + stale + patched + unknown + rejected` equals
the batch size, and `rejected` is present, possibly empty, whenever the
flag is set, which is how a client tells a server that honoured it from
one that predates it. A body that does not parse, an oversized batch
and an overloaded server still fail the whole request: none of those
can be pinned on one report.
Opt-in, as asked in #10: two services write to the same collection with
`netcluster-client ^0.9.0`, whose acknowledgement check counts every
report it sent. A partial answer to a client that did not ask for one
would read to it as a short acknowledgement. Without the flag the
response is byte for byte what it was, including which error a batch
with several bad reports names first.
The checks now run before any value is resolved, in both modes:
position, geometry and id, then props, then values. Resolving a value on
a capacity dimension takes a slot for good, and a report refused for its
props used to take one anyway.
The Node client takes `partial` on report, patch, reportGeoJSON and
reporter, renumbers `index` from each chunk to the caller's whole list,
and counts `rejected` in its acknowledgement check. The Reporter drops a
refused report instead of requeueing it, counts it in stats.rejected,
and passes it to onError as { code: 'rejected', rejected }; the flush
itself succeeds, since the rest of the batch landed. Against a server
older than this the flag is ignored and a bad chunk is refused whole, so
the client can be upgraded first.
Refs #10
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Point 1 of #10, as agreed in the discussion there: one report the server refuses no longer has to cost the rest of its batch. Opt-in, so existing
^0.9.0clients see no change. Independent of #11; the two merge cleanly, and the merged result passes everything (see Testing). Intended for0.10.0with #11.Problem
A batch is all-or-nothing. One refused report (an unknown value, a full
capacitydimension, oversizedprops, half a position) fails the request with a 400 naming only that report, and nothing in the batch lands. The Node client sends up to 1000 reports per request, so one bad device costs up to 999 good ones.It's worse in the
Reporter: it puts a failed batch back in its queue, and a report refused on its merits is refused every time. So one bad device makes every later flush fail and blocks every device queued with it.Server:
POST /positions?partial=truerejected, in batch order:{"accepted":2,"stale":0,"devices":2, "rejected":[{"index":1,"id":"truck-9","code":"bad_request","error":"device \"truck-9\": unknown value \"idel\" ..."}]}codeanderrorare exactly what the 400 would have carried.idisnullfor a GeoJSON Feature that carried no id.accepted + stale + patched + unknown + rejected= batch size.rejectedis present, possibly empty, whenever the flag is set. That's how a client tells a server that honoured it from one that predates it.?partial=set to anything other thantrue/false.Without the flag: a batch is still all-or-nothing, and a batch with one bad report gets the same 400, message and code as before. The existing tests that pin that behaviour pass unchanged.
Change in both modes: reports are now checked before any value is resolved, in this order: position/geometry/id, then
props, then values. Resolving a value on acapacitydimension takes a slot permanently, and a report refused for itspropsused to take one anyway. Covered bya_refused_device_takes_no_capacity_slot.One visible side effect: when a batch has several bad reports of different kinds, the 400 may name a different one than before. Example: a bad value at item 1 and oversized
propsat item 2 used to name item 1, and now name item 2. (The commit message says the first-named error is unchanged too; that's wrong in this case.)Node client
partial: trueoption: available onreport,patch,reportGeoJSONandreporter. The default staysfalse.rejected, withindexrenumbered from each chunk to the caller's whole list.checkAckcountsrejected.Reporter:stats.rejected.onErroras aNetClusterErrorwithbody: { code: 'rejected', rejected }.index.d.tshas the new types (WriteOptions,RejectedReport), and both READMEs describe the behaviour.Testing
tests/partial.rs(8 tests), covering:stale/patched/unknown.verify().report,patch,reportGeoJSON, and aReporterthat drops the refused report and doesn't resend it.rust:1.98-slim-bookworm:cargo fmt --checkandcargo clippy --all-targets --all-featureswith-D warnings: clean.cargo test --release: 207 pass.npm test: 19 unit, 44 integration and 29 CLI tests pass.🤖 Generated with Claude Code