feat(observe): accept batched observations (backwards-compatible) - #1
Merged
Conversation
POST /observe/{hour} now also accepts a batch envelope:
{ "protocol": 2, "observations": [ {location_id, stop_id, layer, fp, nk}, ... ] }
folding one session's fingerprinted stops at a location in a single request
(protocol carried once on the envelope, at most MAX_BATCH_OBSERVATIONS=24).
This lets Dragonite collapse its per-stop POSTs into one per session, cutting
the per-UTC-hour observe volume that drives the coordinator's rollover spike.
Backwards-compatible by construction: a single-observation body (no
`observations` array) is still accepted unchanged, so old and new Dragonite
instances interoperate. The route sniffs the body shape — an `observations`
array => batch, anything else => single. This is why the coordinator MUST be
deployed before any batch-sending client ships: an old coordinator would 400 an
array body and lose those observations.
- validate.ts: shared validateProtocol/validateFields; new validateBatch
(whole-batch reject on a malformed item — the client builds the batch, so a
bad item is a client bug).
- http.ts: body-shape sniff; batch folds every item in one hour transaction and
skips per-item semantic outcomes (unknown_stop/capped/killed) best-effort so
one dead stop can't drop the rest; MAX_OBSERVE_BODY_BYTES 4096 -> 8192.
- No protocol bump: the fingerprint recipe is unchanged, so MIN_PROTOCOL stays
2; bumping it would 426-reject old instances mid-rollout.
Tests: 6 new batch cases (folds all / envelope protocol / below-min 426 /
empty+malformed+too-many 400 / best-effort skip of unknown stop / single body
still works). Full suite 34 passing, typecheck + lint clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
POST /observe/{hour}now also accepts a batch envelope, folding one client session's fingerprinted stops at a location in a single request instead of one POST per stop:{ "protocol": 2, "observations": [ { "location_id": 1, "stop_id": "fort-a", "layer": "no_ar", "fp": "a1b2c3d4e5f60718", "nk": false }, { "location_id": 1, "stop_id": "fort-b", "layer": "no_ar", "fp": "1122334455667788", "nk": false } ] }Why
Dragonite currently POSTs one observation per stop. At each UTC-hour rollover the coordinator's per-hour state resets, so every session re-establishes values from scratch — a front-loaded burst of POSTs that (together with the idle-GET herd, already jittered on the client) makes the rollover spike. Batching collapses a session's multi-stop POSTs into one, cutting that burst exactly where it's worst.
Backwards-compatible by construction
observationsarray ⇒ batch; anything else ⇒ the existing single-observation path, untouched.MIN_PROTOCOLstays2— bumping it would426-reject old instances mid-rollout.400an array body (no top-levellocation_id) and silently lose those observations. This PR is purely additive (new array path; single path untouched), so it's safe to deploy while every existing instance is still single-posting.Changes
validate.ts— extracted sharedvalidateProtocol/validateFields; newvalidateBatch(protocol once on the envelope; a malformed item rejects the whole batch, since the client builds it — a bad item is a client bug worth surfacing).http.ts— body-shape sniff; a batch folds every item in one hour transaction and skips per-item semantic outcomes (unknown_stop/capped/killed) best-effort so one dead stop can't drop the rest of the session;MAX_OBSERVE_BODY_BYTES4096 → 8192.types.ts—MAX_BATCH_OBSERVATIONS = 24(a location has ≤10 expected stops; generous defensive bound).README.md— documents the batch body, the 8KB cap, and the deploy-order note.Tests
6 new cases in
test/http.test.ts: folds all items + echoes once; envelope-level protocol; below-min ⇒426; empty / malformed / too-many ⇒400; best-effort skip of an unknown stop (good item still folds,200); and the single (non-batch) body still works unchanged. Full suite 34 passing,typecheck+lintclean.🤖 Generated with Claude Code