feat(#1975): network-wide Scope Audit page, fed by confirmed scopes - #1976
Merged
Conversation
…1865) The ESP32 observer firmware now emits a periodic /neighbors report carrying the observer's own configured region scopes (`self`) plus, for each zero-hop neighbor, the scopes fetched via an OTA scope query. This records that CONFIRMED configuration into a new nodes.configured_scope column, kept strictly separate from the existing inferred `default_scope` (observed advert transport scope) and transported_scopes (transmissions.scope_name). Provenance is modelled explicitly rather than overloading default_scope: default_scope is overwritten on every advert observation, so writing neighbor scopes there would let the next inferred observation clobber a confirmed value. A dedicated configured_scope (+ configured_scope_at) column preserves the distinction and structurally satisfies the report contract. Report semantics honored: - Only neighbors with status=="responded" update configured_scope. A timeout is NOT evidence the scopes were cleared, so it never writes. - Absence of a neighbor is never a signal: the report is size-capped and truncates by ordering, so missing != gone — no deletes ever happen. - A responded neighbor with empty scopes is a valid "no scopes configured" statement and IS stored (the handler gates on status, not emptiness). - self scopes are keyed by origin_id (the observer node pubkey) and need no OTA query. Report pubkeys are uppercase; nodes.public_key is lowercase hex, so keys are lowercased before the UPDATE. Unknown neighbors are a no-op until a later advert creates the node. - Out-of-order reports can't clobber newer data (last-write-wins on configured_scope_at). Changes: - dbschema: additive ensureConfiguredScopeColumns migration on nodes + inactive_nodes (marker nodes_configured_scope_v1), asserted via mustCol. - ingestor: handleNeighborsReport dispatch on topic meshcore/<region>/<observer_id>/neighbors (analogous to /status); Store.UpdateNodeConfiguredScope writer. - server: PRAGMA-detect configured_scope (hasConfiguredScope) and expose it + configured_scope_at on the node read path. - UI: node-detail (nodes.js + live.js) shows a "Configured scope" row marked confirmed, with last-confirmed timestamp, distinct from the observed scope. - tests: handleNeighborsReport (responded writes, timeout/absence never clears, empty-responded stored, unknown no-op) + last-write-wins. Topic format meshcore/<region>/<observer_id>/neighbors is assumed by analogy to the /status topic; noted for reviewer confirmation against the firmware. Co-Authored-By: Claude <noreply@anthropic.com>
…-wins (#1865) UpdateNodeConfiguredScope stored the report timestamp raw and compared it lexicographically in the last-write-wins guard. Real firmware emits e.g. "2026-07-26T09:43:48.000000+00:00" (microseconds + numeric offset), and other observers may emit "Z" or a non-UTC offset like "+02:00". String-comparing those raw diverges from chronological order across observers/formats, so a stale report could overwrite a newer confirmed scope. normalizeReportTS parses the timestamp (RFC3339Nano/RFC3339) and stores it as canonical UTC RFC3339 ("...Z"), used for both storage and the comparison, so every stored configured_scope_at is either canonical or empty — never a mix. An unparseable/empty input yields "" and skips the ordering guard (writes), preserving prior behavior. New test TestUpdateNodeConfiguredScopeNormalizesAndOrdersByInstant proves a "+02:00" report that is lexicographically greater but chronologically earlier than the stored UTC value does not win, that a later "+02:00" report does, and that the firmware's microsecond/"+00:00" form canonicalizes to "Z". Co-Authored-By: Claude <noreply@anthropic.com>
…vidence Carries SaarMesh-Bot's implementation from the closed #1867 forward onto current master, 67 commits later. The branch was +386/-0 and applied with two conflicts, both from master moving underneath it: - cmd/server/db.go: #1901 replaced the per-table inline PRAGMA loops with schemaColumns(ctx, q, table). Kept master's helper and set the new flag through it (db.hasConfiguredScope = nodes["configured_scope"]) instead of reinstating the old loops. - public/nodes.js: master rewrote the transported_scopes tooltip to explain the #1902 one-byte-hop exclusion. Kept master's wording and appended the new configured_scope row. No behaviour from #1867 was changed. Its five tests pass unmodified: TestHandleNeighborsReportWritesSelfAndResponded, TestHandleNeighborsReportUnknownNeighborIsNoop, TestHandleNeighborsReportRespondedEmptyScopeIsStored, TestUpdateNodeConfiguredScopeLastWriteWins, TestUpdateNodeConfiguredScopeNormalizesAndOrdersByInstant. Both modules build and gofmt is clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
@cwichura raised this on #1865 and @dborup agreed, but #1867 was closed the next day and the change never landed: the OTA scope query returns bare region names ("dk,eu,dk-aarhus") while the rest of CoreScope carries the leading "#". The node detail page therefore showed "Own scope: #be" directly above "Configured scope: be,eu", two spellings of the same thing, and nothing could compare the two columns. Measured before choosing: every default_scope value on a live 1425-node instance starts with "#" (361 of 361). The hashRegions and hashChannels config paths in cmd/ingestor/main.go already prefix a missing "#" before deriving the region key as sha256("#name"), so the "#" is part of the name rather than display sugar. normalizeScopeList applies that same existing rule on write. Deliberate choices, each pinned by a test: - "*" is a wildcard, not a region name, and passes through untouched. - An empty value stays empty. It is a valid "responded, no scopes configured" statement and must not become "#". - Order is preserved and duplicates are not collapsed; the observer's ordering is not ours to reinterpret. - Case is left alone, because folding it would change sha256("#name") and silently retarget the scope. - Idempotent, so re-ingesting the same report after a reconnect is a no-op. CONTRACT CHANGE, flagged for review rather than buried: five assertions in issue1865_test.go were updated from "de" to "#de" and friends. They assert the stored spelling, which is exactly what this commit changes; the ordering and last-write-wins behaviour they exist to test is untouched. If the maintainers prefer to keep the raw OTA string, revert this commit alone and the merge of #1867 still stands. Not done: existing rows written before this commit keep the bare syntax. No backfill, because no instance has shipped this feature yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
The Reach page is the one view operators print and hand to someone else, and it did not say which region the node serves. With /neighbors ingestion landing in the same branch, that answer now exists and belongs here. Server: NodeReachInfo carries default_scope, configured_scope and configured_scope_at, all omitempty. reachNodeScopes does a single-row lookup gated on the schema flags the handle probed at open time, rather than widening buildNodeInfoMap's bulk SELECT, which every other consumer of that map would pay for. Every failure path returns empty strings: a reach report must still render for a node we know nothing else about. Frontend: one muted line under the subtitle. The two claims are kept apart because they are not the same statement. Scope #be observed inferred from observed advert transport Configured scope ✓ #be,#eu confirmed read back off the node itself Confirmed wins when both exist, and an empty confirmed value renders as "none configured" rather than falling back to the inferred guess: a node that answered "I have no scopes" must not be shown a contradicting guess instead. Colour is carried only by the tick, via var(--success), since the tick is the part that is evidence. Tests: seven cases in test-frontend-helpers.js, 666 to 673 passed, 0 failed. They assert the RENDERED markup through a new window.__meshcoreReachInternals handle, following map.js (#1356/#1933) rather than grepping the source. Covered: absent data renders nothing, inferred never gets the confirmed tick, confirmed wins, empty-confirmed is a statement and does not fall back, the instant lives in the title attribute, and node-controlled scope strings are HTML-escaped. Not done: the Live and Nodes views get configured_scope from #1867 as-is; no filtering or sorting by configured scope, which @cwichura raised on #1865 and explicitly called feature creep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
Adds GET /api/scope-audit and the page that reads it: one row per repeater whose configured region list is known, answering a question no existing view answers. You declare these regions, but were you actually seen forwarding them? default_scope says what a node's adverts were observed under and transported_scopes (#1751) says what it carried, but nothing lines the declared list up against observed forwarding, so a repeater configured for eight regions that only ever forwards one looks healthy everywhere else. Measured on a 1179-repeater instance, 197 repeaters with a known list, 24h: 7 rows fully observed, 78 with nothing observed, 44 mixed, 68 declaring nothing. The 78 are the signal. Some of that is a quiet region rather than a fault, which the page states in its own header rather than leaving the reader to infer, and the window is selectable (1h/24h/7d) so an operator can tell a quiet hour from a dead config. DEPENDS ON #1971 and is stacked on it. The declared side reads nodes.configured_scope, written by the observer /neighbors ingest from #1865/#1971. The original fork implementation read a node_declared_regions table fed by a custom companion-app MQTT topic that one operator runs; that would have shipped an empty page to everyone else, so the lookup is rewired. Measured before committing to that: node_declared_regions was referenced 3 times in the whole server file, all inside the declared-region lookup, while ScopeAuditForwarding, scopeAuditPrefixIndex and scopeAuditNodeIdentities have zero references to it and read data upstream already has. Both sides are compared through normScope, so the leading "#" that configured_scope carries after #1971 and a bare region name are one region. The handler already normalised the declared side with a comment explaining why, so the new source needed no change there. The Scopes column is one colour-coded list rather than the declared/not-observed pair the fork shipped: notObserved is a strict subset of declaredRegions (197 of 197 rows), so two columns printed the same set twice and left the reader to diff them. On a typical mixed row that is two lists of eight with one or two entries differing. Nav registered in all three places (top nav, bottom-nav MORE_ROUTES, nav-drawer ROUTES). Missing the latter two is how #1782/#1783 happened. Tests: 17 ported server cases, all passing against the new source, plus 8 frontend cases asserting rendered markup (673 to 681 passed, 0 failed). Two fixtures needed real changes and both are called out rather than quietly adjusted: insertDeclared now writes nodes.configured_scope and IGNORES its truncated argument, because /neighbors does not report whether its size cap fired and claiming false would invent a fact; and the hidden-name-prefix test upserts its nodes row, because upstream the declared list lives ON that row where the fork kept it in a separate table. That test's "no nodes row for unknownPk" becomes "row with a NULL name", which preserves the property under test, that a node whose name we do not know is never hidden by a name rule. TestHandleScopeAuditNoDeclaredRegionsTable is renamed to TestHandleScopeAuditWithoutConfiguredScopeColumn: it covers a real upstream state (a schema predating #1971, or an instance that has never ingested a /neighbors report), not a table that does not exist here. Full cmd/server suite green, gofmt and eslint clean. OpenAPI entry added, the completeness gate catches a route without one. Not carried over: the truncated caveat has no equivalent on this source yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
Collaborator
Author
|
Open question from the description, answered: top-level nav entry stays, not a tab under analytics. The page is network-wide rather than per-node, so it does not belong under a per-node view. Second, a correction to this PR's design that I want to make before anyone reviews it in depth. As written, the declared side reads The fix is small and it makes the feature better upstream too. @cwichura noted on #1865 that neighbor reports have landed in meshcore-packet-capture and openHop as well, so Pushing that shortly. Upstream behaviour is unchanged: only |
…ence red Two corrections from operator review, both of which change what the page is worth in practice. 1. The declared side was single-source and would have shipped an empty page AllCurrentDeclaredRegions read nodes.configured_scope only. That column is written by the observer /neighbors ingest (#1865/#1971), which requires the ESP32-native observer firmware: real, public, validated on two meshes, but deployed on a small minority of observers today. On the instance every number in this PR came from, configured_scope does not exist at all while a second collector holds 199 targets. So the page would have rendered empty on exactly the deployment that produced its evidence. The lookup now merges every confirmed-scope source the database carries, newest answer per node wins. Upstream behaviour is unchanged, because only configured_scope exists there and the merge degrades to the one-source case. node_declared_regions is probed like any other optional schema: absent means a false flag, not an error. @cwichura noted on #1865 that neighbor reports have also landed in meshcore-packet-capture and openHop, so one hard-coded source was the wrong shape regardless of which one it named. Precedence is "newest answer wins", not "source A beats source B", and is tested both ways round so nobody can satisfy it by accident. Both sources store canonical UTC RFC3339, so the lexicographic compare is chronological. An answer with no recorded instant loses to any dated answer and wins only against nothing at all. truncated survives from whichever source records it. 2. Absence is no longer red Declared-but-not-observed chips are neutral grey; only observed forwarding is green. On live data 78 of 197 rows have nothing observed, so red painted most of the table as an alarm for what is often a quiet region over a short window, which is the caveat the page header already states in words. Colour now marks the finding rather than its absence. The dead .sa-chip-missing rule is removed. Six new tests cover the merge: each source alone, newest-wins in both directions, dated beating undated while undated is still used when it is all there is, truncated surviving, and no sources being empty rather than an error. Fixtures re-probe the schema and assert the flag flipped, so a fixture that silently failed to create the table cannot make the assertions vacuous. Full cmd/server suite green. Frontend 681 passed, 0 failed. gofmt clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
…ll be empty Neither collector ships with CoreScope, so on a stock install this page is empty, and nothing on it explained why or what to run. The empty state said the table "fills in as devices drive and answer over RF", which is jargon from one specific collector and means nothing to an operator running the observer firmware instead. - A provenance line under the intro, always visible: the declared side is the repeater's own answer read back off the node, the observed side is forwarding CoreScope already sees in its own traffic. That distinction is the entire point of the page and it was never stated anywhere. - The empty state now carries the full explanation, because on most deployments it IS the page. It names both collectors with links (the ESP32 neighbour-report firmware and the CoreDrive RX app), says an empty table is the normal state rather than a fault, gives the newest-answer-wins rule now that there are two sources, and states what justifies the page existing: nothing else in CoreScope knows which regions a repeater is CONFIGURED for, only which ones its traffic was seen under. emptyStateHtml is extracted and exported rather than left inline so it can be asserted rather than grepped. Six tests: both collectors named, both linked, empty-is-normal stated, precedence stated, the configured-versus-observed distinction present, and the old drive-around wording gone. 681 to 687 passed, 0 failed. Maintainers: naming a specific companion app in shipped UI copy is a judgement call. It is a public repo and it is one of the two things that actually fills this table, so leaving it out would make the empty state less useful to the people most likely to read it. Say the word and it becomes a generic "or another collector that records a confirmed scope list". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
The links to both collectors lived only in the empty state, which by definition never renders on an instance that HAS data. On the deployment this page was built against, 197 rows from 199 collected targets, the provenance note named the two collectors in prose and linked neither. The readers best placed to act on it were exactly the ones who could not. sourcesLineHtml now carries both links and is extracted and exported so the gap is asserted rather than assumed. Four tests: two links present, rel="noopener" on both, the declared-versus-observed distinction stated, and the newest-answer-wins rule visible to a reader who never sees the empty state. The empty-state assertion now accepts either the repo URL or a fork's own hosted instance, rather than pinning one deployment's URL into a shared test. 681 to 691 passed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
…udit # Conflicts: # cmd/server/db.go # test-frontend-helpers.js
…ctation test-nav-priority-1391-e2e.js asserts the More menu contains EXACTLY the non-high-priority routes, so adding a nav link fails it until that list is updated. CI caught it: 15/30 passed, with the diff naming '#/scope-audit' as an unexpected extra. This is the fourth place a nav link has to be registered, after index.html, bottom-nav.js MORE_ROUTES and nav-drawer.js ROUTES, and the only one of the four that is not recognisable as nav wiring from its filename. Noted in a comment at the list so the next person does not have to learn it from a red build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
…udit # Conflicts: # test-frontend-helpers.js
I picked `ph-shield-check` for the Scope Audit nav entry without checking that
the project ships it. It does not: the sprite holds 99 symbols and that is not
one of them, so both mobile nav surfaces rendered a reference that resolves to
nothing.
CI caught it, which is the point of that gate:
✗ (i) 1 sprite ref(s) not resolved: ph-shield-check
test-issue-1648-m2-icons-e2e.js: 20 passed, 1 failed
Replaced with `ph-clipboard-text`, verified present in
public/icons/phosphor-sprite.svg and not already used by another nav route, so
the entry stays distinguishable in the More sheet and the drawer.
The lesson is the ordinary one: an icon name that sounds right is not evidence
the icon exists. The sprite is a fixed list and it takes one grep to check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1975. Stacked on #1971 and based on its branch, so the diff here shows only the new work. Merge #1971 first and this retargets to master cleanly.
What it adds
GET /api/scope-auditand the page that reads it. One row per repeater whose configured region list is known, answering a question nothing else answers: you declare these regions, but were you actually seen forwarding them?default_scopesays what a node's adverts were observed under andtransported_scopes(#1751) says what it carried, but nothing lines the declared list up against observed forwarding. A repeater configured for eight regions that only ever forwards one looks healthy in every existing view.Measured on a 1179-repeater instance, 197 repeaters with a known list, 24h window:
The window is selectable (1h / 24h / 7d) and the page says in its own header that a short window makes absence weak evidence, rather than leaving the reader to infer it.
The data source, and why the fork version could not ship as-is
On the fork the declared list comes from a
node_declared_regionstable fed by a custom companion-app MQTT topic. That would have shipped an empty page to everyone who does not run that app, so the lookup is rewired tonodes.configured_scopefrom #1971, read back off the node itself by the observer/neighborsreport.I measured how much that costs before committing to it:
node_declared_regionswas referenced 3 times in the whole server file, all inside the declared-region lookup.ScopeAuditForwarding,scopeAuditPrefixIndexandscopeAuditNodeIdentitieshave zero references to it and read data upstream already has (scope_nameappears in 9 server files,unscopedin 11). So this is one data source swapped, not a feature rebuilt.Both sides compare through
normScope, so the leading#thatconfigured_scopecarries after #1971 and a bare region name are the same region. The handler already normalised the declared side, with a comment explaining why, so the new source needed no change there.One deliberate difference from the fork's UI
The Scopes column is a single colour-coded list rather than the DECLARED / NOT OBSERVED pair.
notObservedis a strict subset ofdeclaredRegions(197 of 197 rows), so two columns printed the same set twice and left the reader to diff them: on a typical mixed row, two lists of eight with one or two entries differing. Green is observed, red is declared-but-not-observed.Nav
Registered in all three places: the top nav in
index.html,MORE_ROUTESinbottom-nav.js, andROUTESinnav-drawer.js. Missing the latter two is exactly how #1782 and #1783 happened, andnav-drawer.jssays "keep in sync with bottom-nav.js" in a comment for that reason.Tests
17 server cases ported and passing against the new source, plus 8 frontend cases asserting the rendered markup through a
window.__meshcoreScopeAuditInternalshandle (673 → 681 passed, 0 failed).Two fixtures needed real changes, called out rather than quietly adjusted:
insertDeclarednow writesnodes.configured_scopeand ignores itstruncatedargument. The old source recorded whether an answer was cut short;/neighborshas its own size cap but does not report whether it fired, so claimingfalsewould invent a fact. The argument stays in the signature so the ported cases still document which fixtures meant "this answer was incomplete".TestHandleScopeAuditFiltersHiddenNamePrefixupserts its nodes row. Upstream the declared list lives on that row, where the fork kept it in a separate table, so a plain INSERT now collides. Its "deliberately no nodes row for unknownPk" becomes "row with a NULL name". The property under test is unchanged: a node whose name we do not know must never be hidden by a name-prefix rule.TestHandleScopeAuditNoDeclaredRegionsTableis renamed toTestHandleScopeAuditWithoutConfiguredScopeColumn, because it now covers a real upstream state (a schema predating #1971, or an instance that has never ingested a/neighborsreport) rather than a table that does not exist here. I checked it is not vacuous:hasConfiguredScopeis false in that fixture and the handler returns an empty audit rather than erroring.Full
cmd/serversuite green,gofmtandeslintclean. An OpenAPI entry is included;TestOpenAPICompletenesscatches a route without one, which is how I found I had missed it.Not carried over
The
truncatedcaveat has no equivalent on this source yet. #1971 could grow the field later and the page can then show it; inventing a value now would be worse than omitting the caveat.Open question from #1975 that nobody has answered yet
Top-level nav entry, as here, or a tab under analytics? I picked top-level because the page is network-wide rather than per-node, but I have no strong view and it is a two-line change.