Skip to content

fix(gateway): every observability sink leaves under one closed schema (N107b) - #538

Open
LamaSu wants to merge 140 commits into
masterfrom
fix/n107b-closed-schema
Open

LamaSu wants to merge 140 commits into
masterfrom
fix/n107b-closed-schema

Conversation

@LamaSu

@LamaSu LamaSu commented Oct 3, 2026

Copy link
Copy Markdown
Owner

Summary

N107b applies the PR steward's strict observability ruling (#5315/#5319, DECISIONS.md 10/03): no request-controlled value reaches any sink or console line, except as a keyed truncated hash or a coarse class. It is stacked on #514 (N107), which is frozen at b8a3b5c0. Retarget it to master after #514 merges. #441 rebases onto this PR.

The schema is enforced once, at the five chokepoints every producer passes through, rather than at about 300 call sites:

  • Shared schema (observability/closed-schema.ts). keyedHash is an HMAC under PCC_TELEMETRY_KEY; without it, each process uses its own random key. User-Agent becomes one of bot, browser, sdk or unknown. Route values must be registered route templates. Free text is kept only when the whole string is one of the gateway's own string literals, or a registered value. Anything else becomes a keyed hash.
  • The five chokepoints (observability/closed-sinks.ts):
    • PostHog: event names are closed, the distinctId is hashed, and properties are closed by key and value. No body field leaves under any name.
    • Audit: actor, resource id and address are hashed, the User-Agent becomes a class, and metadata is closed.
    • Logger: a formatter plus a check on every written line, so child loggers cannot bypass it. The request serializer logs only route, method and status. requestIdHeader is set to false.
    • Console: routed through the closed schema inside a request scope.
    • Sentry: collects no request data at the source, and every event, transaction, span and breadcrumb is rebuilt from closed fields.
  • The two tracked MEDIUMs from fix(gateway): the security monitor and payment stats keep no request content (N107) #514 r2. acceptLanguage is dropped, and the country is an ISO-3166 code or other. The MPP failure log carries a fixed code and the error class only.

Tests

  • __tests__/observability/every-position.test.ts sends a marker in every request position through the real createGateway: the query, the fragment, encoded separators, every header, cookies, form, JSON and prose bodies, and path segments. It asserts that the Sentry, log, console, audit and PostHog outputs contain none of them.
    • At b8a3b5c0 it found 18 leaked markers in Sentry, 9 in logs, 8 in audit, 2 in console and 2 in PostHog.
  • closed-schema.test.ts holds unit tests for each chokepoint.
  • 12 mutation checks: each chokepoint's protection, when removed, fails a test.
  • The full gateway suite at the head runs 211 files, with 3730 passed. tsc --noEmit is clean.

Consequences (for gateway and the operator)

  • Data now hashed at a boundary. Producer code is unchanged, but these values leave as hashes or are dropped:
    • provision: email, name, capability;
    • near: quoteId, workflowId;
    • agent-feedback and feedback text;
    • evidence-encrypted, marketplace, onboard, setup, zk-proofs and lit-provision;
    • the job, kernel and settlement facades;
    • in audit rows: actor, resource id and address.
  • Interpolated log and Sentry messages become hashes.
  • Operator: set PCC_TELEMETRY_KEY (at least 32 characters), so that hashes line up across restarts and instances.
  • Readable agent feedback would need a dedicated table. That is a schema change, so it is the operator's call.

Agent: pcc-readmodels (c255d7dc).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Sbd5dpwvmRsJqdff9zvNW6

…s (F3)

Gateway #2831: align the /api/jobs/:id read family with #353's predicate through
one shared helper, not copies. Before, any API key read any job's record,
status, evidence, drift alerts and settlement; only /execution checked the caller.

readmodels/job-read-gate.ts gateJobRead reads the job row, applies
TENANT_ENFORCE, and authorizes with authorizeJobRead: an admin (X-Admin-Key), the
kernel operator, or the recorded buyer. refuseJobRead answers 401 for an anonymous
caller, 503 when the record cannot be read, and otherwise the ROUTE'S OWN 404 for a
missing job, so 'not yours' and 'no such job' are byte-identical.

Gated: GET /api/jobs/:jobId, /execution (now through the gate), /status,
/settlement, /evidence and /drift-alerts; GET /api/settlement/:jobId (via the
legacy settlement loader); and the job-id form of GET /api/evidence/:jobId. The
hash form stays content-addressed for the oracle.

agent: pcc-readmodels (c255d7dc)
readmodels/job-read-gate.test.ts (42) covers all 8 reads: anonymous 401; a
stranger gets the byte-identical answer for a missing job; the kernel operator
(case-insensitive), the recorded buyer and an admin read; a wrong admin key is a
stranger; another tenant's job is 404 under TENANT_ENFORCE; a failed row read is
503; the evidence-by-hash form stays content-addressed.

Existing route tests that were not about authorization now read as a party through
helpers/job-read-party.ts (the seeded kernel operator, a stand-in for the API gate).
The facade-mocked compliance wiring test mocks the gate open.

Gateway 3107 passed / 6 skipped / 0 failed. Mutation check: 8/8 killed.

agent: pcc-readmodels (c255d7dc)
CLAUDE.md and docs/AGENT_INTEGRATION.md state the rule for the job read family:
an admin, the kernel operator or the recorded buyer; a 404 identical to a missing
job for anyone else; 401 without auth.

agent: pcc-readmodels (c255d7dc)
Brings #313 @8f946499 and #353's evidence fix (read through the job, not
the never-written evidence_bundles.tenant_id). No conflicts; the gated
execution route drops its now-unused tenant variable, since gateJobRead
already refuses a job outside the caller's tenant.

agent: pcc-readmodels (c255d7dc)
…#403

#353's review-r3 fixes change the predicate this family's gate is built on, so the
merge adapts the gate. The merged tree does not build without it.

- gateJobRead: identity first (precheckJobRead). No credential is 401 and a credential
  without a PROVEN wallet (WP-A's req.provenWallet, SIWE) is 403 identity_unverified,
  both before the job row is read. Then the job, TENANT_ENFORCE, and
  authorizeJobRead(job, provenWallet): kernel operator or recorded buyer, compared as
  addresses. An operatorId or email is never trusted as an identity.
- refuseJobRead and the two legacy settlement routes answer 401 and 403 with the shared
  JOB_READ_REFUSAL bodies, the same for every job id.
- Conflict in routes/jobs.ts: the execution route keeps #403's gate call.
- Tests: the stand-in gate sets a proven wallet for a wallet principal
  (helpers/job-read-party provenWalletFor), the F3 buyer is a wallet, and every route in
  the family pins the new 403 for a key that claims the operator's or buyer's id without
  proof, identical for a missing job.

Effect: until WP-A (#326) merges, no caller has a proven wallet, so the whole job read
family is admin-only (fail closed). After it, email-provisioned and legacy keys cannot
read job records. That is the operator's call (decision posted with the #353 triage).

Tests: gateway 3131/6/0, dashboard 258; tsc clean. Gate mutation check: 5/5 killed.

agent: pcc-readmodels (c255d7dc)
…job-read-family-auth

#382's round-1 fixes come forward:
- milestones[].forThisJob from the attributed milestone, plus association;
- the corrected route comments;
- the job-party test helper. It was already byte-identical here.

Conflicts resolved to this branch's family gate: gateJobRead and its
unauthenticated, identity_unverified, not_found and unavailable kinds
replace #382's inline precheck and authorize. Both implement #353's rule,
and the gate is the shared one. The legacy-settlement integration tests
keep #382's explicit stand-in gate (identity only from headers), so its
401 and 403 tests hold. Their reads name the kernel-nyc operator
explicitly.

agent: pcc-readmodels (c255d7dc)
…read (F3)

Astra round 1 on #403, CRITICAL 1. Reproduced first: 18 tests fail at
74225e1. GET /api/jobs listed every job to any caller, because the gate
covered single-job reads only.

- jobsReadableBy (readmodels/job-execution.ts) gives the jobs a proven
  wallet may read. The rule is authorizeJobRead's: the wallet operates
  the job's kernel, or is the buyer recorded by the job's single
  negotiation session. It uses one read of the kernels and one of the
  sessions.
- jobReadScopeOf (job-read-gate.ts):
  - refuses as the gate does (401, 403, 503);
  - an admin reads every job, within its tenant under TENANT_ENFORCE;
  - a proven wallet reads jobsReadableBy's jobs;
  - a tenant-less job matches only a tenant-less caller, as in the gate.
- jobRecordFilterOf serves routes that mix records of many jobs. A record
  naming a job is kept only when that job is readable. A record naming
  no job is kept.
- gateJobRecordRead gates a record found by its own id (an evidence
  bundle) on its job, checking identity first.
- refuseJobRead also takes a scope's refusal.
- GET /api/jobs filters by the scope before counting and paging
  (JobFilters.jobIds).
- Timing (MEDIUM): a missing job, another tenant's job and a stranger's
  job now make the same kernel and session reads.

The existing /api/jobs list test reads as the seeded operator's proven
wallet.

agent: pcc-readmodels (c255d7dc)
…n the job read gate (F3)

Astra round 1 on #403, CRITICAL 1: the gate did not cover the whole
job-read surface. Reproduced first.

- /api/telemetry/pipeline/:jobId: the gate. A job the caller may not read
  gets the empty timeline a missing job gets.
- /api/telemetry/active and /jobs: the scope.
- /api/telemetry/logs:
  - with a jobId, the gate (no lines for an unreadable job);
  - without one, the record filter;
  - the limit applies after filtering.
- /api/telemetry/logs/stream: the history and the live fan-out go
  through the caller's record filter. The filter is fixed at connect
  time, so it fails closed.
- /api/batches/by-job/:jobId: the gate. An unreadable job gets the empty
  list a missing job gets.
- /api/sensors/readings/:channel: the gate with a jobId, otherwise the
  record filter.
- /api/compliance/evidence/:bundleId and /tier-compliance:
  gateJobRecordRead on the bundle's job. An unreadable one gets the
  missing-bundle 404.
- /api/query: the job_status and job_history intents answer only
  readable jobs.
- /api/print-and-mail/:jobId: the identity precheck, then admin only.
  Its :jobId is a courier job in the job-offers store, whose poster and
  driver ids are self-declared, so no party rule can be proven yet. This
  goes to gateway with carrier/Lob (CRITICAL 3).

Existing tests: the compliance-routes gate mock gains gateJobRecordRead,
and print-and-mail reads as an admin.

agent: pcc-readmodels (c255d7dc)
…WE session proves its wallet (F3)

Astra round 1 on #403, CRITICAL 2. Reproduced first.
/sse/stream/job/:jobId had its own ownership check, off by default, which
trusted an API key's self-declared operatorId.

- After SSE auth, the route runs gateJobRead:
  - 401 and 403 as the gate refuses;
  - a job the caller may not read gets the missing-job 404.
- resolveSSEAuth sets req.provenWallet from a SIWE session (cookie,
  bearer or ?token=), lowercased, as WP-A does for HTTP. An API key sets
  none.
- The unused isJobOwnershipCheckEnabled/checkJobOwnership path is removed.

agent: pcc-readmodels (c255d7dc)
…a missing job (F3)

f3-job-read-surface.test.ts has 21 tests. The 18 written first all fail
at 74225e1. They cover:
- no credential (401) and no proven wallet (403) on 8 routes;
- a stranger reads exactly what a missing job gives, on each gated route;
- the compliance bundle routes, and /api/query's job intents;
- SSE: anonymous gets 401 and a stranger 404, before any event;
- timing: a stranger's job and a missing job make the same reads.

Three were added during the fix:
- TENANT_ENFORCE: a job moved to another tenant leaves the party's list,
  and its /execution returns 404;
- the live log stream, over a real socket: a stranger and an anonymous
  caller get no line of another job, and the party gets its own;
- the party of a job still receives that job's events on the per-job
  SSE stream.

The print-and-mail case follows the admin-only design: a stranger and a
party both get the missing-job 404, and an admin gets 200.

agent: pcc-readmodels (c255d7dc)
…ches and nested bindings (F3 r3)

Cross-family review r2 of #403 (rm-f3-403-r2-4988a191, DO-NOT-SHIP)
found four bypasses. f3-r3-bypasses.test.ts reproduced all four at
4988a19 (11 failed, 2 passed). Each is fixed here:

- CRITICAL: the kernel, device and batch SSE streams only called
  resolveSSEAuth, which passes anonymous callers when SSE_AUTH_REQUIRED
  is unset. They carry job-bound sensor readings. They now take the
  kernel's read rule (gateKernelRead): an admin, or the kernel's
  operator with a proven wallet. No credential gets 401, an unproven
  one 403, and anyone else the 404 an unknown kernel, device or batch
  gets, before any subscription.
- CRITICAL: GET /api/batches and /api/batches/:batchId returned every
  batch, and its slots name each sample's job and buyer. The list is
  now the operated kernels' batches (all of them for an admin). The
  detail goes to an admin or the operator; anyone else gets the
  missing batch's answer. By job, a buyer sees only its own job's
  slots of a shared batch.
- HIGH: jobRecordFilterOf read only a top-level jobId. recordBindingsOf
  now reads job and kernel bindings at any depth, in each key spelling.
  A malformed binding fails closed, and a record that names neither is
  an admin's, since its text can name any job. The live log stream uses
  the same rule. Anonymous callers get 401 on the mixed-record routes.
- MEDIUM: the authorization reads were keyed by the requested job.
  jobReaderOf now reads only by the caller's wallet: the kernels it
  operates, and the jobs whose single session names it. A refusal
  costs the same reads whether the job exists or not. Only the
  requested row's own lookup is keyed by the request.

Also (astra r2 on #382, MEDIUM): both legacy settlement routes now have
a test pinning the generic 503 when the job read or the authorization
read throws, for a party, a stranger and a missing job alike.

agent: pcc-readmodels (c255d7dc)
…nd batch streams (F3 r3)

CLAUDE.md's read-access paragraph, the SSE tables (in CLAUDE.md and
docs/AGENT_INTEGRATION.md) and AGENT-SKILLS.md's rows now say who may
read a kernel's batches and its streams, and how log lines and sensor
readings are filtered per record.

agent: pcc-readmodels (c255d7dc)
…the caller may read (F3 r3)

Found while fixing round 3. Reproduced at b170cfa: anonymous got 200
from GET /api/kernels/kernel-nyc/jobs, and the public kernel detail
embedded job-001 (f3-r3-kernels-repro-b170cfaf.log).

- GET /api/kernels/:kernelId/jobs now checks identity first, before the
  jobs are read (jobReadScopeOf): no credential is 401 and an unproven
  one 403. A proven wallet gets the kernel's jobs it may read; an admin
  gets all of them.
- GET /api/kernels/:kernelId stays public. Its recentJobs hold only the
  jobs the caller may read. recentJobsScope ("all",
  "readable_by_caller" or "unavailable") says what the list covers, so
  an empty list never reads as "no jobs".
- The other getById callers (lob, carrier, compose) read only
  operatorAddress.
- The API tables in CLAUDE.md and docs/AGENT_INTEGRATION.md say so.

agent: pcc-readmodels (c255d7dc)
… batches show only the opportunity, filters judge records as sent (F3 r4)

Cross-family review r3 of #403 (rm-f3-403-r3-f004b709, DO-NOT-SHIP).
f3-r4-bypasses.test.ts at f004b70: 7 of 11 failed. The 3 positive
cases and MEDIUM 6 passed.

- CRITICAL 1 (kernel records ignored TENANT_ENFORCE): each job-bound
  part of a kernel's record now follows the tenant-aware job rule
  (jobReadScopeOf):
  - jobPartScopeOf projects batch slots in the list, the detail and
    by-job;
  - streamEventFilterOf drops any job, kernel, device or batch stream
    event, and any batch-detail event, that names a job the caller may
    not read.
  An admin without a tenant keeps everything.
- CRITICAL 2 (shared batches returned every claim publicly): anyone now
  gets sharedBatchFace, the opportunity with claimedSlotCount and no
  claims. Only an admin without a tenant, or the kernel's operator,
  sees the claims, on /open, /:batchId and the claimedBy list of
  /availability.
- HIGH 3 (?jobId= reads skipped the nested filter): logs, sensor
  readings and the pipeline timeline now apply the gate and the record
  filter together. The pipeline timeline is the same class of path;
  it reproduced against f004b70's telemetry.ts.
- HIGH 4 (the filter saw the live object, not what was sent): asSent
  and keepAsSent make every filter judge the record's JSON text parsed
  back, and every route sends that form. This covers REST logs and
  sensor readings, the log stream's history and live fan-out, all four
  SSE topics, and batch-detail events. A toJSON or a getter can no
  longer make the inspected and the sent records differ.
- MEDIUM 5 (record-by-id timing): gateJobRecordRead now refuses a
  non-party from the record's own job and kernel, using the caller's
  wallet-keyed reader, before any job row is read. Parties and admins
  then go through gateJobRead. A stranger's refusal makes the same
  reads whether the bundle exists or not.
- MEDIUM 6 (unmatched SSE paths leak slots): NOT REPRODUCED, so no code
  change. topicSSE is registered encapsulated (server.ts:844), so its
  onRequest hook runs only for its own matched routes. A pin test
  shows 20 unmatched requests followed by a valid stream: 200.

agent: pcc-readmodels (c255d7dc)
… read, as sent; claim release is gated (F3 r5)

Cross-family review r4 of #403 (rm-f3-403-r4-aa7b009a, DO-NOT-SHIP).
f3-r5-bypasses.test.ts at aa7b009: 6 of 7 failed. The 1 that passed
is the unscoped-admin positive.

- CRITICAL 1 (job-bound data outside slots): new readmodels/batch-read.ts
  (batchViewFor, batchEventVisible) decides each batch subrecord by its
  job, with the batch judged as sent:
  - a slot is kept only when every job it names is readable;
  - runConfig shows only when the caller sees every slot and runConfig
    names only readable jobs (otherwise null, with runConfigWithheld);
  - a sample event, which names only its slot, follows that slot;
  - a batch-level event shows only for a batch seen whole.
  This applies to the list, the detail, by-job (a buyer: its job's
  slots only) and the batch SSE stream, which decides each event,
  replayed ones included, against the batch as it is then.
- HIGH 2 (slots filtered live, sent in another form): batchViewFor
  works on asSent(batch). A slot whose toJSON or nested binding names
  another job is judged on that form.
- HIGH 3 (anyone could release a claim and get it back):
  DELETE /api/batches/shared/:batchId/claim/:claimId checks identity
  first (401/403). Only the kernel's operator, an admin without a
  tenant, or the claimant the claim names (proven wallet ==
  agentId) may release it. Anyone else gets "Claim not found".
- MEDIUM 4 (tenant-refused admin timing): under TENANT_ENFORCE, a
  record carrying its own tenantId is refused on it before any job
  row is read.
- MEDIUM 5 (logs sources facet): sources now names only the sources
  of the lines returned.

CLAUDE.md's read-access paragraph says so.

agent: pcc-readmodels (c255d7dc)
… and batch writes need the right identity (F3, review r5 of #403)

Cross-family review r5 of #403 (rm-f3-403-r5-04cdf72d, DO-NOT-SHIP). Each
finding was reproduced at 04cdf72, before any fix, by
f3-r6-bypasses.test.ts: 7 of 7 tests failed.

- CRITICAL 1. A reading of a sample, or a batch-level event, reached a
  caller who may not read its job. recordOwnersOf now takes the owners from
  the live BatchTracker (batch-ownership.ts):
  - a slot or sample is owned by its slot's live job;
  - a record that names a batch but none of its slots is owned by every
    job of the batch, on that batch's own stream too;
  - a job the record names only adds an owner.
  Unknown slots or batches and malformed bindings fail closed. The stream
  filter and the record filter both use it, so an event-type list no
  longer decides what is batch-level.
- CRITICAL 2. A slot whose toJSON named another job was kept. A batch is
  now sent as a typed projection of the live batch: only the spec's
  fields, each read once and kept only as a string. A slot is kept only
  when its live job is readable.
- HIGH 3. Anyone could claim shared-batch slots for any claimant. A claim
  now needs a proven wallet and is made for that wallet; an admin names
  the claimant. slotCount, preferredIndices and sampleLabels are typed.
- Found while fixing, in the same classes:
  - opening a shared batch is now the kernel operator's or an admin's;
  - so is adding a slot, for a job of that kernel the caller may read, and
    only the slot's typed fields are stored;
  - a sensor anomaly or channel aggregate dropped its readings' batch and
    sample, so anomalies and aggregates now carry every reading's source;
  - GET /api/sensors/anomalies and GET /api/sensors/aggregates/:channel had
    no read gate. Both now go through the record filter, and an aggregate
    is computed only over the readings the caller may read.

The r4 test's readings now name their sample, since one naming none is the
batch's. The r4 and r5 setups now open the shared batch and its claims as
an admin.

agent: pcc-readmodels (c255d7dc)
…tch-level source keeps the whole batch's owners (F3, review r6 of #403)

Cross-family review r6 of #403 (rm-f3-403-r6-43ec901a, DO-NOT-SHIP),
CRITICAL 1. recordBindingsOf flattened every slot and batch a record named,
and recordOwnersOf skipped the whole batch when ANY named slot belonged to
it. A rate-of-change or flatline anomaly over a batch-level reading and
tenant A's sample reading of the same batch was therefore owned by tenant
A's job alone.

Reproduced at 43ec901 by f3-r7-bypasses.test.ts, before any fix: 3 of 3
failed (the verdict's rate case, a flatline case, and the resolver).

The fix: the walker now records the batches and slots each object names
itself, and recordOwnersOf resolves each object on its own. A batch an
object names is covered only by a slot of that batch named in the same
object; otherwise every job of the batch owns the record. On a batch's own
stream, the record's top-level object is tied to the batch. A sample named
only in a nested object therefore no longer covers a batch its parent
names. recordBindingsOf's flat result is unchanged.

agent: pcc-readmodels (c255d7dc)
… (N107b)

The PR steward's ruling of 10/03 (bus #5315 and #5319, DECISIONS.md): no request-controlled value
reaches any sink or console line, except as a keyed hash or a coarse class. The schema is enforced
at the five chokepoints every producer passes through, not at each of about 300 call sites:

- observability/closed-schema.ts holds the rules:
  - identifiers leave as keyedHash, an HMAC under PCC_TELEMETRY_KEY, falling back to a per-process
    key with one boot warning;
  - a string leaves as itself only when it is one of the gateway's own string literals (scanned
    once from its source), a registered closed value (route templates, ISO codes) or an ISO
    timestamp; anything else, prose included, leaves as its keyed hash;
  - object keys follow the same rule;
  - numbers leave only under server-side metric names;
  - errors leave as their class, code and code frames;
  - the User-Agent is bot, browser, sdk or unknown.
- PostHog (posthog-service.ts): the distinct id is a keyed hash, the event name is closed, and
  every property is closed.
- The audit writer (audit-service.ts): the actor, resource id and address are keyed hashes, the
  user agent is a class, and the metadata is closed. A query by actor hashes the same way.
- The logger (closed-sinks.ts gatewayLoggerOptions, wired in server.ts):
  - the request is its method, route template and client hash;
  - the response is its status, and an error its class, code and frames;
  - every message and object goes under the text rule;
  - a streamWrite check closes child bindings;
  - requestIdHeader is false, so a caller never names the reqId.
- Console: inside a request's scope (an AsyncLocalStorage opened by a root onRequest hook), a
  console line is closed. Boot output is unchanged.
- Sentry (sentry.ts sentryOptions): the SDK collects no request data. beforeSend,
  beforeSendTransaction, beforeSendSpan and beforeBreadcrumb rebuild each record from closed fields.
  extra.url becomes the route template.
- The write-audit hook records the route template, not the URL.
- funnel-tracker registers its onboarding_* event names, and compares trace ids by keyed hash.

Tests:
- __tests__/observability/every-position.test.ts puts a marker in every request position and
  drives it through the real createGateway. Every position: the query, the fragment, encoded
  separators, path segments, every header, cookies, and JSON, form and prose bodies. Every sink:
  the Sentry envelope, the log stream, the console, every audit row and PostHog.
- __tests__/observability/closed-schema.test.ts drives each chokepoint with markers under
  arbitrary keys and in prose.

agent: pcc-readmodels (c255d7dc)
… the MPP failure log keeps no library text (N107b, #514 r2 MEDIUMs)

The cross-family review r2 of #514 (rm-n107-514-r2-b8a3b5c0, SHIP) tracked two MEDIUMs, and the PR
steward's closed schema (#5315, #5319) sets the fingerprint's shape.

- MEDIUM 1: acceptLanguage and cfCountry passed any two or three letters.
  - The Accept-Language is no longer reported.
  - The country is an ISO 3166-1 alpha-2 code (a registered closed set) or "other", and "unknown"
    when absent.
- MEDIUM 2: the MPP failure log logged the payment library's raw error. It now logs a fixed code
  (mpp_check_failed) and the error's class.
- The fingerprint keeps:
  - the client as a keyed hash (the shared keyedHash, under PCC_TELEMETRY_KEY);
  - the method;
  - the route template;
  - uaClass, the steward's four classes;
  - the coarse classes the ruling allows: the referer kind, an allowlisted content type, the ISO
    country, the edge, and the hop count capped at 5;
  - the timestamp.
- The fingerprint drops uaLength and responseSize.
- An attack event is its type and a closed source name; the matched content's length and the
  summary text are dropped.

Tests changed because they pinned the old shape:
- n107-r1.test.ts:
  - the fingerprint schema has no acceptLanguage, and clientId is h:<32 hex>;
  - adds MEDIUM 1's case.
- n107-telemetry-sinks.test.ts: an attack has no attackLength or attackPayload.
- n107-mpp.test.ts: its logger is the gateway's closed logger, and it adds MEDIUM 2's case.

agent: pcc-readmodels (c255d7dc)
… them (N107b)

The audit log now stores actors, resource ids and metadata ids as keyed hashes (dd84dca). A reader
that compared a raw id against them would silently find nothing.

- The admin journey view (/api/admin/observability/journey/:traceId) compared each report's
  trace_id with the raw trace id. It now compares the keyed hash. A new test (the journey view
  lists a trace's reports) fails without this fix and passes with it.
- funnel-tracker already compared trace ids by hash (dd84dca). AuditService.query hashes an actor
  filter, so /api/telemetry/audit?actor= keeps working.

Tests that pinned the audit log's raw content, updated to the closed contract:
- audit-service.test.ts and telemetry-audit.test.ts: the actor is keyedHash("alice"), and the
  resource id is keyedHash("job-123").
- funnel-tracker.test.ts: its audit mock hashes the actor and resource id as the real writer does.
- feedback-observability-integration.test.ts:
  - a report is found by its hashed trace id;
  - its agent-supplied texts are closedText values, and the raw summary is absent;
  - the histogram key is closedText of the error code.

agent: pcc-readmodels (c255d7dc)
…steward ruling #5664)

The PR steward's ruling on #538's coarse classes (#5664, DECISIONS.md): the referer kind,
content type, edge and hop count stay; the ISO country goes unless gateway names a forensics
need. The fingerprint no longer reads cf-ipcountry or x-vercel-ip-country for a value. The
edge enum still notes that the Vercel header is present, never its value. The ISO list and
its closed-value registration are removed.

n107-r1's MEDIUM 1 case now pins that no country field is sent, and that neither "ZQ" nor
"US" reaches PostHog. A mutation check confirms the pin: with a country field put back, the
test fails.

agent: pcc-readmodels (c255d7dc)
AuditLogRepository.query takes eventType, actor and resourceType as one value
or a list; a row matches when its column equals any listed value (inArray).
An empty list filters nothing. The gateway's closed audit log (N107b round 2)
uses it to read a value as both schemas stored it: the value itself, written
before the closed schema or declared, and its keyed hash.

agent: pcc-readmodels (c255d7dc)
…ucer declares it (N107b r2)

Cross-family review r1 of #538 (DO-NOT-SHIP), MEDIUM 3 and MEDIUM 2. The closed
schema trusted a value by its spelling or its key: an ISO timestamp, a number
under a metric-like key, a boolean, or a value equal to any string literal in
the gateway's source left raw. And the last pass over a log line closed only
top-level strings, so a child logger's nested bindings left raw.

- closed-schema.ts: every value leaves as its keyed hash (strings, numbers,
  booleans, timestamps), and so does an object key, unless its field was
  declared. A producer declares with declare.metric, declare.code (a closed
  vocabulary), declare.serverTime, declare.id (hashed) and declare.flag, and a
  message or name with lit(), whose type takes only a compile-time literal
  (closed-schema.types.ts pins that under tsc). A declared value is a frozen
  object registered in a WeakMap, so nothing parsed from a request can be one.
  The whole-source literal and identifier dictionaries, registerClosedValues,
  and the timestamp and metric-key exceptions are gone; route templates stay.
- closed-sinks.ts: the last line pass rebuilds every log line. The fields the
  call's own chokepoints produced (logMethod and formatters.log, in one
  synchronous write) stay; every other field (a child logger's bindings,
  nested or not) is closed recursively; time, pid and hostname are the
  server's; a request id stays only when the gateway issued it (genReqId).
  Fastify's own records keep their messages and response time, and a Fastify
  logger takes a lit() message.
- The chokepoints declare what they build: a request's method and route, a
  response's status and time, an error's class and code. The PostHog boundary
  and the audit writer close every undeclared field; an audit code filter
  matches a declared code or its keyed hash.
- Producers whose rows are read back, and those N107b already pinned, declare
  their fields: server.ts (write audit, Sentry extra), the security monitor
  (fingerprint, events, logs), the MPP failure log, the funnel tracker and both
  agent.report writers. The remaining call sites are a separate codemod pass;
  until it lands, their output leaves hashed, never raw.

Tests: closed-schema-r2.test.ts pins MEDIUM 2 and 3 through the logger,
PostHog and the audit writer, and every-position.test.ts adds a valid timestamp
and a gateway literal as request values. Changed to the declared policy:
closed-schema, audit-service, telemetry-audit, funnel-tracker, n107-r1,
n107-telemetry-sinks, agent-feedback and feedback tests (producers declare
their codes; test doubles close as their boundary does).

agent: pcc-readmodels (c255d7dc)
…o caller trace id (N107b r2)

Cross-family review r1 of #538 (DO-NOT-SHIP), CRITICAL 1: a caller's
sentry-trace and baggage headers set the trace and span ids Sentry continued,
the rebuild copied those ids raw, and the envelope header's dynamic sampling
context (sdkProcessingMetadata) passed opaque, caller baggage members included.

- First line, sentry.ts: no SDK option ignores incoming trace headers
  (strictTraceContinuation only compares the baggage's org id, which a caller
  can send). startSentry, the gateway's one init path, puts the propagator
  Sentry.init registers behind one whose extract never reads the request's
  headers, so every request starts a new trace. Sentry's http server
  integration and @fastify/otel both extract through it. inject and fields
  pass through, so outgoing propagation is unchanged.
- Second line, closed-sinks.ts: every trace_id, span_id, parent_span_id and
  segment_id in error events, transactions, their spans and standalone spans is
  remapped to keyed hex of the same length (one id, one mapping, so the tree
  still links up). The header's sampling context is rebuilt from the event's
  own remapped trace id, the sampling decision, and the server's public key,
  environment, release and sample rate; nothing else of sdkProcessingMetadata
  is kept. @sentry/core reads it in createEventEnvelope, after beforeSend and
  beforeSendTransaction (_processEvent, then sendEvent). A standalone span's
  envelope header is built outside every hook (createSpanEnvelope); only core's
  SentrySpan makes one, and the gateway's OTel-based SDK makes none.

Tests: closed-schema-r2.test.ts covers the remap, the tree's links, the
rebuilt header, and the wrapped propagator (extract sees no headers; inject
and fields are unchanged). every-position.test.ts sends sentry-trace,
traceparent and baggage markers through the real gateway, under its own
Sentry init path. No raw marker reaches the envelope, header included. No
event carries the caller's trace, even remapped. Two forced 500s with the same
headers are two traces.

agent: pcc-readmodels (c255d7dc)
…a; boot names a missing key (N107b r2)

Cross-family review r1 of #538 (DO-NOT-SHIP), MEDIUM 4. The closed audit log
stores actor, trace and report ids as keyed hashes. Rows written before it
keep the raw ids, and there is no data migration, so the actor query, the
funnel and the journey view lost them after deploy. Without PCC_TELEMETRY_KEY,
every restart also changes every hash.

- Dual read, no migration: every audit query filter (event type, resource
  type, actor) matches the value as given or its keyed hash. The funnel and
  the journey view compare a trace id either way (isStoredId). The cohort
  funnel counts one key per trace (storedIdKey), so a journey that spans the
  change counts once.
- Boot: in NODE_ENV=production without a valid PCC_TELEMETRY_KEY (unset or
  under 32 characters), one error-level warning names the consequence: after a
  restart, rows this process wrote are no longer found by actor, trace or
  report id, and PostHog and Sentry ids no longer join. Elsewhere it stays one
  warning. The gateway still starts; refusing to is the operator's decision
  (#5708), as is a one-time migration that hashes historical rows.

Tests: closed-schema-r2.test.ts inserts rows as the pre-N107b audit log held
them. The actor query, the funnel and the journey view (its funnel and its
reports) all find them. A journey written both ways counts once. The
production warning is error-level, names the consequence and reaches the log.

agent: pcc-readmodels (c255d7dc)
…y-sweeper.ts (N107b codemod)

agent: pcc-readmodels (c255d7dc)
LamaSu added 15 commits October 3, 2026 18:47
…b r4, C11)

tracing.test.ts pinned the old producer shape: pipelineTelemetry.emit passed
a numeric timestamp to Sentry.addBreadcrumb. The C11 commit removed it (no
gateway code passes a time to a Sentry or OpenTelemetry API; Sentry's
breadcrumb chokepoint gives every breadcrumb the server's clock), so the
test now pins that the producer passes none. Found by implementer-mike
before the full-suite run.

agent: pcc-readmodels (c255d7dc)
trace-collector.ts holds spans that GET /api/traces, /api/traces/:traceId
and the trace stream return to any key holder, and it stored what its
producers passed: raw job and bundle ids in the attributes, raw names,
caller-chosen ids and a producer's end time. At f04c858 the property test
found 5,528 of 8,061 positions leaking. A poisoned end time also made
getRecentTraces throw for every later read, and a BigInt attribute broke
the stream.

The collector is now its own chokepoint, whatever a producer passes:
- the attributes go through closeValue: a declared attribute keeps its key,
  anything else leaves keyed, key and value, at any depth;
- the operation, description and service are declared (lit, or a closed
  vocabulary with declare.code), else keyed; the types require Declared;
- a trace, span or parent id is one the collector issued
  (newTraceId/newSpanId, recorded in a bounded registry of the last 10,000),
  else its keyed hash. One producer id keeps one stored id within a trace, so
  the tree still links, and getTrace looks a trace up by the id it returns;
- a span's status is ok, error or in_progress, else keyed;
- every time is the server's clock, read in the collector: endSpan and
  endTrace no longer take an end time.

The producers declare what may stay readable. tracing.ts takes its ids from
the collector and passes Sentry only the declared names. kernel-service.ts
declares job.lifecycle and kernel and keys the job's id, step and tier.
settlement-service.ts declares each span's operation and service, declares
write.enabled and auto_release as flags, and keys every id and count. The
routes' response shapes are unchanged.

trace-collector.test.ts moves to the collector's own ids and declared names
and pins the closed contract; tracing.test.ts pins both producers end to
end. Found by implementer-mike.

agent: pcc-readmodels (c255d7dc)
The generic property test now treats the trace collector as a sink. A marker
goes into every field a span stores:
- its trace, span and parent ids, its operation, description and service;
- every attribute key and value, at every depth;
- endSpan's status and end time;
- the producers' helpers in tracing.ts (startTrace, withSpanSync, endTrace).

Each position is read back through getRecentTraces and getTrace, through
GET /api/traces and /api/traces/:traceId, and through the stream's snapshot,
which is read by calling the stream route's own handler with a reply that
records what it writes. A read that throws, or a stream that never
completes, counts as a sink that broke. A positive control keeps the
collector's own ids, a declared operation, service and attributes, and the
server's clock readable through the route.

At f04c858 the trace collector failed 5,528 of 8,061 positions; the seven
earlier sinks stayed clean. It now leaks 0 of 7,723. The render helper now
prints a bigint as its bare digits, so the bounded-number check sees it.
Written by implementer-mike.

agent: pcc-readmodels (c255d7dc)
…l-query.ts and the API docs (PR steward)

#403 merges after #448 and #389 (the steward's whole-order simulation). Both have
now merged, and master conflicts with #403 in three files:

- routes/nl-query.ts: #403 threads the caller's job read scope through
  executeIntentQuery, so job intents answer only readable jobs. #448 (N68) made
  the same function async, so kernels and capabilities are answered through
  their coarse-location read models. Both are kept: the function is async and
  takes jobScope, the call site awaits it with jobScope, and both imports and
  both doc paragraphs stay.
- CLAUDE.md and docs/AGENT_INTEGRATION.md:
  - In the kernels table, master's /api/kernels row (public, coarse location)
    and POST row (locationVisibility) are kept, with #403's /:kernelId
    (recentJobsScope) and /:kernelId/jobs (read scope) rows.
  - #403's "Read access" paragraph comes before master's new "Operator Work"
    section.

Checked on the resolution: tsc is clean, and nl-query, the read models, F3 and
N68 location privacy pass (13 files, 298 tests).

agent: pcc-readmodels (c255d7dc)
…or the admin

Board N31 (bus #6272; steward ruling #6278). routes/operator.ts changed any kernel's
emergency stop, policy and approvals for any authenticated caller. A stranger's key could
resume a kernel its operator had stopped, approve or reject its jobs, rewrite its guardrails
(PATCH {emergencyStop:false} is a resume by another name), or submit an already-approved
approval that the OT-2 executor then runs. A truthy non-boolean autoApprove ("yes") was also
stored as approved.

Who may act on a kernel now:
- approve, reject, emergency-resume, PUT and PATCH of the policy, and an approval submitted
  with autoApprove: the admin secret (X-Admin-Key), or a wallet the caller PROVED that is the
  kernel's operator (req.provenWallet, which WP-A #326 sets; nothing sets it before then, so
  until WP-A only the admin decides). A claimed identity never decides.
- emergency-stop and a pending approval: also the kernel's own principal, the identity its
  operatorAddress records (POST /api/kernels takes it from the registering caller). An operator
  never loses their own e-stop. Residual: queue item 136.
- anonymous is 401; an unregistered kernel is 404 for a non-admin; a failed kernel read is 503;
  autoApprove must be a boolean (400).

Reads (GET policy, GET approvals) are unchanged. Tests: n31-operator-route-ownership (39 cases,
production-mounted behind apiGate, the kernel registered by the operator's own key); the
existing operator, N32 and capture policy tests now write as admin.

agent: pcc-readmodels (c255d7dc)
…538 r3)

astra's source pack (MEDIUM, OTLP exporter bypasses the Sentry rebuilding
chokepoint): otel.ts sends spans to an OTLP exporter (the console in dev),
and Sentry's beforeSendSpan is no boundary for that path. The event-bus
bridge sent each AppEvent's kind, sponsor, session id, duration and text,
and its exception and status messages, raw. The kernel's job.lifecycle span
and the settlement pipeline's Sentry spans sent raw job, bundle and contract
attributes. At b5718ec an in-memory exporter on the otel.ts path exported
13 spans carrying the marker, and the property test found 449 of 615 OTLP
positions leaking.

observability/closed-otel.ts adds ClosedSpanExporter, which rebuilds every
span before the exporter it wraps serializes it:
- a span, event or tracer name is kept only if it is in the vocabulary of
  declared names (otelName); any other name is keyed;
- an attribute is kept only while the span still carries the value its
  producer declared for that span object (startClosedSpan,
  startDeclaredSpan) or event object (addClosedEvent);
- every other attribute key and value, event and link attribute is keyed;
- a status keeps only its code;
- trace, span and parent ids are remapped under the server key, and no
  trace state is sent;
- the resource is the server's own configured service name and version;
- times are the SDK's clock (the timing ratchet).
otel.ts builds the SDK around it (createOtelSdk, otelSpanExporter), and
getTracer takes a declared name.

The bridge, the kernel's job.lifecycle span and the five settlement spans
now declare their names and attributes. An emitter's or caller's value is
keyed, and the bridge's exception event is declared.

Tests:
- otlp-sink.test.ts drives the otel.ts path with an in-memory exporter and
  a marker in every AppEvent field and every kernel and settlement span
  attribute;
- the property test adds the OTLP exporter as a sink and documents the two
  SDK refusals;
- the bridge and tracing pins that expected raw attributes now expect the
  declared forms.
Found and fixed by implementer-mike.

agent: pcc-readmodels (c255d7dc)
…#538 r3)

astra's source pack (MEDIUM, stored entries can be poisoned after the
chokepoint): StructuredLogger returned its stored entry objects, and
TraceCollector.buildTree() shallow-copied spans but shared each stored
attributes object. A reader that changed what it read changed what every
later reader got: GET /api/telemetry/logs and its stream, GET /api/traces,
/api/traces/:traceId and the trace stream. One subscriber could also change
the tree the next subscriber (the SSE fan-out) serialized. At b5718ec a
reader's marker written into getRecent, getEntries and query results came
back from the logger and the route, and one written into a subscriber's
tree, getTrace and getRecentTraces came back from the collector and the
route.

- closed-schema.ts: frozen() freezes a closed record at every depth.
- structured-logger.ts stores each entry frozen, so every read path
  returns immutable records in a fresh array.
- trace-collector.ts stores each span's closed attributes frozen. Each read
  (getRecentTraces, getTrace, and the tree each subscriber gets) is a tree
  built fresh and frozen: spans, children lists and the trace.

The property test adds the check: each read path's records are tampered
with (every string and number replaced by a marker, a marker key added,
arrays grown), then read back through the store and the route, and no
marker may appear. Found and fixed by implementer-mike.

agent: pcc-readmodels (c255d7dc)
…7b, #538 r3)

astra's source pack (MEDIUM, non-blocking, structured-log filters no longer
match protected fields): since the structured log keys what no producer
declared, GET /api/telemetry/logs compared each filter with the stored
field as given. A job id, kernel id or source arrives in plain text but is
stored as its keyed hash, so ?jobId=, ?kernelId= and ?source= found nothing.
At b5718ec an entry emitted for job-filter-r3 was not returned by
?jobId=job-filter-r3.

StructuredLogger.query now compares the stored form, as the audit query
does. A filter value matches when the field holds the value itself (a
declared readable value) or its keyed hash. The field may sit under its
own key or, when its producer did not declare it, under the keyed key.
This covers level, source, jobId and kernelId. A search also matches a
message no producer declared (stored keyed) by its exact text. Nothing the
log stores or returns changes.

closed-schema-r4.test.ts pins the route (job id, source and level filters,
and a non-matching id) and the query itself: undeclared and declared
fields, a vocabulary member, a keyed message and the default source.
Found and fixed by implementer-mike.

agent: pcc-readmodels (c255d7dc)
…el-authority module (#575 r2)

Cross-family review r1 of #575 (DO-NOT-SHIP):
- HIGH: PUT /api/kernels/:kernelId/agent-package/configure compared kernel.operatorId, a
  column kernels do not have, so it never refused. A stranger could write customTools into
  another kernel's operator policy, and they appeared in that kernel's agent package; a
  missing kernel got an orphan policy. Reproduced at 98f3ea3: stranger 200, claim-only 200,
  unknown kernel 200. Now it is a "decide" write (admin or the proven operator): anonymous 401,
  stranger and claim-only 403, unknown kernel 404 for a non-admin.
- The weakest link, ownership checks copied per route family: the N31 rules now live in one
  module, auth/kernel-authority.ts, used by routes/operator.ts and kernel-agent-package.ts.
- MEDIUM: the onboarding wizard told a new operator to save its policy, approve the test job
  and save tool choices, which its own key now cannot do until WP-A. It now names them operator
  decisions that answer 403 (operator_proof_required), says the choice is not saved, and never
  claims success; the e-stop stays the operator's own. Reproduced with the 98f3ea3 prompt:
  the 3 new wizard tests fail.

New tests: 5 configure cases and 3 wizard cases in n31-operator-route-ownership; a bare-app file
(n31-kernel-authority-fail-closed) pins the routes' own 401 without apiGate and the 503 when the
kernel cannot be read. Mutations 8/8 caught (configure unguarded, configure as a stop, no 401,
claim decides, zero owner, missing kernel passes, case-sensitive claim, read failure passes).

agent: pcc-readmodels (c255d7dc)
…and for a shared trace tree (N107b, #538 r3)

This commit only adds tests. It follows up the mutation proofs for the
first two fixes of this round (astra's source pack, MEDIUM: the OTLP
exporter bypasses the chokepoint; stored entries can be poisoned after it).
At ff7f92b, six mutations of those fixes passed every test:
- trace and span ids sent as the SDK or a remote parent made them;
- the trace state sent;
- the SDK-merged resource sent;
- a raw span kind sent;
- a raw instrumentation scope sent;
- the bridge's tracer name left undeclared;
- the tree a trace read returns left unfrozen, so subscribers share one
  mutable tree.
No marker can witness an id, because a hex id never shows the bounded
number. So each of these gets its own check.

- otlp-sink.test.ts: a span under a remote parent (as a propagated header
  makes it) leaves with its trace and parent ids remapped by keyedHexId and
  no trace state. A declared tracer keeps its name and version, and a raw
  one (the marker) leaves keyed. A raw kind leaves as 0. Every span carries
  the server's own resource. The bridge's spans keep their declared scope.
- property-every-position.test.ts: the OTLP block adds a span's kind, a
  tracer's name and version per kind, and a provider whose resource carries
  the marker. The four positions the SDK refuses (getTracer's cache key
  throws for a symbol or a null-prototype name) are documented, as
  recordException's are. The immutability block adds a subscriber that
  reads the tree after another subscriber wrote to it.

Written by implementer-mike.

agent: pcc-readmodels (c255d7dc)
… writer fails CI (#575 r2)

The steward's ruling #6493 (2) and #6508 (1). n31-kernel-route-inventory.test.ts reads every
production source under packages/gateway/src with the TypeScript parser:
- A: every insert, update or delete of operator_policies or pending_approvals sits inside a
  route whose handler calls the ONE guard (auth/kernel-authority.ts), directly, through a
  same-file helper, or through its plugin's preHandler hook registered before the route.
- B: every mutating :kernelId route is guarded, or is exactly the KNOWN_UNGUARDED list (no more,
  no fewer), or the closed CLASSIFIED set with an executable witness.
KNOWN_UNGUARDED holds the 9 routes of bus #6505 (the 7 device-relay writes, kernel heartbeat
and capability announce) plus the digital-manifest verify, whose builder self-auth trusts an
x-agent-id header anyone can send. The stacked PR guards them and empties the list. CLASSIFIED:
the manifest suspend (its own admin-key check; the witness shows a caller without it refused).
A probe source pins the scanner on each form (handler, helper, hook before and after a route,
writes inside and outside routes). Mutations: un-guarding configure fails A and B; un-guarding
the e-stop fails A.

agent: pcc-readmodels (c255d7dc)
A plain merge with no manual edit; its tree equals git merge-tree --write-tree of master
108c778 and the previous head 2de4298 (6cdb1e0).

agent: pcc-readmodels (c255d7dc)
A plain merge with no manual edit; its tree equals git merge-tree --write-tree of master
108c778 and the previous head 2d685bc (b6b01f1).

agent: pcc-readmodels (c255d7dc)
…; the inventory counts #400's relay guard

The master merge (f2eef6f, master 108c778) met #400 (N4b-gw, merged after #575 opened):
- #400's device-relay suite calls /api/operator/emergency-stop and /emergency-resume with no
  credentials; #575 guards both (a resume is a decision), so its stop and resume calls send
  the admin key. Its assertions are unchanged.
- #400 added relayAccessGuard, the relay plugin's preHandler (a default-deny per-route table
  over the kernel's recorded operator), and a new POST /tool-call/:callId/start. The inventory
  counts a preHandler registered by name as a guard when the name is one (relayAccessGuard is
  listed beside refuseKernelAction; unifying the two ownership checks is a follow-up), and the
  probe pins that form. On master the 8 relay POST routes are guarded, so KNOWN_UNGUARDED is now
  heartbeat, capability announce and the manifest verify, which the stacked N31b guards.
Verify: f2eef6f's full suite failed only these 3 tests (2 in device-relay, 1 inventory); now
those files pass (293 tests), tsc 0.

agent: pcc-readmodels (c255d7dc)
…calls the shared guard

The gateway owner's #6568 (c) and the steward's N126: master's #400 relayAccessGuard checks the
CLAIMED tier for every relay route, actuation included, so it must not count as the shared
guard. GUARD_NAMES is back to refuseKernelAction and refuseKernelRequest (a same-file helper or a
preHandler, inline or by name, counts when it calls one), and the 8 relay POST routes join
heartbeat, capability announce and the manifest verify in KNOWN_UNGUARDED. N31b moves
relayAccessGuard onto the kernel-authority tiers and empties the list. Test-only.

agent: pcc-readmodels (c255d7dc)
…iles

Conflicts and how each keeps both sides:
- db IAuditLogRepository, repositories/audit-log, services/audit-service: master's #469 added a
  resourceId filter; #538 stores resource ids keyed, so resourceId joins eventType, actor and
  resourceType as a both-forms filter (the raw value and its keyed hash).
- facades/job.facade, routes/capabilities, routes/operator-relay: both sides' imports (#538's
  closed-schema helpers; master's settlement-owned-status and funnel-tracker).
- facades/kernel.facade: #538's declared audit fields plus master's locationVisibility, declared
  from the closed vocabulary ["exact","approximate"].
- server.ts: #538's closed logger (gatewayLoggerOptions, requestIdHeader false, genReqId) replaces
  master's GATEWAY_LOGGER_OPTIONS serializer, which it subsumes (no URL, header or raw IP ever).
  The write-audit hook keeps master's module (services/write-audit-hook.ts, #458) with #538's
  declared fields ported in (route template, closed method and action, server metrics), and
  master's telemetryLookalikeHook stays.
- observability/closed-sinks: the request log line omits even the keyed client hash for the
  public telemetry sink (#458's rule).

agent: pcc-readmodels (c255d7dc)
@LamaSu
LamaSu changed the base branch from fix/n107-telemetry-sinks to master October 4, 2026 07:02
…hema

The master merge (63401c2) left 8 failing tests in 4 files; each is master code that
postdates #538's base meeting the closed observability schema:
- the logger ratchet: 7 raw log messages in routes/operator.ts, routes/operator-work.ts and
  readmodels/legacy-settlement.ts now go through lit().
- the timing ratchet: services/funnel-tracker.ts (#469) built its span event with spreads; it
  now uses addClosedEvent with declared fields (the stage from OPERATOR_STAGES, the ids
  declared) and no spread.
- routes/operator-channels.ts: master's N84 added dns_busy to OutboundErrorCode, but #538's
  hardcoded OUTBOUND_ERROR_CODES lacked it, so a saturated resolver's code was keyed out of the
  operator log N84 keeps it in. The vocabulary is now a Record over the union (complete by
  construction: a new code fails to compile until listed). n84-dns-bound reads the warning's
  declared arguments as they leave.
- feedback-attempt-production (#458): its app now mirrors server.ts's closed logger
  (gatewayLoggerOptions, issued request ids, closedLoggerHooks); audit rows are found by their
  declared route template; the test that pinned a raw URL in other routes' audit now pins the
  closed rule (route template only, no query), and the log-line test pins method plus route,
  with the client's keyed hash for other routes and nothing at all for the sink.
Targeted: the 4 files pass (ratchets and funnel 56, feedback 12, n84 19); tsc 0.

agent: pcc-readmodels (c255d7dc)
LamaSu added a commit that referenced this pull request Oct 4, 2026
…into N107c

agent: pcc-readmodels (c255d7dc)
LamaSu added a commit that referenced this pull request Oct 4, 2026
…ver carry raw error text (N71 round 5 followup)

Three sibling sinks to BaseFacade.execute's span (F3/F4) and
checkDeviceHealth's console.warn (F1), all in kernel-service.ts's
submitJob fire-and-forget execution path:

- :364 and :447 (primary Sentry path and the non-Sentry fallback path):
  the settlement-pipeline catch logged err.message/err directly. Now
  logs a fixed "settlement_pipeline_failed" code plus, at most,
  knownErrorClassName(err) (closed set, instanceof-checked) — same
  three-argument console.warn shape as checkDeviceHealth already uses.

- :388 (a REJECTED runner.run(), i.e. the job runner throwing instead
  of resolving {success:false}): the lifecycle span's setStatus message
  was String(err). Now `job_run_failed:${knownErrorClassName(err)}`.

- :373 (a RESOLVED job failure, result.success===false): the span
  message was result.error, which is JobRunner's own JobResult.error
  (job-runner.ts:38) — NOT a facade Result as 83e-n71-report.md's
  span-site-list assumed. It can carry an adapter's raw
  MachineCommandResult.message ("Failed to load G-code: ...") or
  JobRunner's own caught err.message/String(err), so it is in scope.
  Now a fixed "job_run_failed" code with no result.error read at all
  (no knownErrorClassName here either — result.error is already a
  string, not an Error instance, so the classname would always be the
  UNKNOWN_ERROR_CLASS fallback and add no real information).

Same property throughout: a fixed code, plus at most the closed-set
class name — never err.message, String(err), or text derived from the
error. No recordException anywhere in this file (none existed before
either). event-bus-otel-bridge.ts (#538) is untouched — out of scope.

Verified: packages/gateway/src/__tests__/kernel-service-safety.test.ts
10/10 passing (was 6/10 at ee1754a — the 4 new [neg] tests from the
prior commit now pass). base-facade-otel-redaction.test.ts,
tracing.test.ts, redaction.test.ts, device-credentials-redaction.test.ts
(162 tests) also pass, unaffected.

Agent: implementer-n71r7

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Yc3NgnENeksqXdgUVHzwbh
LamaSu added 3 commits October 4, 2026 06:20
…sed schema), per the steward's #6814

#538 conflicted with #403 in 4 files. Resolutions:
- services.ts: both imports. #538's closed-schema import (with EVIDENCE_STORAGE_BACKENDS) and
  #403's batch-ownership import.
- readmodels/legacy-settlement.ts and routes/jobs.ts: #403's hunk. Its job-read gate moved into
  readmodels/job-read-gate.ts (gateJobRead/refuseJobRead) and replaces the inline precheck that
  #538 had put on the closed schema.
- routes/telemetry.ts: both imports, #538's closed schema (with TELEMETRY_STATUSES) and #403's
  job-read-gate.

Semantic conflict, resolved here: #403's new readmodels/job-read-gate.ts logs with raw messages
and a raw jobId. These are the only sink calls #403 adds, checked by grepping its added lines
for .log., console., Sentry, audit, PostHog and trackServerEvent. Under #538's closed schema a
sink keeps only declared values. So its 6 req.log.error calls now carry lit() messages, and its
jobId fields carry declare.id(), as #538's own job reads did.

#403 also changes packages/kernel (SensorPipeline.aggregate takes a filter), so the workspace
deps are rebuilt before the gateway's tsc.

agent: pcc-readmodels (c255d7dc)
…sed schema), per the steward's #6814

#538 conflicted with #575 in routes/operator.ts, at the imports only. The resolution keeps both:
#538's closed-schema `lit` import and #575's kernel-authority imports.

Semantic conflict, resolved here: #575 adds one sink call, in auth/kernel-authority.ts:
`req.log.warn({ kernelId, err }, "kernel read for an operator action failed")`. It is the only
one, checked by grepping #575's added lines for .log., console., Sentry, audit, PostHog and
trackServerEvent. Under #538's closed schema it now carries lit() for its message and
declare.id() for kernelId.

agent: pcc-readmodels (c255d7dc)
…og's keyed ids (#403 x #538)

The fold of #538 with #403 and #575 left 6 tests red in 4 files. The full suite at 4e1d7b0:
268 files, 6 failed. Cause: a semantic conflict between two designs.
- #538's closed structured log (structured-logger.ts) stores an id its producer declared
  (declare.id) as its keyed hash, and an undeclared field with its key keyed too.
- #403's record filter (jobRecordFilterOf) compared a line's job and kernel ids with the
  caller's readable ids as raw strings. So a party lost its own jobs' log lines, in the history
  and on the live stream. This failed closed, not open.

Fix (readmodels/job-read-gate.ts, jobRecordFilterOf): a keyed value ("h:...") counts as the job
or kernel it is the keyed hash of, among those the caller may read. The keyed hashes of exactly
those ids are computed once per filter, when a keyed value is first met. A keyed value that is
no readable id's hash stays unreadable, so the line is left out. An undeclared binding (key
keyed) binds nothing, so its line stays an unscoped admin's only, as free text is.

Tests:
- #403's log lines (f3-job-read-surface, f3-r3-bypasses) are written as producers must now
  write them: lit() messages and declare.id() bindings.
- #538's two log reads (closed-schema-r3, closed-schema-r4) go through #403's gate as the admin.
  Asking for one job's lines is reading that job, so the filter test uses the seeded job-001
  rather than a job with no row. Every assertion is kept.
- New: a line naming only a declared kernel-nyc is the kernel operator's and the admin's,
  never a stranger's or a buyer's.
- Mutations: with the keyed job match removed, 3 f3 tests fail; with the keyed kernel match
  removed, the new test fails. Both files were restored byte-identical.

agent: pcc-readmodels (c255d7dc)
… an unscoped admin's only (astra CRITICAL)

astra's verdict on rm-n107b-538-mu2-b5255279 was DO-NOT-SHIP, with item 3 OPEN and CRITICAL.
- #538's closed log keys an UNDECLARED field's NAME and value, so recordOwnersOf found no job or
  kernel in such a line.
- jobRecordFilterOf then applied #403's free-text rule ("a record naming no job is an admin's")
  to EVERY admin. So under TENANT_ENFORCE, a tenant-A admin received tenant B's line, by the REST
  read and on the live stream.
- b525527's comment claimed "an unscoped admin's only". That was false.

Reproduced at b525527 with astra's cheapest repro: a tenant-B line,
`logger.log(lit(...), { jobId: "job-003" })` with an undeclared binding, read as a tenant-A admin.
Both new tests failed: the body contained the line on GET /api/telemetry/logs and on
/api/telemetry/logs/stream.

Fix (readmodels/job-read-gate.ts, walkBindings):
- A field name the closed log keyed (KEYED_KEY, "h:" + 32 hex) may be a binding the filter cannot
  read. Its record is malformed (owners unknown), so only an unscoped admin keeps it.
- The exception is a keyed name holding a plain object. That is only a container; its declared
  fields keep their names and are walked, as #403's nested-binding line needs.
- The only production producer of the structured log (telemetry emit) declares every field, so
  none of its lines changes. The jobRecordFilterOf comment now states the rule correctly.

Tests: the two cross-tenant tests in f3-r4-bypasses (REST and live stream) pass now. The
tenant-A admin does not get the line, and the unscoped admin still does.

Mutants, each file restored byte-identical:
- keyed names ignored: the 2 cross-tenant tests fail;
- keyed containers not walked: the nested-binding party test fails.

agent: pcc-readmodels (c255d7dc)

This branch had an error being deployed

1 failed deployment
trusted-checks — 1499e012 Deployed Oct 4, 2026 by LamaSu via post-verdicts #144
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant