Record how many telemetry events a Modal stage's emitter left undelivered

README.md

@@ -63,7 +63,7 @@ change:
 ```bash
 uv tool install microcosm-emitter   # once per machine
 uv tool upgrade microcosm-emitter   # to take a new release
-microcosm-emitter version           # the installed release; builds need 0.11.9 or later
+microcosm-emitter version           # the installed release; builds need 0.13.0 or later
 ```
 
 The build finds the `microcosm-emitter` command through
@@ -76,15 +76,20 @@ no remote staging, and queues no Orrery graph publication; a UK build given
 its dataset bundle remotely, as it is by default, is refused before it
 starts, since the emitter uploads the bundle. Before it starts the emitter,
 the build reads its version from `microcosm-emitter version` and does not
-start one older than 0.11.9. Releases before 0.10.0 send telemetry to the
+start one older than 0.13.0, the release that added `microcosm-emitter
+status`, which the US Modal stage runner runs after each stage to record how
+many telemetry events the stage's emitter left undelivered. The release is
+written once, in `packages/microcosm-build/src/microcosm/build/minimum_emitter_version.json`,
+which builds read as package data and the Modal image's install step reads
+from the launching checkout. Releases before 0.10.0 send telemetry to the
 collector's Cloud Run address, which changes when the collector moves to
 another Google Cloud project; from 0.10.0 on, releases send to the
-collector's hostname, which stays the same. 0.10.0 would be enough for that;
-0.11.9 was the newest release when practically no machine ran the emitter
-yet, so requiring it cost nothing (see
+collector's hostname, which stays the same. 0.13.0 was the newest release
+when practically no machine ran the emitter yet, so requiring it cost
+nothing (see
 [graph publication](docs/orrery-publication.md#emitter-dependency)). The
 build prints the installed and required versions, what it loses and the
-upgrade command, `uv tool install --upgrade 'microcosm-emitter>=0.11.9'`,
+upgrade command, `uv tool install --upgrade 'microcosm-emitter>=0.13.0'`,
 and continues as it does without the command. `uv tool upgrade` keeps any
 version constraint given at install time, such as `==0.7.0`; that
 `uv tool install --upgrade` command replaces it. Microcosm supplies build
@@ -107,7 +112,7 @@ if the emitter cannot be prepared or is missing, its
 `microcosm-emitter version` times out or fails, it is older than the
 minimum, it lists no protocol version 1, its `hello` fails, it exits or
 times out before it is ready, or it starts without its telemetry module.
-Every emitter a build starts (0.11.9 and later, the minimum) reads a given
+Every emitter a build starts (0.13.0 and later, the minimum) reads a given
 collector token, and exits at that point when the collector-token
 variables it inherits (`MICROCOSM_COLLECTOR_TOKEN`,
 `MICROCOSM_COLLECTOR_TOKEN_EXPIRES_AT`, `MICROCOSM_COLLECTOR_ORIGIN`) are set

changelog.d/1522-modal-undelivered-events.added.md

@@ -0,0 +1 @@
+The US Modal stage runner records how many telemetry events each run's emitter left undelivered (microcosm#1522). After the tool exits and its output ends, which is after the emitter the build started has stopped, `tools/modal_us_stage.py` runs the image emitter's read-only `microcosm-emitter status` (microcosm-emitter 0.13.0) without `--event-queue-file`, in the tool's environment without credentials or the collector token, with a 60 s timeout, and never reads the emitter's event queue file itself. The receipt's `runner.telemetry.undelivered_events`, `null` until now, becomes `{"runs": [...], "problem": null}` with `status`'s runs as printed (`run_id`, `producer_id`, `upload_state`, `reason`, `undelivered_events`); `{"runs": [], "problem": null}` when no event queue file exists (exit 8); and `{"runs": null, "problem": "<one line>"}` for any other exit code, a timeout, a command that cannot run, output that is not its list of runs, or runs with a field nested more than 32 levels deep. A failure is recorded only and never changes the stage's status or outcome. The runner's reserve after the tool keeps those 60 s, so the check's write probe leaves the mirror 1,560 s of a 30-minute reserve instead of 1,620 s, and its verdict reports the stop's and `status`'s allowances beside the receipt's. Builds and the Modal image now require microcosm-emitter 0.13.0 or later instead of 0.11.9, written once in `packages/microcosm-build/src/microcosm/build/minimum_emitter_version.json`: `microcosm.build.telemetry_emitter.MINIMUM_EMITTER_VERSION` reads it as package data and the runner's image install step reads it by its path in the launching checkout, replacing the repeated value and the test that kept the two equal. The image adds that checkout's file beside the plan module's copy, `/root/minimum_emitter_version.json`, which the copy reads there, having no checkout around it, so the check in the container reports the install step the image was built with; in a checkout the package's file always wins over one left beside the plan module. The installed-emitter job's pin rises to 0.13.0, and a new installed-emitter test runs a stage through the runner with the real emitter.

docs/agent-guide.md

@@ -135,7 +135,14 @@ without `hf_xet`, or with it turned off, for files sent in one part. A request
 it has no route for, or that its handler fails on, fails the test. The tests check
 what builds rely on in the hand-over (one commit per bundle on the head it
 names, `copied_files`, `already_staged`, the recorded error codes and the
-version check), not the emitter's own queue or delivery. The runner warns when
+version check), not the emitter's own queue or delivery.
+`test_installed_emitter_status.py` runs a stage through the US Modal runner's
+`_run_stage` against local directories, with the installed command at the
+image's emitter path: its tool starts the emitter's telemetry module through
+the interface, with no credential, no collector token and a closed loopback
+collector address, so the emitter keeps its events and sends nothing, and the
+test checks what the runner records from the emitter's `status`, and that the
+emitter host has exited before the runner runs it. The runner warns when
 the package index has a newer release than the pin; raise the pin in a pull
 request then. `tools/installed-emitter/constraints.txt` pins the
 `huggingface_hub` release whose requests the stand-in answers; the runner also

docs/orrery-publication.md

@@ -92,7 +92,7 @@ can be rebuilt from it later. This does not hold for:
 | `MICROCOSM_EMITTER_COMMAND` | Optional path of the `microcosm-emitter` command, read by the interface in the build; without it the interface searches `PATH` | Operator's shell; the Modal image sets it |
 | Collector origin | Built into the installed emitter; the build sends no address unless a caller passes a loopback development address in the module settings | No producer variable or destination override |
 | Runs origin | Tracked `DEFAULT_GRAPH_PUBLICATION_ORIGIN` | No new producer variable |
-| `XDG_CACHE_HOME` | Optional existing cache-location setting, read by the interface in the build; the build passes the spool path (`microcosm/telemetry/events.sqlite3`) and the hand-over database beside it (`events.handovers.sqlite3`) to the emitter on its standard input | Defaults to the user's cache directory |
+| `XDG_CACHE_HOME` | Optional existing cache-location setting, read by the interface in the build; the build passes the path of the emitter's event queue file (`microcosm/telemetry/events.sqlite3`) and the hand-over database beside it (`events.handovers.sqlite3`) to the emitter on its standard input. On Modal, the stage runner's `microcosm-emitter status` reads it, with `HOME`, in the tool's environment after the tool, so it reads the file the build's emitter used | Defaults to the user's cache directory; on Modal, the container's defaults (the runner sets neither variable, and a plan may not) |
 | `MICROCOSM_TELEMETRY_REQUIRED` | Optional, read by the build adapter (`microcosm.build.telemetry_emitter`) each time a build starts its emitter; only builds that start the adapter read it. `1`: the build stops with `TelemetryRequiredError` if the emitter cannot be prepared or is missing, its `version` times out or fails, it is older than `MINIMUM_EMITTER_VERSION`, it lists no protocol version 1, its `hello` fails, it exits or times out before it is ready, or it starts without its telemetry module (for a failure the interface reports, its line saying the build will continue does not apply); unset or `0`: it continues without the emitter; any other value: `ValueError` | Whoever starts the build. The Modal stage runner sets `1` unless the plan sets `0`; workstations: the operator's shell. Leaving it off (unset or `0`) exists only for people outside PolicyEngine who run Microcosm without access to the collector; PolicyEngine's own builds keep telemetry required. Not a credential |
 | `MICROCOSM_TELEMETRY_RUN_ID` | Optional, read by the US build programs that open a run through `microcosm.build.program_telemetry`: the telemetry run ID they report under; the part before its last `.` becomes the run's candidate ID. Unset, each attempt makes its own ID (`<pipeline>-<UTC time>-<8 hex digits>`); a value the collector would refuse stops the program with `ValueError` before any work | The Modal stage runner sets `<plan run_id>.<stage>`, and a plan may not set it; workstations: the operator's shell, usually unset. Not a credential |
 | `MICROCOSM_TELEMETRY_PARENT_PID` | Read by the same programs: when it equals the process's parent's ID, the process is a child stage whose parent records its telemetry, and it starts no emitter | Set by `build_us_puf_support_base.py --stage all` for the child interpreters it starts; a plan may not set it. Not a credential |
@@ -162,23 +162,33 @@ versions interfaces still send, and an emitter that lacks a module the build
 asks for starts without it.
 
 A build does not start an emitter older than `MINIMUM_EMITTER_VERSION` in
-`microcosm.build.telemetry_emitter`, now 0.11.9, and the Modal image installs
-at least the same release (`IMAGE_EMITTER_REQUIREMENT` in
-`tools/modal_us_stage_plan.py`, `microcosm-emitter>=0.11.9`). A test checks
-that the two agree in one checkout; at a Modal launch the install step comes
-from the launching checkout and the minimum from the commit the plan pins
-(see the [US Modal stage runbook](us-modal-stage-runbook.md)). Releases
+`microcosm.build.telemetry_emitter`, now 0.13.0, and the Modal image installs
+at least the same release (`image_emitter_requirement()` in
+`tools/modal_us_stage_plan.py`, `microcosm-emitter>=0.13.0`). The release is
+written once, in microcosm-build's package data,
+`packages/microcosm-build/src/microcosm/build/minimum_emitter_version.json`:
+the adapter reads it as package data, and the runner, which imports nothing
+from the workspace, reads it by its path in the checkout that launches the
+app. The image adds that checkout's file beside the runner's copy of the
+plan module, so the check, which runs in the container, reads the same
+file. At a Modal launch the install step therefore comes from the launching
+checkout and the minimum from the commit the plan pins (see the
+[US Modal stage runbook](us-modal-stage-runbook.md)). Releases
 before 0.10.0 send telemetry to the collector's Cloud Run address, which
 changes when the collector moves to another Google Cloud project. From 0.10.0
 on, the emitter's default collector address is the collector's hostname,
 which stays the same. Microcosm passes the emitter no collector address: the
 adapter's `development_collector_url` parameter, which no build command sets,
 accepts only a loopback address.
 
-0.10.0 would be enough for the hostname. The minimum is 0.11.9, the newest
-release on 2026-11-02, because practically no machine ran the separately
-installed emitter then, so refusing 0.10.0 to 0.11.8 cost nothing. A refused
-emitter does cost the build:
+The minimum is 0.13.0, the newest release on 2026-11-05. 0.13.0 added
+`microcosm-emitter status`, which the US Modal stage runner runs after each
+stage to record how many telemetry events the stage's emitter left
+undelivered; builds require the same release so that it is written once.
+Practically no machine ran the separately installed emitter then, so
+refusing 0.11.9 to 0.12.5 cost nothing; the minimum before it, 0.11.9, came
+from the collector's hostname, for which 0.10.0 would have been enough. A
+refused emitter does cost the build:
 
 - no hosted telemetry, and no staging hand-overs (the staging run files are
   still written locally);
@@ -190,17 +200,20 @@ emitter does cost the build:
   do by default, is refused before it starts, and the refusal gives the
   upgrade command built from the minimum.
 
-The build's warning says so. Raise the minimum and the image's floor
-together, and before raising them while machines still run older emitters,
-weigh these losses against having each such machine upgrade first.
+The build's warning says so. Raising the minimum in that one file raises
+the image's floor too; before raising it while machines still run older
+emitters, weigh these losses against having each such machine upgrade
+first. Raise the installed-emitter job's pin
+(`tools/installed-emitter/requirements.txt`) with it, so that the job never
+tests a release older than the minimum.
 
 To adopt a new interface release, regenerate `uv.lock`, update
 `APPROVED_UV_LOCK_SHA256` in `microcosm.build.us_runtime.worker_identity` to
 the new lockfile digest and rerun the tests and wheel checks.
 
 ## Migrating an existing workstation
 
-Install the emitter once with `uv tool install microcosm-emitter` (0.11.9 or
+Install the emitter once with `uv tool install microcosm-emitter` (0.13.0 or
 later; a build does not start an older emitter, says how to upgrade it and
 continues without it). It keeps using the existing spool database,
 `$XDG_CACHE_HOME/microcosm/telemetry/events.sqlite3`, at the same Alembic
@@ -213,7 +226,7 @@ modification. The emitter's README describes the repair for a machine that
 ran an earlier, unreleased build of the hand-over queue. No new credential,
 hosted service or infrastructure is required by this migration.
 
-To upgrade a machine whose emitter is older than 0.11.9, run
-`uv tool install --upgrade 'microcosm-emitter>=0.11.9'`; it also replaces a
+To upgrade a machine whose emitter is older than 0.13.0, run
+`uv tool install --upgrade 'microcosm-emitter>=0.13.0'`; it also replaces a
 version constraint given at install time, which `uv tool upgrade` keeps.
 `microcosm-emitter version` prints the installed release.