feat: harden OIDC and managed runtime delivery (AXI-182)

.github/workflows/release.yml

@@ -107,6 +107,15 @@ jobs:
           echo "Next version: ${{ steps.version.outputs.next }}"
           echo "Tag: ${{ steps.version.outputs.tag }}"
 
+      # Build release assets before the first irreversible push. This also runs
+      # during dry-run so packaging failures are discovered without publishing
+      # a version commit or immutable tag.
+      - name: Package and validate Hermes runtime helpers
+        run: |
+          bash scripts/package-hermes-helpers.sh "${{ steps.version.outputs.tag }}"
+          test -s dist/hermes-helpers/SHA256SUMS
+          (cd dist/hermes-helpers && sha256sum --check SHA256SUMS)
+
       - name: Update version
         if: ${{ !inputs.dry_run }}
         run: |
@@ -151,4 +160,5 @@ jobs:
             --title "$TAG" \
             --generate-notes \
             $PRERELEASE \
+            dist/hermes-helpers/* \
             --notes-start-tag "$(git describe --tags --abbrev=0 "$TAG^" 2>/dev/null || echo '')"

.gitignore

@@ -13,6 +13,8 @@ test-results
 .claude/
 .DS_Store
 *.log
+__pycache__/
+*.py[cod]
 prisma/migrations/dev.db*
 next-env.d.ts
 tsconfig.tsbuildinfo

DEVLOG.md

@@ -2,6 +2,43 @@
 
 > Append-only session log. Read at session start. Update at session end.
 
+## 2026-08-25 — Generic OIDC and recovery identity hardening
+
+- Required PKCE, state, and nonce checks explicitly for every generic OIDC
+  provider and added a strict local provider fixture that rejects weak state,
+  requires S256 PKCE, and binds the returned ID token to the request nonce.
+- Made external identity linking an explicit, confirmed account-security flow,
+  clarified that login identities do not grant integration access, surfaced
+  provider-link collisions, and covered link, unlink, and relink behavior.
+- Added a designated break-glass administrator to instance policy. Recovery
+  now uses an explicit route, records an instance audit event, exposes readiness
+  in admin settings, and protects the designated principal from demotion,
+  suspension, or deletion until recovery is reassigned or disabled.
+- Documented reverse-proxy callback bypass, forwarded-origin, sealed-cookie
+  header-buffer, and sensitive callback logging requirements for OIDC.
+
+## 2026-08-25 — Managed runtime execution-plane and Hermes hardening
+
+- Moved operator-requested runtime probes and no-tool self-tests onto a
+  dedicated BullMQ worker queue so their DNS, routing, and firewall plane is
+  the same one that dispatches managed `/v1/runs` work. Added bounded,
+  tenant-scoped diagnostic history with worker executor and manual/scheduled
+  trigger provenance.
+- Added a reviewable production Compose topology where the worker has outbound
+  egress without published ports or reverse-proxy membership, plus regression
+  coverage for the network contract and contradictory probe history.
+- Replaced `REMOTE_HTTP`-derived “remote webhook” copy with adapter-aware
+  managed-runtime and transport identity, and collapsed duplicate chat
+  readiness warnings when runtime health already reports the same probe fault.
+- Added a stable versioned User-Agent and sanitized HTTP/error classification
+  to the distributable Hermes Forge platform adapter.
+- Shipped versioned, deterministic `forge-presence` and `forge-provision`
+  helper sources, release packaging/checksums, and operator docs. Presence is a
+  script-only recurring heartbeat and remains silent on success.
+- Explicitly documented and tested that `runs.complete.completionCommentId`
+  accepts only a live BODY comment, and added one managed-runtime acceptance
+  path spanning assignment, external dispatch, inbox acknowledgement, output
+  start, final BODY comment, and terminal completion.
 ## 2026-08-25 — Restricted projects and explicit integration grants
 
 - Added workspace-visible and restricted projects with explicit Viewer,
@@ -31,7 +68,6 @@
   integration ceiling after defaults are resolved, and now revokes derived
   grants when credential source, exact app binding, or capability ceilings
   change.
-
 ## 2026-08-25 — Canonical local and external user identity foundation
 
 - Added a provider-neutral instance authentication policy for local-only,
@@ -14498,3 +14534,21 @@ reasoning output. Added integration coverage for delegated runtime completion,
 durable final comments, wait-request deduplication/resolution, waiting-reason
 preservation, and webhook relation precedence, plus lifecycle Playwright
 coverage for collapsed/expanded progress detail.
+
+## 2026-08-25 — AXI-182 final hardening review
+
+Closed the final managed-runtime and helper-distribution review gaps before
+release. Unexpected diagnostic worker failures now finalize their provenance
+row with a sanitized terminal result instead of leaving an indefinite pending
+state. A separate-process regression proves that runtime verification reports
+worker-plane failure even when the web/test process can reach the endpoint.
+
+Hermes presence heartbeats now authorize strictly through the authenticated
+key's linked agent identity without requiring broad workspace-member read
+access. Release automation builds and checksum-validates the versioned Hermes
+helper archives before any version or tag push, including during dry runs, and
+publishes those prevalidated assets after tagging.
+
+Verification: focused runtime-diagnostic, MCP-execution, and helper-distribution
+tests passed (11/11); TypeScript passed; lint completed with only the existing
+repository warnings; helper tarballs and `SHA256SUMS` verified successfully.

RELEASE.md

@@ -144,6 +144,14 @@ docker compose up -d                  # entrypoint runs `prisma migrate deploy`
 # run any one-time data backfill explicitly (prod image has no tsx — copy a .cjs + `node`)
 ```
 
+The deployment Compose must preserve the checked-in
+`docker/docker-compose.production.example.yml` network contract. In
+particular, `forge-worker` needs the private data network plus a non-internal
+egress network so managed-runtime probes and Runs dispatch use the same
+reachable execution plane. It must not publish ports or join the public proxy
+network. The web app may join the proxy network and must set
+`FORGE_DISABLE_IN_PROCESS_WORKER=1` when the sidecar is present.
+
 ### 4. Smoke test
 
 `forge.axiom-labs.dev` loads · sign-in works · core flows (issue create, agent dispatch,

docker/docker-compose.production.example.yml

@@ -0,0 +1,70 @@
+name: forge
+
+# Reviewable production topology. The worker receives outbound egress without
+# publishing ports and without joining the public reverse-proxy network.
+services:
+  postgres:
+    image: postgres:16-alpine
+    env_file: .env.production
+    volumes: ["forge-postgres:/var/lib/postgresql/data"]
+    networks: [forge-data]
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER:-forge}"]
+      interval: 5s
+      retries: 10
+
+  redis:
+    image: redis:7-alpine
+    volumes: ["forge-redis:/data"]
+    networks: [forge-data]
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      retries: 10
+
+  minio:
+    image: minio/minio:latest
+    command: server /data
+    env_file: .env.production
+    volumes: ["forge-minio:/data"]
+    networks: [forge-data]
+
+  forge:
+    build:
+      context: ${FORGE_SOURCE_PATH:?set FORGE_SOURCE_PATH to the exact tagged checkout}
+      target: runner
+      args: { GIT_SHA: "${GIT_SHA:-}", BUILD_TIME: "${BUILD_TIME:-}" }
+    env_file: .env.production
+    environment:
+      FORGE_DISABLE_IN_PROCESS_WORKER: "1"
+    depends_on:
+      postgres: { condition: service_healthy }
+      redis: { condition: service_healthy }
+    expose: ["3000"]
+    networks: [forge-data, forge-egress, forge-proxy]
+
+  forge-worker:
+    build:
+      context: ${FORGE_SOURCE_PATH:?set FORGE_SOURCE_PATH to the exact tagged checkout}
+      target: worker
+      args: { GIT_SHA: "${GIT_SHA:-}", BUILD_TIME: "${BUILD_TIME:-}" }
+    env_file: .env.production
+    depends_on:
+      postgres: { condition: service_healthy }
+      redis: { condition: service_healthy }
+    networks: [forge-data, forge-egress]
+
+networks:
+  forge-data:
+    internal: true
+  forge-egress:
+    # Non-internal bridge supplies outbound DNS/TCP without inbound exposure.
+    driver: bridge
+  forge-proxy:
+    external: true
+    name: ${FORGE_PROXY_NETWORK:-proxy}
+
+volumes:
+  forge-postgres:
+  forge-redis:
+  forge-minio:

docs/agents/runtime-credentials.md

@@ -152,7 +152,10 @@ Where to run it:
   **Settings → Runtimes → (a runtime) → Provisioning** (download button + ready
   bootstrap), or wire it as a session-start step.
 - **Persistent Hermes host:** install the `forge-provision` Hermes skill
-  (`~/.hermes/skills/forge-provision/`) — `bin/setup.sh <profile>` installs an
+  from the version-matched Forge release artifact
+  (`forge-provision-vX.Y.Z.tar.gz`, verified against `SHA256SUMS`) under
+  `~/.hermes/skills/forge-provision/`. Source installations may copy
+  `integrations/hermes/forge-provision/`. `bin/setup.sh <profile>` installs an
   hourly cron that fetches + runs the script, keeping the token + checkouts
   fresh. Companion to the `forge-presence` heartbeat skill; shares the same
   `forge.env` (`FORGE_URL` + `FORGE_API_KEY`).

docs/agents/runtime-modes.md

@@ -70,13 +70,20 @@ agent offline.
 
 ## The forge-presence Hermes skill
 
+Forge releases publish a versioned `forge-presence-vX.Y.Z.tar.gz` artifact and
+`SHA256SUMS`. Verify the checksum, unpack it below
+`~/.hermes/skills/forge-presence/`, then use its setup script. The source ships
+at `integrations/hermes/forge-presence/` for source-based installations.
+
 `~/.hermes/skills/forge-presence/` is a small cron-driven script that calls
 `agents.heartbeat` on behalf of a Hermes profile. It requires only:
 - `FORGE_URL` — the Forge instance base URL.
 - `FORGE_API_KEY` — an AGENT-kind API key with `linkedAgentId` set.
 
 The agent id is inferred from the key's `linkedAgentId`; no agent id needs to be
-hardcoded.
+hardcoded. Self-heartbeat does not require `READ_USERS` or another broad
+workspace scope: the linked agent identity is the complete authorization
+boundary for this endpoint.
 
 ### Setup for the default agent (Victor)
 
@@ -101,7 +108,8 @@ The `setup.sh` script registers a system cron entry that calls the heartbeat end
 every minute. Without the skill, a Hermes agent is still considered reachable whenever
 the worker successfully delivers a webhook (implicit heartbeat) — but chat shows
 "offline · queued" until the first delivery lands. With the skill, presence is honest
-from the moment Hermes starts.
+from the moment Hermes starts. The heartbeat is script-only, repeats indefinitely,
+and is silent on success; it never opens a Hermes session or wakes an LLM.
 
 ## Cross-references
 

docs/agents/runtimes.md

@@ -144,6 +144,17 @@ For containerised Codex app-server deployments, use the Docker bridge pattern in
 should call `runtimes.reportInfo` on boot so operators can see the bridge,
 Codex, container, and workspace versions from the runtime detail page.
 
+## Diagnostics use the execution worker
+
+**Test connection** and the no-tool **Self-test** enqueue a diagnostic on the
+same worker process that dispatches managed Runs. The runtime detail page keeps
+the most recent diagnostic history and labels each result with its worker
+executor plus its manual or scheduled trigger. This prevents a web-container
+network pass from being presented as proof that the worker can dispatch work.
+If an operator-requested diagnostic times out, inspect the worker queue and its
+outbound DNS/network policy; Forge does not fall back to probing from the web
+process.
+
 ## MCP tools (for runtimes that auto-register)
 
 `runtimes.register`, `runtimes.heartbeat`, `runtimes.configure`, and

docs/guide/instance-admin.md

@@ -90,6 +90,37 @@ minimum length, reset expiry, and lockout behavior. Provider/client secrets
 remain encrypted with `AUTH_SECRET`; the environment operator remains a
 separate recovery credential when break glass is enabled.
 
+Break-glass recovery is mapped to one designated, active instance
+administrator whose email matches `ADMIN_EMAIL`. Create and activate a
+dedicated local administrator under `/admin/users`, select it under **Identity
+& sign-in**, and keep it separate from a person's normal OIDC identity. The
+recovery form is `/signin/local?breakGlass=1`; successful uses are recorded in
+the instance security audit. Forge prevents demotion, suspension, or deletion
+of the designated account until recovery is reassigned or disabled.
+
+### Reverse proxy requirements for OIDC
+
+An outer forward-auth layer must not intercept Forge's exact Auth.js callback
+paths (`/api/auth/callback/<provider-id>`). The identity provider redirects the
+browser directly to Forge so Auth.js can validate the sealed state, PKCE, and
+nonce cookies. Keep the rest of the application behind the normal access
+policy, but configure the callback path as an explicit bypass in Authelia,
+Authentik, nginx `auth_request`, Traefik ForwardAuth, or an equivalent proxy.
+
+The proxy must also:
+
+- preserve the public host and scheme in `X-Forwarded-Host` and
+  `X-Forwarded-Proto`, with `AUTH_URL` set to that same public origin;
+- allow request and response header buffers large enough for Auth.js's sealed
+  PKCE, state, nonce, and session cookies—do not truncate or silently drop
+  multiple `Set-Cookie` headers;
+- avoid logging cookie values, authorization codes, or callback query strings.
+
+A provider redirect that repeatedly returns to sign-in, reports missing state,
+or succeeds only when outer authentication is disabled usually indicates a
+callback bypass or header-buffer problem rather than an IdP client-secret
+failure.
+
 All identity-policy and account-lifecycle mutations write the instance-wide
 security audit ledger with actor, target, request metadata, and timestamp.
 

docs/guide/settings.md

@@ -282,8 +282,11 @@ Configure how people sign in, without a redeploy:
   attempt threshold, and lockout duration. Passwords are stored as versioned
   scrypt hashes; raw passwords and raw reset/setup tokens are never persisted.
 - **Break glass** — keeps only the `ADMIN_EMAIL` / `ADMIN_PASSWORD` operator
-  credential available at `/signin/local`. It does not turn every local user
-  into a break-glass administrator.
+  credential available through the explicit `/signin/local?breakGlass=1`
+  recovery flow. Select one active instance administrator whose email matches
+  `ADMIN_EMAIL`; Forge audits recovery sign-ins and protects that account from
+  lifecycle changes until recovery is reassigned or disabled. Use a dedicated
+  local administrator rather than a person's normal OIDC identity.
 - **Add a provider** — pick a type:
   - **OpenID Connect (OIDC)** — the generic, discovery-based type. Covers
     any OIDC IdP: self-hosted **Authelia**, Authentik, Keycloak, or hosted

docs/reference/env.md

@@ -49,6 +49,13 @@ instance administrator can use `/signin/local` when external identity is
 unavailable. Keep them unique, protected, and available to the operator; do not
 reuse a person's normal password.
 
+After configuring these variables, create and activate a dedicated instance
+administrator with the same email and designate it under **Identity & sign
+in**. Recovery uses `/signin/local?breakGlass=1`; the ordinary local form never
+accepts the environment credential implicitly. See [Instance Admin → Reverse
+proxy requirements for OIDC](/guide/instance-admin.html#reverse-proxy-requirements-for-oidc)
+for callback bypass and header-buffer requirements.
+
 Authentication mode, registration, automatic provider redirect, password
 minimum length, reset expiry, and lockout thresholds are runtime database
 settings under **Identity & sign-in**. They are intentionally not environment
@@ -287,6 +294,17 @@ The instrumentation hook is a no-op when it detects an external worker is
 already serving the queue — workers coordinate via Redis, so it is safe to
 leave the in-process boot enabled even with a sidecar.
 
+### Worker outbound networking
+
+Managed runtime handshake checks, self-tests, periodic health sweeps, and
+`/v1/runs` dispatch execute in the worker. The reference production topology is
+`docker/docker-compose.production.example.yml`: the worker joins the internal
+data network and a dedicated non-internal egress bridge, publishes no ports,
+and does not join the public reverse-proxy network. A manual runtime diagnostic
+is queued to that worker rather than executed from the web container, so its
+result reflects the same DNS, routing, and firewall plane that dispatches real
+work.
+
 ## Cross-references
 
 - [/guide/architecture.html](/guide/architecture.html) — how these pieces