Add optional Runtime history and Dashboard durable time ranges

[sam2tom]

Goal

Add optional durable Runtime resource history for the Core Web Dashboard without changing current-snapshot execution ownership or making a telemetry backend mandatory for Sessions.

This follows the current-snapshot/API/Web work in #30 and Phase 4 of contracts/agents-api/runtime-observability-design.md.

Operator outcome

• Select a durable time range such as 1h, 6h, or 24h in Runtime monitoring.
• See CPU utilization, memory usage/limit, compute uptime boundaries, and observation coverage from an explicitly labeled history source.
• Keep browser-local live samples visually and semantically distinct when no history backend is configured.
• Preserve exact tenant, Session, Environment, and Runtime-incarnation attribution.

Proposed delivery slices

4A. Optional telemetry export

• Add an internal provider-neutral exporter seam after a validated current observation is produced.
• Export cumulative CPU seconds, CPU capacity, memory usage/limit, sample result, and sample duration using the instrument names documented in the design.
• Keep export asynchronous/bounded so backend failure never fails Session execution or the current observation read.
• Avoid high-cardinality global Prometheus labels. If Session or Runtime identity is exported, require an operator backend and tenant-isolated resource attributes suitable for authorized queries.
• No provider-native IDs, labels, paths, errors, or credentials.

4B. History query adapter and public Core extension

• Define a separate runtimehistory read interface; do not overload runtimeobs.Service or current observation routes.
• Resolve tenant and Session authorization in Core before querying the backend.
• Define range, step/downsampling, maximum points/series, whole-request deadline, and response byte limits.
• Fence series by provider-neutral Runtime incarnation so replacement/restart produces a gap rather than a false continuous line.
• Return explicit unsupported, unavailable, and partial/coverage metadata; never coerce missing samples to zero.
• Generate the OpenAPI extension and consume it only through packages/agents-client.

4C. Web durable ranges

• Add advertised durable ranges only when the history capability is configured.
• Keep Live as the existing browser-local bounded window and reset it on reload/navigation as today.
• Label the selected source and freshness. Never merge durable and ephemeral points without an explicit boundary.
• Preserve the current 2x2 CPU, memory, uptime, and token chart layout; token history remains sourced from canonical Session/Turn usage or a separately authorized history contract, not Runtime telemetry duplication.
• Retain accessible tabular summaries for latest value and missing points.

Boundaries

• No PostgreSQL time-series table or product database dependency.
• No lifecycle action, idle inference, suspension, restart, or allocation mutation.
• No Kubernetes, E2B, or self-hosted source implementation in this issue.
• No billing or cost data in Core Web.
• No browser access to OTLP/Prometheus credentials or direct backend queries.
• Current snapshot API remains available and correct when history is absent or unhealthy.

Acceptance

• Core starts and all execution/current-observation workflows pass with history fully disabled.
• Export/backend outages do not change execution, allocation, observation status, or API latency beyond the configured bounded handoff.
• Cross-tenant history reads are impossible and tested.
• Restart/allocation replacement/counter regression creates a gap; no synthetic interpolation or zero fill.
• Query budgets and retention limits are enforced in service and client tests.
• Core Web clearly distinguishes Live browser memory from durable backend ranges.
• Architecture, operator configuration, generated contract, coverage ledger, tests, and deployment acceptance are updated.

Open implementation decision

Qualify one operator backend and transport before implementation. Prefer OTLP-compatible export plus a tenant-authorized query adapter, but record the chosen backend's identity model, retention, downsampling, and query API before adding dependencies.

[sam2tom]

Phase 4 qualification update (PR #30, commit 8de7f12): public E2B Runtime evidence shows OTLP Collector fan-out, with ordinary operational metrics sent to Mimir and high-cardinality team_id + sandbox_id Runtime history sent to ClickHouse; authenticated reads enforce both identities, validate ranges, calculate query steps, and use bounded retention. Dify independently supports optional/no-op OTLP enablement but not the Runtime-history query boundary. Decision: keep a provider-neutral, bounded exporter handoff; use an optional operator Collector; keep a separate backend-neutral runtimehistory query adapter; treat ClickHouse as the first high-cardinality reference backend; do not use a shared Prometheus label filter as the tenant security boundary or let Web query telemetry backends directly. Implemented now: default-disabled sanitized async handoff, bounded queue/drop behavior, per-export timeout, panic isolation, deadline-aware shutdown, and tests proving provider receipts/raw provider text cannot reach export. Still intentionally unimplemented: concrete OTLP transport, ClickHouse/runtimehistory adapter, public history API, durable Web ranges, retention config, and exporter coverage telemetry. Verification: focused and race tests, go vet, complete Agents API + agents-client Go tests, and independent blind re-review all pass. Full make check remains locally gated by the required dedicated PostgreSQL URL; GitHub core-check is the full gate.

[sam2tom]

Phase 4A update on PR #30: commit fabcba4 adds the optional, default-disabled server-only OTLP/HTTP protobuf transport for validated Runtime observations. Configuration is strict and file-based; credentials/native provider identifiers/raw errors remain excluded. CPU/capacity/memory points now fail closed unless the sample carries the provider-qualified compute started_at incarnation fence; unfenced samples export coverage and read duration only. Focused tests, runtimeobs race tests, full Agents API + agents-client tests, go vet, server build, and independent blind re-review passed. Local make check still stops at the documented dedicated PostgreSQL URL prerequisite. Remaining Phase 4 work is unchanged: deployment-wide sampler, exporter coverage metrics, backend-neutral history query adapter, ClickHouse reference backend, tenant-scoped history API/client, and durable Dashboard ranges.

[sam2tom]

Phase 4 sampling update pushed to PR #30 in commit a1513ff. This adds optional execution-owner singleton background sampling (sample_interval_seconds 5..300), bounded deployment-wide managed-Session keyset scans, per-provider deadlines, lease-loss cancellation and pre-export ownership fencing, plus on_read/periodic OTLP classification. No Runtime lifecycle mutation or product database dependency is introduced. Verification: focused Runtime/store/server tests passed; runtimeobs race tests passed; full Agents API and agents-client Go tests passed; go vet, sqlc regeneration check, and server build passed; independent blind re-review reported no findings. Local make check still stops at the documented missing PARSAR_AGENTS_API_TEST_DATABASE_URL prerequisite; GitHub core-check remains authoritative.

[sam2tom]

Phase 4 query-boundary update pushed to PR #30 in commit 7169eab. Added backend-neutral internal/runtimehistory types/service plus a tenant-scoped Core store resolver. Reader calls now occur only after authenticated tenant/Session/Environment resolution; query ranges, server-selected resolution, total point budget, allocation+started_at incarnation fences, half-open bucket bounds, null/zero semantics, coverage, safe provider labels and numeric values are validated. OTLP exports now include Core resolved/observed nanosecond attributes so lower-precision generic metric tables can correlate one observation. No production Reader, public API, Web Durable range, product DB table or lifecycle behavior is enabled yet. Focused and full Agents API/client tests, vet and server build passed; independent blind re-review reported no findings. Local make check still stops at missing PARSAR_AGENTS_API_TEST_DATABASE_URL.

[sam2tom]

Runtime observability phase update (commit 2d48083, PR #30):

• Added authenticated public capability discovery: GET /v1/agents/runtime-history/capabilities.
• Added bounded Session-scoped history query: GET /v1/agents/sessions/{session_id}/runtime-history with inclusive start, exclusive end, and bounded max_points.
• Tenant identity is derived only from authentication. The caller supplies only the Session ID; Session ownership is resolved before any Reader call.
• Added fail-closed validation for request bounds, retention, canonical identities, response scope/range echoes, half-open ordered buckets, collection and aggregate limits, safe JSON numbers, temporal consistency, and lossless allocation incarnation fences.
• Added strict packages/agents-client methods and projection. Malformed, cross-scope, mixed-Environment, stale, oversized, duplicate, or temporally inconsistent responses are rejected as invalid upstream data.
• Updated generated OpenAPI and contract/architecture documentation. OpenAPI generation now preserves response collection limits, metric enums, and the provider-type pattern.
• Token history is intentionally absent; no token series is synthesized from Runtime samples.

Verification:

• Agents API, Go client, and contract tests passed.
• Agents client: 332 tests passed.
• Web: 592 unit tests passed; production build and typecheck passed.
• Playwright acceptance: 75/75 passed.
• go vet, SQLC consistency, and Agents API binary build passed.
• Independent blind review completed with no remaining findings.
• make check still stops only at the repository prerequisite requiring PARSAR_AGENTS_API_TEST_DATABASE_URL for a dedicated test PostgreSQL database.

Explicit remaining boundary: no production Runtime history Reader is configured and the Dashboard Durable source UI is not enabled yet. Current Dashboard trends remain Live/browser-local. The next slice is a real durable backend/Reader plus acceptance proving background collection, persistence, retention, incarnation separation, and tenant isolation before enabling Live / Durable in Web.

[sam2tom]

Sequencing update: Durable Runtime history remains tracked here, but persistence-layer implementation is intentionally deferred until the Live Dashboard effect is accepted. PR #30 now focuses on truthful browser-local time-series behavior and is deployed for review at commit 1771074. No ClickHouse dependency is enabled on the validation site; the temporary acceptance containers are stopped. We will resume this issue after the Live visual/data semantics are confirmed.

[sam2tom]

ClickHouse-backed Runtime history is now implemented in PR #30 and deployed to the authorized validation host.

Validated end to end:

• periodic Runtime samples -> OTLP Collector -> ClickHouse -> Core history API -> Dashboard
• 30-second sampling, seven-day retention, and bounded 1h/6h/24h queries
• Session-scoped, tenant-authorized reads with Environment and Runtime-incarnation fences
• CPU, memory, and compute-uptime History survives a Core restart and browser reload
• Live and History remain explicit sources; token history remains Live-only
• public validation site serves the updated Web bundle and reports History available

The deployed acceptance records are explicitly marked validation points and do not trigger model calls. Full Web, client, Playwright, PostgreSQL, Go, build, and Linux Rust gates are recorded in the PR description. Independent re-review returned no findings after the CPU-capacity gap fix.