feat(supervisor): stage gateway configuration snapshot delivery by pimlock · Pull Request #3244 · NVIDIA/OpenShell

pimlock · 2026-09-09T19:39:58Z

Summary

TL;DR Supervisors currently poll the gateway for configuration every 10 seconds. This PR adds an opt-in push path. Polling remains authoritative in 0.1.x.

Stage 1 sends full snapshots when the gateway setting is push. Each update reaches the gateway that owns the affected supervisor session. The supervisor ignores the snapshot. Its existing polling path still applies configuration.

The setting defaults to poll in 0.1.x. The plan makes push the default in 0.2.0, deprecates polling there, and removes polling APIs in 0.3.0. Helm users can set this field through gatewayConfig after #3384 lands.

Related Issue

Part of #1731. Stage 2 (#3265) will apply and acknowledge snapshots. Stage 3 (#3273) will track completion of global updates.

Changes

Add bootstrap, snapshot, and acknowledgement contracts with protocol revision checks.
Share snapshot builders with polling; coalesce updates and route them to the owning HA gateways.
Add the opt-in gateway setting while keeping polling and older supervisors working.
Initialize policy history without overwriting existing apply results.

Testing

mise run pre-commit; Rust workspace and server tests; mise run go:ci; mise run sdk:ts:ci
Helm lint and rendering; verified that feat(helm): migrate gateway configuration to gatewayConfig #3384 renders config_delivery_mode = "push" from gatewayConfig
Full mise run ci remains blocked by an existing Snap packaging assertion. Live HA E2E was not run locally.

Checklist

Conventional Commits and DCO sign-off
Existing polling APIs remain available in 0.1.x

github-actions · 2026-09-09T19:40:25Z

🌿 Preview your docs: https://nvidia-preview-pr-3244.docs.buildwithfern.com/openshell

copy-pr-bot · 2026-09-09T23:55:05Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-09-10T21:49:20Z

Label test:e2e applied for e7729ec. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

The startup repair that creates version-one policy history for legacy sandboxes propagated validation failures, so a single stored policy that no longer passes current validation rules prevented the gateway from starting. Skip such sandboxes with a warning and a completion summary so they keep the pre-repair behavior where only their own configuration reads report the failure. Store errors remain fatal. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Fleet-wide configuration changes spawned one snapshot build per connected sandbox and component with no concurrency limit, so a global setting or provider change issued every store query and credential-driver call at once. Gate builds behind a semaphore sized from the database pool and start the build deadline only once a permit is held. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Sandboxes keep their supervisor binary until they are recreated, so a gateway upgrade meets supervisors that predate the handshake and report revision zero. Rejecting them severs every running sandbox with no automatic recovery. Accept revision zero for one release, log a warning per session, and count them in openshell_supervisor_protocol_legacy_sessions_total. The supervisor mirrors the allowance for gateways that predate the handshake. Add a shared ConnectSupervisor test harness and handler-level tests for legacy acceptance and unknown-revision rejection. Move the skill troubleshooting paragraph out of the numbered deployment list so the list renders. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock

gator-agent

PR Review Status

I reconciled the current head against the full Gator feedback ledger and reviewed the author-only delta since bd9f91e5 in critical-only mode. The range-diff shows all ten PR-authored commits are patch-identical after the merge from main, the earlier fanout-bounding obligation remains resolved, and no new Critical defect was found. Current-head Branch Checks and E2E workflows are running.

Blocking findings:

No blocking findings remain

Carried findings:

None

Gator metadata

Validation: Project-valid Stage 1 of accepted issue #1731; the effective PR patch remains within the reviewed gateway/supervisor configuration-delivery scope.
Docs: Architecture and Fern gateway-reference documentation remain included for the current behavior.
Checks: OpenShell Helm Lint and Trivy Changes are green; Branch Checks and E2E are pending on the current head.
E2E: test:e2e and test:e2e-kubernetes are applied; the current-head Branch E2E Checks workflow is running.
Head SHA: 8dfe13b575a931586a5bb26238230227bdc40d82
Base SHA: fa0bfa490e42c87a74a70be6ebb40faee7fb8faa
Merge base SHA: fa0bfa490e42c87a74a70be6ebb40faee7fb8faa
Patch ID: ca184a2ffbb6b7e1d73cbd0fc2a97533eca331e3
Gator payload: 9
Review mode: critical_only
Previous reviewed SHA: bd9f91e593c2096542599aebaad43a2c179426e9
Review budget exhausted: yes
Maintainer decision required: no
Next state: gator:watch-pipeline

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 9, 2026 19:39

pimlock mentioned this pull request Sep 9, 2026

feat(supervisor): push gateway desired state #2967

Closed

6 tasks

This comment has been minimized.

Sign in to view

This comment was marked as outdated.

Sign in to view

pimlock added gator:in-review Gator is reviewing or awaiting PR review feedback gator:blocked Gator is blocked by process or repository gates and removed gator:in-review Gator is reviewing or awaiting PR review feedback labels Sep 10, 2026

This comment has been minimized.

Sign in to view

pimlock added the test:e2e Requires end-to-end coverage label Sep 10, 2026

This comment was marked as outdated.

Sign in to view

pimlock mentioned this pull request Sep 10, 2026

feat(supervisor)!: apply streamed configuration snapshots #3265

Open

7 tasks done

pimlock added this pull request to stack #3266 September 10, 2026 23:38

pimlock added 8 commits September 10, 2026 18:58

feat(supervisor): stage gateway configuration snapshot delivery

ea6c689

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore(sdk): regenerate Go proto bindings for protocol revision comment

fe2fa4e

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(supervisor): bound optional bootstrap latency on reconnect

3053b2f

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(server): bound configuration delivery workers

86773ee

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(server): update configuration tests for workspace selector

70773d3

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock force-pushed the 1731-config-update-stage-1/pimlock branch from e7729ec to 70773d3 Compare September 11, 2026 02:08

pimlock added 2 commits September 10, 2026 19:49

fix(proto): refresh public schema inventory

c645112

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into configuration snapshot branch

36bd16d

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock mentioned this pull request Sep 11, 2026

Push gateway-owned desired state to supervisors #1731

Open

This comment was marked as outdated.

Sign in to view

pimlock force-pushed the 1731-config-update-stage-1/pimlock branch from 7f704dd to 43d7b28 Compare September 16, 2026 18:27

pimlock added 7 commits September 16, 2026 11:32

chore: merge current main into configuration snapshot base

70ae8f2

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into configuration snapshot branch

4cb972f

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into configuration snapshot branch

e651690

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge latest main into config activation

33e8449

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

test(server): assert repairable invalid policy snapshot

1e49913

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into stage 1 configuration updates

99d99cc

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into stage 1 configuration updates

bd9f91e

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

This comment was marked as outdated.

Sign in to view

pimlock added the test:e2e-kubernetes Requires Kubernetes end-to-end coverage label Sep 19, 2026

This comment was marked as outdated.

Sign in to view

pimlock added gator:blocked Gator is blocked by process or repository gates and removed gator:approval-needed Gator completed review; maintainer approval needed labels Sep 19, 2026

chore: merge main into config snapshot stack

8dfe13b

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

pimlock commented Sep 19, 2026

View reviewed changes

pimlock added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 19, 2026

pimlock added 10 commits September 21, 2026 09:44

docs(architecture): correct configuration snapshot boundaries

7f88f5d

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot delivery

2091c38

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot delivery

f56e7f6

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot delivery

6344aad

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore: merge main into config snapshot stack

9d5a3f2

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

chore(config): merge main into gateway snapshot branch

d7cbee6

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

feat(config): route shadow push through HA owners behind gateway mode

c013645

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(config): coalesce direct and fleet updates per component

0e3d78b

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

fix(config): use generic Helm gateway configuration

b04c20f

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

docs(config): remove public guidance for internal push rollout

f1e8503

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

Navigation Menu

feat(supervisor): stage gateway configuration snapshot delivery - #3244

feat(supervisor): stage gateway configuration snapshot delivery#3244
pimlock wants to merge 31 commits into
mainfrom
1731-config-update-stage-1/pimlock

pimlock commented Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

copy-pr-bot Bot commented Sep 9, 2026

Uh oh!

This comment has been minimized.

This comment was marked as outdated.

Uh oh!

This comment has been minimized.

github-actions Bot commented Sep 10, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

pimlock left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pimlock commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

copy-pr-bot Bot commented Sep 9, 2026

Uh oh!

This comment has been minimized.

This comment was marked as outdated.

Uh oh!

This comment has been minimized.

github-actions Bot commented Sep 10, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

pimlock left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pimlock commented Sep 9, 2026 •

edited

Loading