From 9a3dfcc410179143ccd471294dd2954a2b13dedf Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 07:06:42 +0100 Subject: [PATCH 1/9] spec: VRRP first v6 advert-source is async-install, not FSM order Ground truth from the keepalived IPv6 interop lab plus the address-owner source: the first IPv6 advert sources from the macvlan's transient EUI-64 link-local, not fe80::1, because iface.RegisterOwnedAddresses (address_owner.go) applies asynchronously -- the kernel installs fe80::1 on a later reconcile pass, after the first SendAdvert has already resolved a source. Reordering promoteToMaster to install-first was tried and proven ineffective (ordering cannot win an async race) and only churns the spec-annotated AC-3/R-4 order. Record the root cause and a do-not-retry warning; the behavior is cosmetic (both are valid RFC 9568 link-local sources, keepalived accepts both, election/failover/dataplane unaffected). --- .../1122-vrrp-macvlan-vmac-dataplane.md | 27 +++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/plan/learned/1122-vrrp-macvlan-vmac-dataplane.md b/plan/learned/1122-vrrp-macvlan-vmac-dataplane.md index 158803013b..c7530455d2 100644 --- a/plan/learned/1122-vrrp-macvlan-vmac-dataplane.md +++ b/plan/learned/1122-vrrp-macvlan-vmac-dataplane.md @@ -66,11 +66,28 @@ worked: tentative (unreachable) for ~1s after every promotion. Proven end to end against keepalived IPv6 (QEMU): adverts accepted both ways, election, node-death failover, unsolicited NA burst, and the observer resolving the VIP to - 00:00:5e:00:02:{vrid}. One cosmetic quirk left un-fixed: the FIRST advert - sources from the macvlan's auto (EUI-64) link-local, not the configured - `fe80::1`, because the FSM sends the first advert before InstallVIPs puts - `fe80::1` on the device; the auto link-local is a valid RFC 9568 5.1.2.2 source - and keepalived accepts it, so it is a non-issue. + 00:00:5e:00:02:{vrid}. +- **The FIRST v6 advert sources from the macvlan's auto (EUI-64) link-local, not + `fe80::1` -- and this CANNOT be fixed by FSM action ordering. Do not try.** + Ground truth (QEMU, captured resolver output): the kernel gives the macvlan a + transient EUI-64 link-local (`fe80::200:5eff:fe00:20a`) at creation; installing + the VIPs REPLACES it, so at steady state the macvlan holds only `fe80::1` + + the global VIP (the auto link-local is gone). The source resolver + (`macvlanLinkLocal`) returns the first non-tentative link-local, so once VIPs + are installed it yields `fe80::1`. The first advert predates that: `InstallVIPs` + calls `iface.RegisterOwnedAddresses`, which is ASYNCHRONOUS (it registers the + addr in a map + fires a reconcile trigger and returns; the kernel apply happens + on a LATER reconcile pass -- `address_owner.go:80`). So the `SendAdvert` that + runs in the same dispatcher loop resolves before `fe80::1` exists and picks the + auto link-local. Reordering `promoteToMaster` to `InstallVIPs, SendAdvert, ...` + was tried and PROVEN INEFFECTIVE against the keepalived IPv6 lab (first advert + still from the auto link-local) because ordering cannot win an async race; it + only churns the spec-annotated AC-3/R-4 order. A real fix would gate the first + advert on `fe80::1` being present (delays the mastership claim by ~1 advert + interval -- a failover-latency regression) or make the shared address-owner + registry apply synchronously (invasive). Both are disproportionate: the auto + link-local is a valid RFC 9568 link-local source, keepalived accepts it, and + election/failover/dataplane are unaffected. Leave it; it is genuinely cosmetic. - **You are fighting the iface component for the parent's sysctls.** iface emits `arp_ignore`/`arp_filter`/`rp_filter` from unit config on every apply, which clobbers the recipe. The engine re-asserts on every config apply From 9af30c4400948ce7566886f9e5693287c47145bf Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 07:39:48 +0100 Subject: [PATCH 2/9] test(l2tp): install the route observer before Start(), not after TestPeerTeardownWithdrawsSubscriberRoute and TestDeadPeerKeepaliveTeardownWithdrawsRoute called SetRouteObserver AFTER buildLogReactorWithClock had already started the reactor, so the test goroutine's write to r.routeObserver raced the run loop's read in notifyRouteObserverDown. -race reports it on the first run, not 1-in-3: reactor_setters.go:114 (write) vs reactor_kernel.go:263 (read). This is the test misusing the API, not a product bug. SetRouteObserver documents 'MUST be called before Start(); the goroutine creation barrier synchronizes the write here with reads in the run loop', and the sole production caller honours it -- subsystem.go installs the observer at :241 and starts the reactor 72 lines later at :313, the same discipline its neighbour comment spells out for SetKernelWorker. The lock-free write is deliberate: the reload-time setters (setHelloRetries and friends) DO take tunnelsMu because subsystem_reload.go calls them on a live reactor, while the install-time setters trade the lock for the Start() happens-before edge. Adding a mutex would weaken a working design to accommodate a misusing test. Fixed by giving the builder an observer parameter that installs before Start(), via buildLogReactorWithClockObserver; the existing 11 callers are untouched and the wrapper passes nil (a documented no-op). setHelloRetries stays after Start(), which is safe -- it locks. Verified: the two tests 5/5 under -race, and the whole internal/component/l2tp/... tree green under -race (was: race on run 1). make ze-lint-changed: 0 issues. --- internal/component/l2tp/reactor_test.go | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/internal/component/l2tp/reactor_test.go b/internal/component/l2tp/reactor_test.go index ab86395eea..06760e4e65 100644 --- a/internal/component/l2tp/reactor_test.go +++ b/internal/component/l2tp/reactor_test.go @@ -1053,6 +1053,19 @@ func TestTunnelFSM_TieBreakerEqual(t *testing.T) { // //nolint:unparam // all current callers use 60s but the parameter keeps call sites self-documenting. func buildLogReactorWithClock(t *testing.T, now *time.Time, helloInterval time.Duration, secret string) (*UDPListener, *L2TPReactor, *lockedBuffer, func()) { + t.Helper() + return buildLogReactorWithClockObserver(t, now, helloInterval, secret, nil) +} + +// buildLogReactorWithClockObserver is buildLogReactorWithClock with a +// RouteObserver installed. It installs before Start() because that is the +// setter's contract: SetRouteObserver writes r.routeObserver with no lock and +// relies on Start()'s goroutine creation to publish the write to the run loop, +// which reads the field from the reactor goroutine. Installing after Start() +// is a genuine data race, not a timing flake -- unlike the reload-time setters +// (setHelloRetries and friends), which take tunnelsMu and are safe any time. +// A nil observer is a documented no-op, so the wrapper above passes nil. +func buildLogReactorWithClockObserver(t *testing.T, now *time.Time, helloInterval time.Duration, secret string, obs RouteObserver) (*UDPListener, *L2TPReactor, *lockedBuffer, func()) { t.Helper() buf := &lockedBuffer{} logger := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug})) @@ -1064,6 +1077,7 @@ func buildLogReactorWithClock(t *testing.T, now *time.Time, helloInterval time.D Defaults: TunnelDefaults{HostName: "ze-test", FramingCapabilities: 0x3, RecvWindow: 16, SharedSecret: secret}, Clock: func() time.Time { return *now }, }) + r.SetRouteObserver(obs) require.NoError(t, r.Start()) stop := func() { @@ -1170,11 +1184,9 @@ func (o *recordingRouteObserver) downs() [][2]uint16 { func TestPeerTeardownWithdrawsSubscriberRoute(t *testing.T) { now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) helloInterval := 60 * time.Second - ln, r, _, stop := buildLogReactorWithClock(t, &now, helloInterval, "") - defer stop() - spy := &recordingRouteObserver{} - r.SetRouteObserver(spy) + ln, r, _, stop := buildLogReactorWithClockObserver(t, &now, helloInterval, "", spy) + defer stop() client, localTID := driveToEstablished(t, ln, "") defer client.Close() @@ -1223,11 +1235,10 @@ func TestPeerTeardownWithdrawsSubscriberRoute(t *testing.T) { func TestDeadPeerKeepaliveTeardownWithdrawsRoute(t *testing.T) { now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) helloInterval := 5 * time.Second - ln, r, logs, stop := buildLogReactorWithClock(t, &now, helloInterval, "") + spy := &recordingRouteObserver{} + ln, r, logs, stop := buildLogReactorWithClockObserver(t, &now, helloInterval, "", spy) defer stop() - spy := &recordingRouteObserver{} - r.SetRouteObserver(spy) r.setHelloRetries(2) // dead-peer detection at 2 * 5s = 10s client, localTID := driveToEstablished(t, ln, "") From 9ff54bb8f407ebb075306d6828d04c6265e2b88f Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 07:39:48 +0100 Subject: [PATCH 3/9] test(install): fake kernel config needs CONFIG_MACVLAN for VRRP appliance-kernel-runtime failed with 'CONFIG_MACVLAN did not resolve to =y'. gokrazy/kernel/runtime.require gained CONFIG_MACVLAN for the VRRP macvlan work, but this test's fake docker kernel config was not updated to match, so enforceKernelRequirements rejected it (kernelreq.go:36 reads the require manifests at :41 and errors at :54 on any symbol that is not =y). The test file already documents exactly this coupling in the comment above the fake config: 'Adding a symbol to runtime.require (e.g. CONFIG_HUGETLBFS for VPP) requires adding it here too, or this test fails with did not resolve to =y'. This is that case; the fix is the one the comment prescribes. Worth noting for triage: run.sh does cd "$repo" and reads the WORKING TREE runtime.require, not the committed one. So this red reproduces against any commit you build while runtime.require is dirty, which is why it looked like a pre-existing failure rather than a consequence of the uncommitted VRRP change. Scope checked: appliance-kernel-runtime.ci is the only test carrying that fake config (sole file with CONFIG_VETH=y). kernel-wiring.ci also names --target runtime but only greps a make -n dry run, so it never enforces requirements. Verified: ze-test install --pattern appliance-kernel-runtime PASS; full install suite 37/37 PASS (3 environment skips). --- test/install/appliance-kernel-runtime.ci | 1 + 1 file changed, 1 insertion(+) diff --git a/test/install/appliance-kernel-runtime.ci b/test/install/appliance-kernel-runtime.ci index 02ac215776..faf33ee720 100644 --- a/test/install/appliance-kernel-runtime.ci +++ b/test/install/appliance-kernel-runtime.ci @@ -53,6 +53,7 @@ CONFIG_L2TP=y CONFIG_PPPOL2TP=y CONFIG_L2TP_V3=y CONFIG_VETH=y +CONFIG_MACVLAN=y CONFIG_BPF_SYSCALL=y CONFIG_BPF_JIT=y CONFIG_VIRTIO_NET=y From 4a60de6bc8b40f15ffc234d446ee274793393a65 Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 07:52:11 +0100 Subject: [PATCH 4/9] fix(chaos): keep virtual time moving through Run's teardown TestInProcessChaosReconnect failed 3/3 in 92.00s with established==1. It was logged as a -race flake ('passes 3/3 without -race'); it is neither flaky nor -race-specific -- it fails identically in both modes, i.e. a deterministic red, which plan/known-failures.md's own scope rule says never belongs there. Bisected (8 steps, git archive per commit -- git bisect needs a forbidden checkout) to 44ad25d23 'reconnect backoff floor 5s, not 120s'. That fix is CORRECT and stays: it only changed WHEN ze lands in the sleep below. The latent defect predates it. A goroutine dump 30s into the freeze pinned the mechanism, after three plausible theories (the new iface chaos weights; the blocking timer send at virtualclock.go:168; 'the advance loop is slow') were each disproven by experiment: runner -> simWg.Wait() (runner.go:594) -- advance loop ALREADY finished ze session -> VirtualClock.Sleep (virtualclock.go:49) from session.go:767 The advance loop costs what runner.go:427-430 implies -- 60s of virtual time in ~0.6s real -- then exits, and nothing advances the clock again. session.Run() polls for its connection with s.clock.Sleep(10ms) (session.go:762-768), and VirtualClock.Sleep is a bare <-ch (virtualclock.go:47-50): clock.Clock.Sleep takes no ctx, so simCancel() cannot reach a goroutine parked there -- only Advance can. ze's session was stranded mid-sleep, never finished the handshake, the simulator blocked forever on the reply that never came (executeReconnectStorm -> readMsg, simulator_actions.go:233), and simWg.Wait() hung until the caller's 90s context tore the sockets down. Hence 92.00s, and established==1 because the peer was asleep, not because reconnect was broken. Because the advance loop finishes in ~0.6s real, a chaos action firing late in the virtual window is still mid-handshake when time stops. The old 120s floor parked the retry outside the window and hid it. Fix: advance the clock from a goroutine during teardown until both the simulators and the reactor are down. Real time does not stop while a system shuts down, and neither may virtual time. runner.go:370 already warned that session.Run()'s handshake 'cannot complete until the virtual clock advances'; that requirement just did not survive past the loop. Verified: 3/3 PASS in 3.70s -- matching 3.69s measured at 8f5f2ff4b (2026-07-08, pre-regression) vs 92.00s broken. Full ./internal/chaos/... tree green; target test 2/2 green under -race. make ze-lint-changed: 0 issues. --- internal/chaos/inprocess/runner.go | 35 ++++++++ plan/known-failures.md | 88 +++++++++++-------- ...-fixit-redistribute-establishment-stall.md | 42 ++++++--- 3 files changed, 117 insertions(+), 48 deletions(-) diff --git a/internal/chaos/inprocess/runner.go b/internal/chaos/inprocess/runner.go index 6f871ace5b..3981e2120f 100644 --- a/internal/chaos/inprocess/runner.go +++ b/internal/chaos/inprocess/runner.go @@ -591,6 +591,37 @@ func Run(ctx context.Context, cfg RunConfig) (*RunResult, error) { // Stop simulators and reactor. simCancel() + + // Virtual time MUST keep moving while everything shuts down. + // + // session.Run() polls for its connection with s.clock.Sleep(10ms) + // (session.go:767), and VirtualClock.Sleep is a bare channel receive + // (virtualclock.go:49) -- clock.Clock.Sleep takes no ctx, so a goroutine + // parked there is unreachable by simCancel(): only Advance can release it. + // The advance loop above burns cfg.Duration of virtual time in + // (Duration/step)*stepDelay of REAL time -- 60s of virtual time in ~0.6s -- so + // a chaos action that fires late in the window is still mid-handshake when the + // loop exits. Stopping the clock here stranded ze's session mid-sleep forever: + // it never finished the handshake, the simulator blocked forever on the reply + // that therefore never came, and simWg.Wait() hung until the caller's context + // tore the sockets down. Real time does not stop while a system shuts down, + // and neither may virtual time. + stopAdvance := make(chan struct{}) + advanceDone := make(chan struct{}) + go func() { + defer close(advanceDone) + ticker := time.NewTicker(stepDelay) + defer ticker.Stop() + for { + select { + case <-stopAdvance: + return + case <-ticker.C: + vc.Advance(step) + } + } + }() + simWg.Wait() reactorCancel() @@ -598,6 +629,10 @@ func Run(ctx context.Context, cfg RunConfig) (*RunResult, error) { defer waitCancel() _ = reactor.Wait(waitCtx) + // Both are down, so nothing can still be waiting on the clock. + close(stopAdvance) + <-advanceDone + // Close events channel and wait for the drain goroutine to finish. close(events) <-eventsDone diff --git a/plan/known-failures.md b/plan/known-failures.md index 0c2b6275b2..ecb851e1b8 100644 --- a/plan/known-failures.md +++ b/plan/known-failures.md @@ -179,44 +179,56 @@ failing run before attributing. A separate `rsvpte-lsp-teardown` exit-2 (no stac in the 200-line capture) was seen once on 2026-07-13 and did not reproduce in 160 runs; it is not the same panic and its cause is unverified. -### `internal/chaos/inprocess` `TestInProcessChaosReconnect` -- fails in isolation since 44ad25d23; was flaky under `-race` - -Observed 2026-07-08 in `ze-verify-changed` (`-race`): `assert.Greater(established, -1)` failed with `established==1` at `runner_test.go:688`. Timing-dependent: chaos -rate 1.0 should disconnect and re-establish the peer, but under the `-race` build's -slowdown the re-establishment did not complete within the test window. Unrelated to -unify-replay -- `established` counts peer-FSM establishments, not route replay, and -`internal/chaos/inprocess` is untouched. - -**2026-07-16 update: the original "Passes 3/3 without `-race`" claim above was -removed because it is no longer true.** `44ad25d23` (`fix(bgp): reconnect backoff -floor 5s, not 120s connect-retry`) widened this from a `-race`-only flake into a -failure that reproduces WITHOUT `-race` whenever the test runs in isolation. -Measured with the Makefile's build tags per this file's rule above, patching -`peer.go` in place and restoring it: - -| Build | Result | -|-------|--------| -| pre-44ad25d23 (`reconnectMin := settings.ConnectRetry`) | PASS 2/2, ~4.3s | -| HEAD (`reconnectMin := DefaultReconnectMin`) | FAIL, 92.00s, `established==1` | - -Still order-dependent, so it stays in scope for this file: a full-package `-race` -run of `./internal/chaos/inprocess/` PASSED once, meaning `make ze-chaos-unit-test` -may be green while the isolated test is red. The 92.00s is the test's own 90s -context deadline (`runner_test.go:658`): `Run()` never reaches its 60s virtual -duration. With the old 120s floor the peer never dialed inside the window; with the -intended 5s floor it dials every 5s virtual and the advance loop burns 90s real, -though `runner.go:427-430` (`step = 1s` virtual, `stepDelay = 10ms` real) implies -under a second. Where that real time goes is UNVERIFIED. - -Do NOT resolve this by reverting 44ad25d23: the old 120s floor exceeded its own 60s -ceiling and contradicts `peer_run.go:19-25`. The harness documents the intended 5s -backoff (`runner_test.go:246`, `runner.go:518-522`), so this is chaos-harness -timing work, not a BGP defect. Full analysis, including the disproven -"reactor drops inbound while cycling" theory, is in -`plan/spec-fixit-redistribute-establishment-stall.md` - -Owner: the session that landed 44ad25d23 (spec-fixit-migrate-sleeps-infra work). +### ~~`internal/chaos/inprocess` `TestInProcessChaosReconnect`~~ -- FIXED 2026-07-16: the runner stopped virtual time while the system still needed it + +**Fixed in `runner.go` (Run's teardown). Never a flake, and never a BGP defect.** + +History: logged 2026-07-08 as a `-race` flake ("passes 3/3 without `-race`"); a +2026-07-16 update correctly bisected the widening to `44ad25d23` (`fix(bgp): +reconnect backoff floor 5s, not 120s connect-retry`) and correctly said DO NOT +revert it. It left one open question -- "the advance loop burns 90s real ... where +that real time goes is UNVERIFIED". + +**The premise was wrong: none of it is spent in `vc.Advance`.** A goroutine dump +taken 30s into the freeze pinned it in two stacks: + +| Goroutine | Where | +|---|---| +| runner | `simWg.Wait()` (`runner.go:594`) -- the advance loop had ALREADY finished | +| ze session | `VirtualClock.Sleep` (`virtualclock.go:49`) from `session.go:767` | + +The advance loop does exactly what `runner.go:427-430` implies: 60 virtual seconds +in ~0.6s real. Then it exits -- and **nothing advances the clock again**. +`session.Run()` polls for its connection with `s.clock.Sleep(10ms)` +(`session.go:762-768`), and `VirtualClock.Sleep` is a bare `<-ch` +(`virtualclock.go:47-50`); `clock.Clock.Sleep` takes no ctx, so `simCancel()` +cannot reach a goroutine parked there -- only `Advance` can. ze's session was +stranded mid-sleep, never finished the handshake, the simulator blocked forever on +the reply that never came (`executeReconnectStorm` -> `readMsg`, +`simulator_actions.go:233`), and `simWg.Wait()` hung until the test's own 90s +context tore the sockets down. Hence 92.00s, and `established==1` because the peer +was asleep, not because reconnect was broken. + +`44ad25d23` only changed WHEN ze lands in that sleep: the advance loop finishes in +~0.6s real, so a chaos action firing late in the virtual window is still +mid-handshake when time stops. The 120s floor parked the retry outside the window +and hid it. The latent defect predates it. + +Fix: keep advancing the virtual clock during teardown, until both the simulators +and the reactor are down. Real time does not stop while a system shuts down, and +neither may virtual time. + +Verified: 3/3 PASS in **3.70s** -- matching the 3.69s measured at `8f5f2ff4b` +(2026-07-08, before the regression), vs 92.00s broken. Full `./internal/chaos/...` +tree green; the target test 2/2 green under `-race`. `make ze-lint-changed`: 0 issues. + +Two lessons worth keeping. (1) This was logged as non-deterministic but failed +**3/3 in BOTH modes** -- a deterministic red, which this file's own scope rule says +never belongs here. Re-measure before inheriting a "flaky" label. (2) Three +plausible mechanisms (the new iface chaos weights; the blocking timer send at +`virtualclock.go:168`; "the advance loop is slow") were each disproven by +experiment. The goroutine dump settled in one run what code-reading had got wrong +three times: when a test hangs, dump the stacks before theorising. ### `reload` suite -- 6 iface tunnel/wireguard tests time out without CAP_NET_ADMIN (unprivileged sandbox) diff --git a/plan/spec-fixit-redistribute-establishment-stall.md b/plan/spec-fixit-redistribute-establishment-stall.md index 4036bd694d..12c1147e45 100644 --- a/plan/spec-fixit-redistribute-establishment-stall.md +++ b/plan/spec-fixit-redistribute-establishment-stall.md @@ -37,7 +37,7 @@ recorded here so the next session does not repeat them. `SetReconnectDelay` overrides, so unaffected. Verified: converted `announce` 25/25 (was ~80% flaky pre-fix); reactor package unit tests green. -### Regression caused by that fix (OPEN, 2026-07-16) +### Regression caused by that fix (RESOLVED 2026-07-16 -- fixed in `runner.go`, see end of section) 44ad25d23 widened `internal/chaos/inprocess` `TestInProcessChaosReconnect`, already logged flaky-under-`-race` in `plan/known-failures.md` since 2026-07-08, into a failure that also reproduces WITHOUT `-race` when the test runs in isolation. @@ -55,19 +55,41 @@ Order-dependent: a full-package `-race` run of `./internal/chaos/inprocess/` PAS `make ze-chaos-unit-test` (`mk/test-chaos.mk:27`, `go test -race ./internal/chaos/...`) may still be green. The isolation failure is the reliable reproducer. -Mechanism, partly pinned: the 92.00s is the test's own 90s context deadline -(`runner_test.go:658`); `Run()` never reaches the 60s virtual duration (`runner_test.go:664`). -`runner.go:427-430` sets `step = 1s` virtual and `stepDelay = 10ms` real, so 60 iterations -should cost well under a second. With the 120s floor the peer never dialed inside the window, -so that cost never appeared; with the intended 5s floor it dials every 5s virtual and the loop -burns 90s real. WHERE that real time goes inside `vc.Advance` is UNVERIFIED and is the open -question. `MockDialer.DialContext` fails fast (`mocknet.go:118-120`) and the virtual timer -channel is buffered (`virtualclock.go:81`), so neither is the stall. - This is chaos-harness work, NOT a BGP defect: the harness itself documents the intended 5s backoff (`runner_test.go:246`, `runner.go:518-522` "DefaultReconnectMin = 5s virtual"). The test was green only because the 120s bug parked the retry loop outside the window. +**RESOLVED 2026-07-16. The open question ("where does the real time go inside `vc.Advance`") +rested on a false premise: none of it is spent there, and the advance loop is not slow.** + +A goroutine dump taken 30s into the freeze answered it in one run: + +| Goroutine | Where | +|---|---| +| runner | `simWg.Wait()` (`runner.go:594`) -- the advance loop had ALREADY finished | +| ze session | `VirtualClock.Sleep` (`virtualclock.go:49`) from `session.go:767` | + +The advance loop costs exactly what `runner.go:427-430` implies (~0.6s real for 60s virtual) +and then EXITS -- after which nothing advances the clock. `session.Run()` polls for its +connection with `s.clock.Sleep(10ms)` (`session.go:762-768`) and `VirtualClock.Sleep` is a +bare `<-ch` (`virtualclock.go:47-50`). `clock.Clock.Sleep` takes no ctx, so `simCancel()` +cannot reach a goroutine parked there -- only `Advance` can. ze's session was stranded +mid-sleep, never completed the handshake, the simulator blocked forever on the reply that +never came (`executeReconnectStorm` -> `readMsg`, `simulator_actions.go:233`), and +`simWg.Wait()` hung until the 90s context tore the sockets down. That is the 92.00s, and +`established==1` because the peer was asleep -- not because reconnect was broken. + +44ad25d23 is exonerated as a cause and stays: it only changed WHEN ze lands in that sleep. +Because the advance loop finishes in ~0.6s real, a chaos action firing late in the virtual +window is still mid-handshake when time stops; the 120s floor parked the retry outside the +window and hid a defect that predates it. + +Fix (`runner.go`, Run's teardown): keep advancing the virtual clock until both the simulators +and the reactor are down. Real time does not stop while a system shuts down, and neither may +virtual time. Verified 3/3 PASS in 3.70s, matching the 3.69s measured at `8f5f2ff4b` +(pre-regression) vs 92.00s broken; `./internal/chaos/...` green; target test 2/2 green under +`-race`; `make ze-lint-changed` 0 issues. `plan/known-failures.md` entry closed. + Do NOT "fix" this by reverting 44ad25d23: the 120s floor exceeded the 60s ceiling and contradicts `peer_run.go:19-25`, which documents this loop as deliberately replacing the RFC 4271 ConnectRetryTimer with "min 5s, max 60s". From 0b32599494d4c99c1a9c85a68b011f15fe4f6e57 Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 08:12:56 +0100 Subject: [PATCH 5/9] fix(config): bracket leaf-list must store the slice, not just the mirror parseBracketLeafList stored only the joined scalar mirror (tree.Set), never the members (tree.SetSlice). Set writes t.values (tree.go:141-152); SetSlice writes t.multiValues (tree.go:176-181); GetSlice reads multiValues (tree.go:186-190). Different maps -- so GetSlice returned nil for EVERY bracket leaf-list, and ToMap emitted the joined text as one string. Consumers saw '[ a b ]' as the single value 'a b'. The sibling path storeValueOrArray (parser_list.go:414) always did both; this one just forgot the SetSlice. Proven, not inferred: the new test run against the unfixed parser gives expected: []string{"foo", "bar", "baz"} actual : []string(nil) Reachable from real config, not just the hand-built schema: yang_schema.go:319-322 builds a BracketLeafListNode for any leaf annotated ze:syntax "bracket", and three modules use it -- ze-iface-conf.yang address (:266, :354), member (:608), allowed-ips (:1004), sysctl-profile (:253) ze-vrrp-conf.yang virtual-address (:54, :163) -- min-elements 1, max-elements 16 exabgp.yang processes, processes-match Two production consumers read those back as slices and got nil: config/graph.go:339 (fam.GetSlice("address")) and web/page_interfaces.go:157 (familyTree.GetSlice("address")). So an interface unit could not carry two addresses -- ParseAddr saw the joined "10.0.0.1/24 10.0.0.2/24" as one value. It stayed invisible because every config in the tree used exactly one member. Get() still returns the joined form, so the consumers reading the scalar mirror are unaffected. Verified: ./internal/component/config/... , ./internal/component/bgp/config/ and ./internal/component/iface/ all green with the fix; the new test FAILS on the unfixed parser (above). --- internal/component/config/parser_list.go | 7 ++++++ internal/component/config/parser_test.go | 30 ++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/internal/component/config/parser_list.go b/internal/component/config/parser_list.go index 3b429569e8..0ea5e059ac 100644 --- a/internal/component/config/parser_list.go +++ b/internal/component/config/parser_list.go @@ -332,6 +332,13 @@ func (p *Parser) parseBracketLeafList(tree *Tree, name string, node *BracketLeaf } } + // Store the members AND the joined scalar mirror, exactly like the sibling + // leaf-list path (storeValueOrArray). Without SetSlice, GetSlice returns nil + // and ToMap emits the joined text as one string, so every consumer sees + // `[ a b ]` as the single value "a b" -- which is why a unit could not carry + // two addresses. Get() keeps returning the joined form for the consumers + // that read the mirror. + tree.SetSlice(name, items) tree.Set(name, textbuf.Join(items, " ")) return nil } diff --git a/internal/component/config/parser_test.go b/internal/component/config/parser_test.go index f661af883c..b95f7bed8f 100644 --- a/internal/component/config/parser_test.go +++ b/internal/component/config/parser_test.go @@ -500,6 +500,36 @@ func TestParserArray(t *testing.T) { require.Equal(t, "foo bar baz", val) // stored space-separated } +// TestParserArrayStoresSlice verifies a bracket leaf-list is stored as a SLICE, +// not only as the joined scalar mirror. +// +// VALIDATES: every member of `name [ a b ]` is retrievable as its own value, so +// consumers (and the JSON delivered to plugins via ToMap) see a list. +// +// PREVENTS: the multi-member regression this test was written for -- +// parseBracketLeafList used to call only tree.Set(joined), so GetSlice returned +// nil and ToMap emitted "a b" as ONE string. Every consumer then parsed the +// joined text as a single value: `interface ... ipv4 { address [ a b ]; }` +// failed with `ParseAddr("10.0.0.1/24 10.0.0.2/24")`, i.e. ze could not put two +// addresses on a unit at all. It stayed invisible because every config in the +// tree used exactly one member. The sibling path (storeValueOrArray) always did +// SetSlice + Set; this one just forgot the SetSlice. +func TestParserArrayStoresSlice(t *testing.T) { + schema := NewSchema() + schema.Define("items", BracketLeafList(TypeString)) + + p := NewParser(schema) + tree, err := p.Parse(`items [ foo bar baz ]`) + require.NoError(t, err) + + require.Equal(t, []string{"foo", "bar", "baz"}, tree.GetSlice("items")) + + // The scalar mirror stays joined: existing consumers read it via Get. + val, ok := tree.Get("items") + require.True(t, ok) + require.Equal(t, "foo bar baz", val) +} + // TestParserArraySingle verifies single-item array. // // VALIDATES: Single item arrays work. From bc4626ebbaebdb315181fd299a7071947e716920 Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 08:30:27 +0100 Subject: [PATCH 6/9] feat: add reproducible terminal demos --- .github/workflows/pages.yml | 42 ++- Makefile | 1 + cmd/ze/hub/session_factory_test.go | 21 ++ demos/terminal/.gitignore | 3 + demos/terminal/Dockerfile | 12 + demos/terminal/cards.sh | 184 +++++++++++ demos/terminal/cli-dashboard/demo.tape | 57 ++++ demos/terminal/cli-dashboard/run.sh | 57 ++++ demos/terminal/cli-dashboard/transcript.txt | 5 + demos/terminal/cli-dashboard/ze.conf | 93 ++++++ demos/terminal/common.tape | 10 + demos/terminal/container-entrypoint.sh | 36 +++ demos/terminal/install-vhs.sh | 98 ++++++ demos/terminal/launcher/demo.tape | 56 ++++ demos/terminal/launcher/transcript.txt | 6 + demos/terminal/manifest.json | 63 ++++ demos/terminal/rbac/demo.tape | 50 +++ demos/terminal/rbac/rbac.conf | 34 ++ demos/terminal/rbac/run.sh | 47 +++ demos/terminal/rbac/transcript.txt | 6 + demos/terminal/render.py | 324 ++++++++++++++++++++ demos/terminal/shell.sh | 16 + demos/terminal/traceroute/demo.tape | 47 +++ demos/terminal/traceroute/run.sh | 92 ++++++ demos/terminal/traceroute/transcript.txt | 5 + demos/terminal/traceroute/ze.conf | 9 + demos/terminal/zefs-config/demo.tape | 76 +++++ demos/terminal/zefs-config/run.sh | 56 ++++ demos/terminal/zefs-config/transcript.txt | 13 + demos/terminal/zefs-config/ze.conf | 42 +++ docs/guide/production-diagnostics.md | 2 + docs/guide/quickstart.md | 2 + docs/guide/terminal-demonstrations.md | 31 ++ mk/terminal-demo.mk | 13 +- 34 files changed, 1600 insertions(+), 9 deletions(-) create mode 100644 demos/terminal/.gitignore create mode 100644 demos/terminal/Dockerfile create mode 100755 demos/terminal/cards.sh create mode 100644 demos/terminal/cli-dashboard/demo.tape create mode 100755 demos/terminal/cli-dashboard/run.sh create mode 100644 demos/terminal/cli-dashboard/transcript.txt create mode 100644 demos/terminal/cli-dashboard/ze.conf create mode 100644 demos/terminal/common.tape create mode 100755 demos/terminal/container-entrypoint.sh create mode 100755 demos/terminal/install-vhs.sh create mode 100644 demos/terminal/launcher/demo.tape create mode 100644 demos/terminal/launcher/transcript.txt create mode 100644 demos/terminal/manifest.json create mode 100644 demos/terminal/rbac/demo.tape create mode 100644 demos/terminal/rbac/rbac.conf create mode 100755 demos/terminal/rbac/run.sh create mode 100644 demos/terminal/rbac/transcript.txt create mode 100755 demos/terminal/render.py create mode 100755 demos/terminal/shell.sh create mode 100644 demos/terminal/traceroute/demo.tape create mode 100755 demos/terminal/traceroute/run.sh create mode 100644 demos/terminal/traceroute/transcript.txt create mode 100644 demos/terminal/traceroute/ze.conf create mode 100644 demos/terminal/zefs-config/demo.tape create mode 100755 demos/terminal/zefs-config/run.sh create mode 100644 demos/terminal/zefs-config/transcript.txt create mode 100644 demos/terminal/zefs-config/ze.conf create mode 100644 docs/guide/terminal-demonstrations.md diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index f280a4a08a..6188dbc6a6 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -1,11 +1,13 @@ -name: Deploy Pages +name: Deploy website on: push: - branches: [ "main" ] + branches: ["main"] paths: - - "pages/**" - ".github/workflows/pages.yml" + - "demos/terminal/**" + - "docs/**" + - "mk/terminal-demo.mk" workflow_dispatch: permissions: @@ -24,8 +26,38 @@ jobs: url: ${{ steps.deployment.outputs.page_url }} runs-on: ubuntu-latest steps: - - name: Checkout repository + - name: Checkout Ze uses: actions/checkout@v4 + with: + ref: main + path: main + + - name: Checkout website + uses: actions/checkout@v4 + with: + ref: gh-pages + path: site + + - name: Set up Go + uses: actions/setup-go@v5 + with: + go-version-file: main/go.mod + cache-dependency-path: main/go.sum + + - name: Set up uv + uses: astral-sh/setup-uv@v6 + + - name: Install VHS and ffmpeg + run: | + make -C main ze-terminal-demo-tools + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + + - name: Generate terminal media + run: make -C main ze-terminal-demos-release TERMINAL_DEMO_OUTPUT="$GITHUB_WORKSPACE/site/assets/demos" + + - name: Build website + run: uv run --with markdown python3 tools/build.py + working-directory: site - name: Configure Pages uses: actions/configure-pages@v5 @@ -33,7 +65,7 @@ jobs: - name: Upload Pages artifact uses: actions/upload-pages-artifact@v3 with: - path: pages + path: site - name: Deploy Pages id: deployment diff --git a/Makefile b/Makefile index 47caee3ded..67142df9de 100644 --- a/Makefile +++ b/Makefile @@ -591,6 +591,7 @@ help-test: @echo " ze-perf-gate Perf bench (ze DUT) + regression check" @echo " ze-release-assets Rebuild every release-owned website asset" @echo " ze-terminal-demos-release Re-record all terminal demos for this release" + @echo " ze-terminal-demo-tools Install native VHS, ffmpeg, and ttyd (macOS/Ubuntu)" @echo "" @echo " Escalation: single test -> package -> component group -> ze-verify" @echo " See docs/contributing/testing.md for the full workflow." diff --git a/cmd/ze/hub/session_factory_test.go b/cmd/ze/hub/session_factory_test.go index 625c5ee6b3..ab7a056234 100644 --- a/cmd/ze/hub/session_factory_test.go +++ b/cmd/ze/hub/session_factory_test.go @@ -90,3 +90,24 @@ func TestSessionEditorWithoutReloadFn(t *testing.T) { assert.False(t, ed.HasReloadNotifier(), "nil reload function must not register a notifier") } + +// TestDashboardFactoryUsesPublicSummaryCommand verifies that SSH dashboard +// polling uses the registered CLI path rather than the internal RPC nickname. +// +// VALIDATES: monitor bgp reaches the live BGP summary over an SSH session. +// PREVENTS: a healthy dashboard rendering an empty peer table. +func TestDashboardFactoryUsesPublicSummaryCommand(t *testing.T) { + var command string + factory := dashboardFactoryFromExecutor(func(input string) (string, error) { + command = input + return `{"summary":{"peers-configured":3}}`, nil + }) + + poller, err := factory() + require.NoError(t, err) + output, err := poller() + require.NoError(t, err) + + assert.Equal(t, "show bgp summary", command) + assert.JSONEq(t, `{"summary":{"peers-configured":3}}`, output) +} diff --git a/demos/terminal/.gitignore b/demos/terminal/.gitignore new file mode 100644 index 0000000000..a60e11c70f --- /dev/null +++ b/demos/terminal/.gitignore @@ -0,0 +1,3 @@ +**/meta/ +**/rollback/ +**/ze.audit.jsonl diff --git a/demos/terminal/Dockerfile b/demos/terminal/Dockerfile new file mode 100644 index 0000000000..ee501471bb --- /dev/null +++ b/demos/terminal/Dockerfile @@ -0,0 +1,12 @@ +FROM ghcr.io/charmbracelet/vhs:v0.11.0@sha256:9d5fc3dc0c160b0fb1d2212baff07e6bdf3fa9438c504a3237484567302fcf93 + +USER root + +RUN apt-get update \ + && apt-get install --yes --no-install-recommends iproute2 iputils-ping openssh-client procps sshpass \ + && rm -rf /var/lib/apt/lists/* + +COPY container-entrypoint.sh /usr/local/bin/terminal-demo-entrypoint + +WORKDIR /src +ENTRYPOINT ["/usr/local/bin/terminal-demo-entrypoint"] diff --git a/demos/terminal/cards.sh b/demos/terminal/cards.sh new file mode 100755 index 0000000000..9ba0e8de61 --- /dev/null +++ b/demos/terminal/cards.sh @@ -0,0 +1,184 @@ +#!/usr/bin/env bash +set -euo pipefail + +clear +case "${1:-}:${2:-}" in + launcher:intro) + cat <<'EOF' + + Ze command launcher + =================== + + Goal: discover Ze commands without memorizing the command tree. + + This recording will: + 1. Open the interactive launcher. + 2. Filter the Show commands down to traceroute. + 3. Return to the root and find the doctor command. + + Watch the breadcrumb and filter labels as the menu changes. +EOF + ;; + launcher:recap) + cat <<'EOF' + + WHAT THIS PROVED + ================= + + Type to filter. Enter drills down. Escape moves back. + + The launcher is generated from Ze's live command registry, so it stays + aligned with the commands available in the installed binary. + + Recording complete. +EOF + ;; + cli-dashboard:intro) + cat <<'EOF' + + Ze live BGP dashboard + ===================== + + Goal: inspect active BGP sessions without leaving the CLI. + + This recording will: + 1. Connect through Ze's SSH management plane. + 2. Open the continuously refreshed BGP dashboard. + 3. Sort the peer table and inspect one session in detail. + + Keys used: s sorts, arrows move, Enter opens detail, Escape returns. +EOF + ;; + cli-dashboard:recap) + cat <<'EOF' + + WHAT THIS PROVED + ================= + + One interactive view combines peer state, uptime, message counters, + update rates, and per-session detail. The data refreshes while you work. + + Recording complete. +EOF + ;; + zefs-config:intro) + cat <<'EOF' + + Create ZeFS, then configure Ze over SSH + ======================================= + + Goal: show the complete configuration path from storage to commit. + + This recording will: + 1. Create a fresh ZeFS database with ze init. + 2. List and validate the stored configuration. + 3. Connect to Ze's SSH configuration editor. + 4. Change a setting, review the diff, commit, and verify it live. +EOF + ;; + zefs-config:ssh) + cat <<'EOF' + + STORAGE CHECK COMPLETE + ====================== + + ZeFS now contains a valid active configuration. + + Next: connect over SSH, change the default CLI output format, inspect + the pending diff, commit it, then run an operational command. +EOF + ;; + zefs-config:recap) + cat <<'EOF' + + WHAT THIS PROVED + ================= + + ze init created the ZeFS database. The SSH editor changed a draft, + show | compare exposed the exact change, and commit made it active. + + No configuration file was edited on the running router. + + Recording complete. +EOF + ;; + rbac:intro) + cat <<'EOF' + + Prove role-based command authorization + ====================================== + + Goal: verify that a read-only NOC account can observe but not change state. + + The NOC user holds the read-only profile: + run ... default allow, deny "debug", deny "clear" + edit ... default deny + + This recording will: + 1. Run show version successfully as the NOC user. + 2. Ask the same user to clear interface counters. + 3. Observe an explicit access-control denial. + + Passwords are injected outside the recording. +EOF + ;; + rbac:deny) + cat <<'EOF' + + Next: test the denied path + ========================== + + The profile denies every command matching the "clear" prefix. + + Ze resolves the command first, then checks the profile, and refuses + before the command runs. The response names the reason explicitly: + "command restricted by access control". +EOF + ;; + rbac:recap) + cat <<'EOF' + + WHAT THIS PROVED + ================= + + The same authenticated user could run show version but could not clear + interface counters. Authorization is enforced by the daemon, not by + shell policy, and a refusal is reported as such rather than as a typo. + + Recording complete. +EOF + ;; + traceroute:intro) + cat <<'EOF' + + Trace a live path without the Internet + ====================================== + + Goal: show Ze's one-shot and continuously refreshed traceroute views. + + This recording will: + 1. Probe 192.0.2.53 through an isolated Linux namespace router. + 2. Show the path once as an operational command. + 3. Monitor per-hop loss and latency over several rounds. + + The lab uses documentation addresses. No public host or DNS is required. +EOF + ;; + traceroute:recap) + cat <<'EOF' + + WHAT THIS PROVED + ================= + + Ze sent real ICMP probes through the isolated lab and measured every hop. + The live view continuously updated loss, latency, and variation without + contacting any third-party service. + + Recording complete. +EOF + ;; + *) + printf 'usage: %s \n' "$0" >&2 + exit 2 + ;; +esac diff --git a/demos/terminal/cli-dashboard/demo.tape b/demos/terminal/cli-dashboard/demo.tape new file mode 100644 index 0000000000..0a8ced9c82 --- /dev/null +++ b/demos/terminal/cli-dashboard/demo.tape @@ -0,0 +1,57 @@ +Output artifacts/cli-dashboard.webm +Source common.tape + +Hide +Type "export ZE_CONFIG_DIR=/src/tmp/terminal-demos/state/cli-dashboard/config ZE_SSH_PASSWORD=secret123 SSHPASS=secret123" +Enter +Type "/src/demos/terminal/cli-dashboard/run.sh start" +Enter +Wait+Screen /terminal demo ready/ +Type "bash /src/demos/terminal/cards.sh cli-dashboard intro" +Enter +Show +Sleep 10s + +Hide +Type "clear" +Enter +Show +Type "sshpass -e ssh ze-demo" +Enter +Wait+Screen /ze#/ +Sleep 3s +Type "exit" +Enter +Wait+Screen /ze>/ +Sleep 3s +Type "monitor bgp" +Enter +Wait+Screen /connected/ +Sleep 6s + +Type "s" +Sleep 4s +Down +Sleep 3s +Enter +Wait+Screen /Peer Detail/ +Sleep 6s +Screenshot artifacts/cli-dashboard.png + +Escape +Sleep 3s +Type "q" +Wait+Screen /ze>/ +Sleep 2s +Type "exit" +Enter +Wait+Screen /\$ / + +Hide +Type "/src/demos/terminal/cli-dashboard/run.sh stop" +Enter +Wait+Screen /terminal demo stopped/ +Type "bash /src/demos/terminal/cards.sh cli-dashboard recap" +Enter +Show +Sleep 8s diff --git a/demos/terminal/cli-dashboard/run.sh b/demos/terminal/cli-dashboard/run.sh new file mode 100755 index 0000000000..03660f3cbb --- /dev/null +++ b/demos/terminal/cli-dashboard/run.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +set -euo pipefail + +state=/src/tmp/terminal-demos/state/cli-dashboard +config_dir="${state}/config" +pid_file="${state}/pids" +log_file="${state}/daemon.log" +export ZE_CONFIG_DIR="${config_dir}" +export ZE_SSH_PASSWORD=secret123 + +stop() { + if [[ -f "${pid_file}" ]]; then + while read -r pid; do + kill "${pid}" 2>/dev/null || true + done < "${pid_file}" + fi + rm -f "${pid_file}" +} + +start() { + rm -rf "${state}" + mkdir -p "${config_dir}" + + printf 'admin\nsecret123\n127.0.0.1\n2222\nze-demo\n' \ + | ze init >"${state}/init.log" 2>&1 + ze config cat ze.conf >"${state}/active.conf" + cat /src/demos/terminal/cli-dashboard/ze.conf >>"${state}/active.conf" + ze config import --name ze.conf "${state}/active.conf" \ + >"${state}/import.log" 2>&1 + + : >"${pid_file}" + for binding in 127.0.0.2:65001 127.0.0.3:65002 127.0.0.4:64496; do + address=${binding%%:*} + asn=${binding##*:} + ze-test peer --mode sink --bind "${address}" --port 1179 --asn "${asn}" \ + >"${state}/peer-${asn}.log" 2>&1 & + echo "$!" >>"${pid_file}" + done + + ze ze.conf >"${log_file}" 2>&1 & + echo "$!" >>"${pid_file}" + + for _ in {1..100}; do + if [[ -f "${log_file}" ]] && grep -q "SSH server listening" "${log_file}"; then + return 0 + fi + sleep 0.1 + done + cat "${log_file}" >&2 + return 1 +} + +case "${1:-}" in + start) start; echo "terminal demo ready" ;; + stop) stop; echo "terminal demo stopped" ;; + *) echo "usage: $0 start|stop" >&2; exit 2 ;; +esac diff --git a/demos/terminal/cli-dashboard/transcript.txt b/demos/terminal/cli-dashboard/transcript.txt new file mode 100644 index 0000000000..7d4231aa35 --- /dev/null +++ b/demos/terminal/cli-dashboard/transcript.txt @@ -0,0 +1,5 @@ +$ ssh ze-demo +ze# exit +ze> monitor bgp + +The dashboard polls three local BGP sessions. Press "s" to sort by the next column, use the arrow keys to select a peer, and press Enter for live session details. Press Escape to return and "q" to leave the dashboard. diff --git a/demos/terminal/cli-dashboard/ze.conf b/demos/terminal/cli-dashboard/ze.conf new file mode 100644 index 0000000000..802eb898e8 --- /dev/null +++ b/demos/terminal/cli-dashboard/ze.conf @@ -0,0 +1,93 @@ +environment { + ssh { + enabled enable + server main { + ip 127.0.0.1 + port 2222 + } + } +} + +bgp { + router-id 192.0.2.1 + session { + asn { + local 65000 + } + } + + peer edge-a { + connection { + remote { + ip 127.0.0.2 + port 1179 + } + local { + ip 127.0.0.1 + } + } + session { + asn { + local 65000 + remote 65001 + } + family { + ipv4/unicast { + prefix { + maximum 1000 + } + } + } + } + } + + peer edge-b { + connection { + remote { + ip 127.0.0.3 + port 1179 + } + local { + ip 127.0.0.1 + } + } + session { + asn { + local 65000 + remote 65002 + } + family { + ipv4/unicast { + prefix { + maximum 1000 + } + } + } + } + } + + peer transit-c { + connection { + remote { + ip 127.0.0.4 + port 1179 + } + local { + ip 127.0.0.1 + } + } + session { + asn { + local 65000 + remote 64496 + } + family { + ipv4/unicast { + prefix { + maximum 1000 + } + } + } + } + } +} diff --git a/demos/terminal/common.tape b/demos/terminal/common.tape new file mode 100644 index 0000000000..d97a389a6d --- /dev/null +++ b/demos/terminal/common.tape @@ -0,0 +1,10 @@ +Set Shell "bash" +Set FontFamily "JetBrains Mono" +Set FontSize 20 +Set Width 1200 +Set Height 720 +Set Padding 24 +Set Framerate 30 +Set TypingSpeed 75ms +Set WaitTimeout 30s +Set Theme { "name": "Ze", "black": "#1d1133", "red": "#ff6f9d", "green": "#3ee6a8", "yellow": "#ffc93c", "blue": "#6fc3ff", "magenta": "#c792ea", "cyan": "#1fc9c0", "white": "#e6d9f2", "brightBlack": "#8f7aa8", "brightRed": "#ff8db4", "brightGreen": "#65f0bc", "brightYellow": "#ffda70", "brightBlue": "#94d3ff", "brightMagenta": "#d8a9f2", "brightCyan": "#68ddd7", "brightWhite": "#fffaff", "background": "#1d1133", "foreground": "#e6d9f2", "selection": "#4b315f", "cursor": "#3ee6a8" } diff --git a/demos/terminal/container-entrypoint.sh b/demos/terminal/container-entrypoint.sh new file mode 100755 index 0000000000..31e9dafbe9 --- /dev/null +++ b/demos/terminal/container-entrypoint.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +set -u + +export HOME=/src/tmp/terminal-demos/home +export XDG_CONFIG_HOME="${HOME}/.config" +export XDG_DATA_HOME="${HOME}/.local/share" +export XDG_RUNTIME_DIR=/src/tmp/terminal-demos/runtime +export PATH="/src/tmp/terminal-demos/bin:${PATH}" +export LANG=C.UTF-8 +export LC_ALL=C.UTF-8 +export TZ=UTC +export TERM=xterm-256color +export PS1='$ ' +mkdir -p "${HOME}" "${XDG_CONFIG_HOME}" "${XDG_DATA_HOME}" "${XDG_RUNTIME_DIR}" +cp /src/demos/terminal/shell.sh "${HOME}/.bashrc" +mkdir -p "${HOME}/.ssh" +printf '%s\n' \ + 'Host ze-demo' \ + ' HostName 127.0.0.1' \ + ' Port 2222' \ + ' User admin' \ + ' StrictHostKeyChecking no' \ + ' UserKnownHostsFile /dev/null' \ + >"${HOME}/.ssh/config" +chmod 600 "${HOME}/.ssh/config" +mkdir -p /root/.ssh +cp "${HOME}/.ssh/config" /root/.ssh/config + +vhs "$@" +status=$? + +if [[ -n "${HOST_UID:-}" && -n "${HOST_GID:-}" ]]; then + chown -R "${HOST_UID}:${HOST_GID}" /src/demos/terminal/artifacts 2>/dev/null || true +fi + +exit "${status}" diff --git a/demos/terminal/install-vhs.sh b/demos/terminal/install-vhs.sh new file mode 100755 index 0000000000..fe84f4db53 --- /dev/null +++ b/demos/terminal/install-vhs.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +set -euo pipefail + +VHS_VERSION="${VHS_VERSION:-v0.11.0}" +TTYD_VERSION="${TTYD_VERSION:-1.7.7}" + +fail() { + printf 'error: %s\n' "$*" >&2 + exit 1 +} + +as_root() { + if [[ "$(id -u)" == 0 ]]; then + "$@" + else + command -v sudo >/dev/null 2>&1 || fail "sudo is required to install system packages" + sudo "$@" + fi +} + +install_macos_dependencies() { + command -v brew >/dev/null 2>&1 || fail "Homebrew is required on macOS" + brew list ffmpeg >/dev/null 2>&1 || brew install ffmpeg + if command -v ttyd >/dev/null 2>&1 && ttyd --version >/dev/null 2>&1; then + return + fi + if brew list ttyd >/dev/null 2>&1; then + brew reinstall ttyd + else + brew install ttyd + fi +} + +install_linux_ttyd() ( + local machine asset checksum temporary + machine="$(uname -m)" + case "$machine" in + x86_64) + asset="ttyd.x86_64" + checksum="8a217c968aba172e0dbf3f34447218dc015bc4d5e59bf51db2f2cd12b7be4f55" + ;; + aarch64|arm64) + asset="ttyd.aarch64" + checksum="b38acadd89d1d396a0f5649aa52c539edbad07f4bc7348b27b4f4b7219dd4165" + ;; + *) + fail "unsupported Linux architecture for ttyd: $machine" + ;; + esac + + temporary="$(mktemp)" + trap 'rm -f "$temporary"' EXIT + curl --fail --location --silent --show-error \ + "https://github.com/tsl0922/ttyd/releases/download/${TTYD_VERSION}/${asset}" \ + --output "$temporary" + printf '%s %s\n' "$checksum" "$temporary" | sha256sum --check --status + as_root install -m 0755 "$temporary" /usr/local/bin/ttyd +) + +install_ubuntu_dependencies() { + [[ -r /etc/os-release ]] || fail "cannot identify this Linux distribution" + # shellcheck disable=SC1091 + source /etc/os-release + case "${ID:-}" in + ubuntu|debian) ;; + *) fail "native installation supports Ubuntu and Debian, found ${ID:-unknown}" ;; + esac + + as_root apt-get update + as_root apt-get install --yes ca-certificates curl ffmpeg + if ! command -v ttyd >/dev/null 2>&1 || ! ttyd --version >/dev/null 2>&1; then + install_linux_ttyd + fi +} + +case "$(uname -s)" in + Darwin) + install_macos_dependencies + default_bin_dir="$(brew --prefix)/bin" + ;; + Linux) + install_ubuntu_dependencies + default_bin_dir="${HOME}/.local/bin" + ;; + *) + fail "native installation supports macOS and Ubuntu only" + ;; +esac + +command -v go >/dev/null 2>&1 || fail "Go is required to install VHS" +install_dir="${VHS_BIN_DIR:-$default_bin_dir}" +mkdir -p "$install_dir" +GOBIN="$install_dir" go install "github.com/charmbracelet/vhs@${VHS_VERSION}" + +"$install_dir/vhs" --version +ffmpeg -version | sed -n '1p' +ttyd --version +printf 'VHS and its native dependencies are installed in %s\n' "$install_dir" diff --git a/demos/terminal/launcher/demo.tape b/demos/terminal/launcher/demo.tape new file mode 100644 index 0000000000..30443b8a38 --- /dev/null +++ b/demos/terminal/launcher/demo.tape @@ -0,0 +1,56 @@ +Output artifacts/launcher.webm +Source common.tape + +Hide +Type "bash /src/demos/terminal/cards.sh launcher intro" +Enter +Show +Sleep 10s + +Hide +Type "clear" +Enter +Show +Type "ze" +Enter +Wait+Screen /Operations/ +Sleep 4s + +Type "show" +Wait+Screen /filter: show/ +Sleep 4s +Enter +Wait+Screen /ze > show/ +Sleep 3s + +Type "traceroute" +Wait+Screen /filter: traceroute/ +Sleep 6s +Screenshot artifacts/launcher.png + +Escape +Sleep 3s +Left +Sleep 3s +Type "doctor" +Wait+Screen /filter: doctor/ +Sleep 6s +Escape +Sleep 2s +Escape +Sleep 2s +Escape +Sleep 2s +Wait+Screen /\$ / +Hide +Sleep 2s +Type " true" +Enter +Sleep 1s +Type "clear" +Enter + +Type "bash /src/demos/terminal/cards.sh launcher recap" +Enter +Show +Sleep 8s diff --git a/demos/terminal/launcher/transcript.txt b/demos/terminal/launcher/transcript.txt new file mode 100644 index 0000000000..9f930f9afa --- /dev/null +++ b/demos/terminal/launcher/transcript.txt @@ -0,0 +1,6 @@ +$ ze + +Type "show" to filter the command launcher, then press Enter to open the show command tree. +Type "traceroute" to find the path diagnostic command. +Press Escape and Left to return, then type "doctor" to find the readiness checker. +Press Escape to move back through the menu and return to the shell. diff --git a/demos/terminal/manifest.json b/demos/terminal/manifest.json new file mode 100644 index 0000000000..b749a23ab6 --- /dev/null +++ b/demos/terminal/manifest.json @@ -0,0 +1,63 @@ +{ + "schema": 1, + "renderer": { + "name": "vhs", + "version": "0.11.0", + "image": "ze-terminal-demos:vhs-0.11.0", + "platform": "linux/native" + }, + "gallery_page": "guide/terminal-demonstrations.md", + "demos": [ + { + "id": "cli-dashboard", + "title": "Operate BGP from the live dashboard", + "description": "Connect to Ze over SSH, open the live BGP dashboard, sort peers, and inspect one session.", + "page": "features/cli-commands.md", + "anchor": "live-peer-dashboard", + "platform": "portable", + "duration": "50 seconds", + "tape": "cli-dashboard/demo.tape" + }, + { + "id": "zefs-config", + "title": "Create ZeFS and commit over SSH", + "description": "Create the ZeFS database, edit the active configuration through Ze's SSH management plane, and verify the committed setting.", + "page": "guide/quickstart.md", + "anchor": "initialize", + "platform": "portable", + "duration": "85 seconds", + "tape": "zefs-config/demo.tape" + }, + { + "id": "rbac", + "title": "Prove read-only RBAC enforcement", + "description": "Run an allowed NOC command, then show Ze explicitly refuse a known state-changing command.", + "page": "guide/operator-access-rbac.md", + "anchor": "test-each-account", + "platform": "portable", + "duration": "45 seconds", + "tape": "rbac/demo.tape" + }, + { + "id": "traceroute", + "title": "Trace a live path without external services", + "description": "Run Ze's live traceroute through a deterministic Linux network-namespace lab.", + "page": "guide/production-diagnostics.md", + "anchor": "continuous-monitoring", + "platform": "linux", + "privileged": true, + "duration": "55 seconds", + "tape": "traceroute/demo.tape" + }, + { + "id": "launcher", + "title": "Discover Ze commands interactively", + "description": "Use type-ahead filtering and drill-down navigation in Ze's interactive command launcher.", + "page": "guide/command-reference.md", + "anchor": "ze", + "platform": "portable", + "duration": "55 seconds", + "tape": "launcher/demo.tape" + } + ] +} diff --git a/demos/terminal/rbac/demo.tape b/demos/terminal/rbac/demo.tape new file mode 100644 index 0000000000..abce94c7a1 --- /dev/null +++ b/demos/terminal/rbac/demo.tape @@ -0,0 +1,50 @@ +Output artifacts/rbac.webm +Source common.tape +Set WaitTimeout 60s + +Hide +Type "export ZE_CONFIG_DIR=/src/tmp/terminal-demos/state/rbac/config NOC_PASSWORD=noc-secret ADMIN_PASSWORD=admin-secret" +Enter +Type "/src/demos/terminal/rbac/run.sh start" +Enter +Wait+Screen /terminal demo ready/ +Type "bash /src/demos/terminal/cards.sh rbac intro" +Enter +Show +Sleep 10s + +Hide +Type "export ZE_SSH_PASSWORD=noc-secret" +Enter +Type "clear" +Enter +Show +Type "ze cli --user noc -c 'show version'" +Enter +Sleep 6s + +Hide +Type "bash /src/demos/terminal/cards.sh rbac deny" +Enter +Show +Sleep 8s + +Hide +Type "clear" +Enter +Show +Type "ze cli --user noc -c 'clear interface counters'" +Enter +Wait+Screen /restricted by access control/ +Sleep 6s +Screenshot artifacts/rbac.png +Sleep 2s + +Hide +Type "/src/demos/terminal/rbac/run.sh stop" +Enter +Wait+Screen /terminal demo stopped/ +Type "bash /src/demos/terminal/cards.sh rbac recap" +Enter +Show +Sleep 8s diff --git a/demos/terminal/rbac/rbac.conf b/demos/terminal/rbac/rbac.conf new file mode 100644 index 0000000000..86a8d8715f --- /dev/null +++ b/demos/terminal/rbac/rbac.conf @@ -0,0 +1,34 @@ +environment { + ssh { + enabled enable + server main { + ip 127.0.0.1 + port 2222 + } + } +} + +system { + authorization { + profile read-only { + run { + default-action allow + entry 10 { + action deny + match debug + } + entry 20 { + action deny + match clear + } + } + edit default-action deny + } + } + authentication { + user noc { + password $2a$10$EHVw7e4FvIc2NlB.sTHrGuRGv8PHte7aHyGQkiuvHWdC6NxQx/TIO + profile read-only + } + } +} diff --git a/demos/terminal/rbac/run.sh b/demos/terminal/rbac/run.sh new file mode 100755 index 0000000000..281983141a --- /dev/null +++ b/demos/terminal/rbac/run.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +set -euo pipefail + +state=/src/tmp/terminal-demos/state/rbac +config_dir="${state}/config" +pid_file="${state}/pids" +log_file="${state}/daemon.log" +export ZE_CONFIG_DIR="${config_dir}" + +stop() { + if [[ -f "${pid_file}" ]]; then + while read -r pid; do + kill "${pid}" 2>/dev/null || true + done < "${pid_file}" + fi + rm -f "${pid_file}" +} + +start() { + rm -rf "${state}" + mkdir -p "${config_dir}" + + printf 'admin\nadmin-secret\n127.0.0.1\n2222\nze-demo\n' \ + | ze init >"${state}/init.log" 2>&1 + ze config cat ze.conf >"${state}/active.conf" + cat /src/demos/terminal/rbac/rbac.conf >>"${state}/active.conf" + ze config import --name ze.conf "${state}/active.conf" \ + >"${state}/import.log" 2>&1 + + : >"${pid_file}" + ze ze.conf >"${log_file}" 2>&1 & + echo "$!" >>"${pid_file}" + for _ in {1..100}; do + if [[ -f "${log_file}" ]] && grep -q "SSH server listening" "${log_file}"; then + return 0 + fi + sleep 0.1 + done + cat "${log_file}" >&2 + return 1 +} + +case "${1:-}" in + start) start; echo "terminal demo ready" ;; + stop) stop; echo "terminal demo stopped" ;; + *) echo "usage: $0 start|stop" >&2; exit 2 ;; +esac diff --git a/demos/terminal/rbac/transcript.txt b/demos/terminal/rbac/transcript.txt new file mode 100644 index 0000000000..e4a019a1de --- /dev/null +++ b/demos/terminal/rbac/transcript.txt @@ -0,0 +1,6 @@ +$ ze cli --user noc -c 'show version' +version: ze 26.07.15 +$ ze cli --user noc -c 'clear interface counters' +error: command restricted by access control + +The read-only NOC profile allows operational show commands and denies every command matching "clear". The daemon resolves the command, finds the profile forbids it, and refuses before running it. The password is supplied outside the recording. diff --git a/demos/terminal/render.py b/demos/terminal/render.py new file mode 100755 index 0000000000..75e8f0d2e9 --- /dev/null +++ b/demos/terminal/render.py @@ -0,0 +1,324 @@ +#!/usr/bin/env python3 +"""Render and verify Ze terminal demonstrations from the checked-in VHS tapes.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import pathlib +import re +import shutil +import subprocess +import sys +from typing import Any + +ROOT = pathlib.Path(__file__).resolve().parents[2] +DEMO_ROOT = ROOT / "demos" / "terminal" +DEFAULT_ARTIFACT_ROOT = ROOT.parent / "gh-pages" / "assets" / "demos" +ARTIFACT_ROOT = pathlib.Path( + os.environ.get("ZE_TERMINAL_DEMO_OUTPUT", DEFAULT_ARTIFACT_ROOT) +).resolve() +MANIFEST_PATH = DEMO_ROOT / "manifest.json" +ARTIFACT_MANIFEST_PATH = ARTIFACT_ROOT / "manifest.json" +BINARY_PATH = ROOT / "tmp" / "terminal-demos" / "bin" / "ze" +SHARED_SOURCE_PATHS = ( + DEMO_ROOT / "common.tape", + DEMO_ROOT / "cards.sh", + DEMO_ROOT / "Dockerfile", + DEMO_ROOT / "container-entrypoint.sh", +) +ID_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$") + + +def sha256(path: pathlib.Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def load_json(path: pathlib.Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as stream: + value = json.load(stream) + if not isinstance(value, dict): + raise ValueError(f"{path}: expected a JSON object") + return value + + +def demo_by_id(manifest: dict[str, Any]) -> dict[str, dict[str, Any]]: + demos = manifest.get("demos") + if not isinstance(demos, list) or not demos: + raise ValueError("manifest.json: demos must be a non-empty list") + + indexed: dict[str, dict[str, Any]] = {} + for demo in demos: + if not isinstance(demo, dict): + raise ValueError("manifest.json: every demo must be an object") + demo_id = demo.get("id") + if not isinstance(demo_id, str) or not ID_RE.fullmatch(demo_id): + raise ValueError(f"manifest.json: invalid demo id {demo_id!r}") + if demo_id in indexed: + raise ValueError(f"manifest.json: duplicate demo id {demo_id}") + indexed[demo_id] = demo + return indexed + + +def validate_contract(manifest: dict[str, Any]) -> dict[str, dict[str, Any]]: + if manifest.get("schema") != 1: + raise ValueError("manifest.json: unsupported schema") + + renderer = manifest.get("renderer") + if not isinstance(renderer, dict): + raise ValueError("manifest.json: renderer must be an object") + for field in ("name", "version", "image", "platform"): + if not isinstance(renderer.get(field), str) or not renderer[field]: + raise ValueError(f"manifest.json: renderer.{field} is required") + + gallery_page = manifest.get("gallery_page") + if not isinstance(gallery_page, str) or not gallery_page: + raise ValueError("manifest.json: gallery_page is required") + if not (ROOT / "docs" / gallery_page).is_file(): + raise ValueError(f"manifest.json: gallery page does not exist: {gallery_page}") + indexed = demo_by_id(manifest) + required = ( + "title", + "description", + "page", + "anchor", + "platform", + "duration", + "tape", + ) + for demo_id, demo in indexed.items(): + for field in required: + if not isinstance(demo.get(field), str) or not demo[field]: + raise ValueError(f"manifest.json: {demo_id}.{field} is required") + if "privileged" in demo and not isinstance(demo["privileged"], bool): + raise ValueError(f"manifest.json: {demo_id}.privileged must be a boolean") + + page = ROOT / "docs" / demo["page"] + tape = DEMO_ROOT / demo["tape"] + transcript = tape.parent / "transcript.txt" + if not page.is_file(): + raise ValueError( + f"manifest.json: page does not exist: {page.relative_to(ROOT)}" + ) + if not tape.is_file(): + raise ValueError( + f"manifest.json: tape does not exist: {tape.relative_to(ROOT)}" + ) + if not transcript.is_file(): + raise ValueError( + f"manifest.json: transcript does not exist: {transcript.relative_to(ROOT)}" + ) + return indexed + + +def source_digest(demo: dict[str, Any]) -> str: + tape = DEMO_ROOT / demo["tape"] + files = [MANIFEST_PATH, *SHARED_SOURCE_PATHS] + files.extend( + path + for path in tape.parent.rglob("*") + if path.is_file() and not path.name.startswith(".") + ) + digest = hashlib.sha256() + for path in sorted(files): + digest.update(path.relative_to(ROOT).as_posix().encode()) + digest.update(b"\0") + digest.update(path.read_bytes()) + digest.update(b"\0") + return digest.hexdigest() + + +def run_demo( + manifest: dict[str, Any], demo: dict[str, Any], release: str +) -> dict[str, Any]: + demo_id = demo["id"] + renderer = manifest["renderer"] + tape = pathlib.PurePosixPath("/src/demos/terminal") / demo["tape"] + ARTIFACT_ROOT.mkdir(parents=True, exist_ok=True) + + expected = { + "video": ARTIFACT_ROOT / f"{demo_id}.webm", + "poster": ARTIFACT_ROOT / f"{demo_id}.png", + "transcript": ARTIFACT_ROOT / f"{demo_id}.txt", + } + for name, path in expected.items(): + if name != "transcript": + path.unlink(missing_ok=True) + + uid = str(os.getuid()) if hasattr(os, "getuid") else "0" + gid = str(os.getgid()) if hasattr(os, "getgid") else "0" + command = [ + "docker", + "run", + "--rm", + "--network", + "none", + "--cap-add", + "NET_ADMIN", + "--cap-add", + "NET_RAW", + "--cap-add", + "SYS_ADMIN", + "--security-opt", + "seccomp=unconfined", + "--env", + f"HOST_UID={uid}", + "--env", + f"HOST_GID={gid}", + "--env", + f"ZE_DEMO_RELEASE={release}", + "--volume", + f"{ROOT}:/src", + "--volume", + f"{ARTIFACT_ROOT}:/src/demos/terminal/artifacts", + "--workdir", + "/src/demos/terminal", + renderer["image"], + str(tape), + ] + if demo.get("privileged", False): + command[3:3] = ["--privileged"] + if not renderer["platform"].endswith("/native"): + command[3:3] = ["--platform", renderer["platform"]] + print(f"rendering {demo_id}...") + subprocess.run(command, cwd=ROOT, check=True) + + transcript_source = DEMO_ROOT / demo["tape"] + shutil.copyfile(transcript_source.parent / "transcript.txt", expected["transcript"]) + + assets: dict[str, dict[str, Any]] = {} + for name, path in expected.items(): + if not path.is_file() or path.stat().st_size == 0: + raise RuntimeError(f"{demo_id}: missing generated {name}: {path}") + assets[name] = { + "path": path.relative_to(ARTIFACT_ROOT).as_posix(), + "bytes": path.stat().st_size, + "sha256": sha256(path), + } + + return { + "release": release, + "binary_sha256": sha256(BINARY_PATH), + "source_sha256": source_digest(demo), + "assets": assets, + } + + +def load_artifact_manifest(manifest: dict[str, Any]) -> dict[str, Any]: + if ARTIFACT_MANIFEST_PATH.is_file(): + generated = load_json(ARTIFACT_MANIFEST_PATH) + if generated.get("schema") == 1 and isinstance(generated.get("demos"), dict): + return generated + return { + "schema": 1, + "renderer": manifest["renderer"], + "demos": {}, + } + + +def write_artifact_manifest(generated: dict[str, Any]) -> None: + ARTIFACT_ROOT.mkdir(parents=True, exist_ok=True) + text = json.dumps(generated, indent=2, sort_keys=True) + "\n" + ARTIFACT_MANIFEST_PATH.write_text(text, encoding="utf-8") + + +def verify_assets( + manifest: dict[str, Any], + indexed: dict[str, dict[str, Any]], + selected: list[str], + release: str | None, +) -> None: + generated = load_json(ARTIFACT_MANIFEST_PATH) + if generated.get("schema") != 1: + raise ValueError("generated manifest: unsupported schema") + if generated.get("renderer") != manifest.get("renderer"): + raise ValueError("generated manifest: renderer contract is stale") + generated_demos = generated.get("demos") + if not isinstance(generated_demos, dict): + raise ValueError("generated manifest: demos must be an object") + + for demo_id in selected: + entry = generated_demos.get(demo_id) + if not isinstance(entry, dict): + raise ValueError(f"generated manifest: missing {demo_id}") + if release is not None and entry.get("release") != release: + raise ValueError( + f"{demo_id}: rendered for {entry.get('release')!r}, expected {release!r}" + ) + if entry.get("source_sha256") != source_digest(indexed[demo_id]): + raise ValueError(f"{demo_id}: source changed since the last render") + assets = entry.get("assets") + if not isinstance(assets, dict): + raise ValueError(f"{demo_id}: assets are missing") + for name in ("video", "poster", "transcript"): + asset = assets.get(name) + if not isinstance(asset, dict) or not isinstance(asset.get("path"), str): + raise ValueError(f"{demo_id}: missing {name} metadata") + path = ARTIFACT_ROOT / asset["path"] + if not path.is_file(): + raise ValueError(f"{demo_id}: missing generated asset: {path}") + if path.stat().st_size != asset.get("bytes") or sha256(path) != asset.get( + "sha256" + ): + raise ValueError(f"{demo_id}: {name} digest mismatch") + print("terminal demo artifacts verified: " + ", ".join(selected)) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + selection = parser.add_mutually_exclusive_group(required=True) + selection.add_argument( + "--all", action="store_true", help="render or verify every demo" + ) + selection.add_argument( + "--demo", action="append", help="demo id to render or verify" + ) + parser.add_argument( + "--release", help="Ze release identity recorded in artifact metadata" + ) + parser.add_argument( + "--check", action="store_true", help="verify existing artifacts only" + ) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + manifest = load_json(MANIFEST_PATH) + indexed = validate_contract(manifest) + selected = list(indexed) if args.all else list(dict.fromkeys(args.demo)) + unknown = [demo_id for demo_id in selected if demo_id not in indexed] + if unknown: + raise ValueError("unknown demo id(s): " + ", ".join(unknown)) + + if args.check: + verify_assets(manifest, indexed, selected, args.release) + return 0 + + if not args.release: + raise ValueError("--release is required when rendering") + if not BINARY_PATH.is_file(): + raise ValueError(f"missing demo binary: {BINARY_PATH.relative_to(ROOT)}") + + generated = load_artifact_manifest(manifest) + generated["renderer"] = manifest["renderer"] + entries = generated["demos"] + for demo_id in selected: + entries[demo_id] = run_demo(manifest, indexed[demo_id], args.release) + write_artifact_manifest(generated) + verify_assets(manifest, indexed, selected, args.release) + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except (OSError, subprocess.CalledProcessError, RuntimeError, ValueError) as exc: + print(f"error: {exc}", file=sys.stderr) + raise SystemExit(1) diff --git a/demos/terminal/shell.sh b/demos/terminal/shell.sh new file mode 100755 index 0000000000..92b1aa4930 --- /dev/null +++ b/demos/terminal/shell.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash + +export PATH="/src/tmp/terminal-demos/bin:${PATH}" +export HOME="/src/tmp/terminal-demos/home" +export XDG_CONFIG_HOME="${HOME}/.config" +export XDG_DATA_HOME="${HOME}/.local/share" +export XDG_RUNTIME_DIR="/src/tmp/terminal-demos/runtime" +export LANG=C.UTF-8 +export LC_ALL=C.UTF-8 +export TZ=UTC +export TERM=xterm-256color +export PS1='$ ' + +mkdir -p "${HOME}" "${XDG_CONFIG_HOME}" "${XDG_DATA_HOME}" "${XDG_RUNTIME_DIR}" +chmod 700 "${XDG_RUNTIME_DIR}" +cd /src diff --git a/demos/terminal/traceroute/demo.tape b/demos/terminal/traceroute/demo.tape new file mode 100644 index 0000000000..5b97993b4c --- /dev/null +++ b/demos/terminal/traceroute/demo.tape @@ -0,0 +1,47 @@ +Output artifacts/traceroute.webm +Source common.tape + +Hide +Type "export ZE_CONFIG_DIR=/src/tmp/terminal-demos/state/traceroute/config ZE_SSH_PASSWORD=secret123 SSHPASS=secret123" +Enter +Type "/src/demos/terminal/traceroute/run.sh start" +Enter +Wait+Screen /terminal demo ready/ +Type "bash /src/demos/terminal/cards.sh traceroute intro" +Enter +Show +Sleep 10s + +Hide +Type "clear" +Enter +Show +Type "sshpass -e ssh ze-demo" +Enter +Wait+Screen /ze#/ +Sleep 4s +Type "run show traceroute 192.0.2.53" +Enter +Wait+Screen /192.0.2.53/ +Sleep 6s +Type "run monitor traceroute 192.0.2.53" +Enter +Wait+Screen /Loss/ +Sleep 10s +Screenshot artifacts/traceroute.png +Sleep 2s +Type "q" +Sleep 3s +Escape +Sleep 2s +Escape +Sleep 3s + +Hide +Type "/src/demos/terminal/traceroute/run.sh stop" +Enter +Wait+Screen /terminal demo stopped/ +Type "bash /src/demos/terminal/cards.sh traceroute recap" +Enter +Show +Sleep 8s diff --git a/demos/terminal/traceroute/run.sh b/demos/terminal/traceroute/run.sh new file mode 100755 index 0000000000..8a536ee03c --- /dev/null +++ b/demos/terminal/traceroute/run.sh @@ -0,0 +1,92 @@ +#!/usr/bin/env bash +set -euo pipefail + +state=/src/tmp/terminal-demos/state/traceroute +config_dir="${state}/config" +pid_file="${state}/pids" +log_file="${state}/daemon.log" +export ZE_CONFIG_DIR="${config_dir}" +export ZE_SSH_PASSWORD=secret123 + +cleanup_network() { + ip route del 192.0.2.53/32 via 198.51.100.2 2>/dev/null || true + ip link del ze-edge 2>/dev/null || true + ip netns del edge 2>/dev/null || true + ip netns del core 2>/dev/null || true +} + +stop() { + if [[ -f "${pid_file}" ]]; then + while read -r pid; do + kill "${pid}" 2>/dev/null || true + done < "${pid_file}" + fi + rm -f "${pid_file}" + cleanup_network +} + +start_network() { + cleanup_network + ip netns add edge + ip netns add core + + ip link add ze-edge type veth peer name edge-ze + ip link set edge-ze netns edge + ip address add 198.51.100.1/30 dev ze-edge + ip link set ze-edge up + ip -n edge address add 198.51.100.2/30 dev edge-ze + ip -n edge link set edge-ze up + ip -n edge link set lo up + + ip link add edge-core type veth peer name core-edge + ip link set edge-core netns edge + ip link set core-edge netns core + ip -n edge address add 203.0.113.1/30 dev edge-core + ip -n edge link set edge-core up + ip -n core address add 203.0.113.2/30 dev core-edge + ip -n core link set core-edge up + ip -n core address add 192.0.2.53/32 dev lo + ip -n core link set lo up + + ip netns exec edge sysctl -q -w net.ipv4.ip_forward=1 + ip netns exec core sysctl -q -w net.ipv4.ip_forward=1 + ip route add 192.0.2.53/32 via 198.51.100.2 + ip -n edge route add 192.0.2.53/32 via 203.0.113.2 + ip -n core route add 198.51.100.0/30 via 203.0.113.1 + + if ! grep -q 'dns.demo' /etc/hosts; then + printf '198.51.100.2 edge-gw.demo\n203.0.113.2 core-gw.demo\n192.0.2.53 dns.demo\n' \ + >>/etc/hosts + fi +} + +start() { + cleanup_network + rm -rf "${state}" + mkdir -p "${config_dir}" + start_network + + printf 'admin\nsecret123\n127.0.0.1\n2222\nze-demo\n' \ + | ze init >"${state}/init.log" 2>&1 + ze config cat ze.conf >"${state}/active.conf" + cat /src/demos/terminal/traceroute/ze.conf >>"${state}/active.conf" + ze config import --name ze.conf "${state}/active.conf" \ + >"${state}/import.log" 2>&1 + : >"${pid_file}" + ze ze.conf >"${log_file}" 2>&1 & + echo "$!" >>"${pid_file}" + for _ in {1..100}; do + if [[ -f "${log_file}" ]] && grep -q "SSH server listening" "${log_file}"; then + return 0 + fi + sleep 0.1 + done + cat "${log_file}" >&2 + return 1 +} + +case "${1:-}" in + start) start; echo "terminal demo ready" ;; + stop) stop; echo "terminal demo stopped" ;; + *) echo "usage: $0 start|stop" >&2; exit 2 ;; +esac diff --git a/demos/terminal/traceroute/transcript.txt b/demos/terminal/traceroute/transcript.txt new file mode 100644 index 0000000000..a9b269eaa8 --- /dev/null +++ b/demos/terminal/traceroute/transcript.txt @@ -0,0 +1,5 @@ +$ ssh ze-demo +ze# run show traceroute 192.0.2.53 +ze# run monitor traceroute 192.0.2.53 + +The destination and router live in an isolated Linux network-namespace lab. Ze sends real ICMP probes, then shows the same path as a one-shot trace and as a continuously refreshed loss and latency table. No public DNS or Internet route is used. diff --git a/demos/terminal/traceroute/ze.conf b/demos/terminal/traceroute/ze.conf new file mode 100644 index 0000000000..350e35d3e7 --- /dev/null +++ b/demos/terminal/traceroute/ze.conf @@ -0,0 +1,9 @@ +environment { + ssh { + enabled enable + server main { + ip 127.0.0.1 + port 2222 + } + } +} diff --git a/demos/terminal/zefs-config/demo.tape b/demos/terminal/zefs-config/demo.tape new file mode 100644 index 0000000000..9cfbe6cc80 --- /dev/null +++ b/demos/terminal/zefs-config/demo.tape @@ -0,0 +1,76 @@ +Output artifacts/zefs-config.webm +Source common.tape + +Hide +Type "export ZE_CONFIG_DIR=/src/tmp/terminal-demos/state/zefs-config/config ZE_INIT_INPUT=/src/tmp/terminal-demos/state/zefs-config/init.input ZE_SSH_PASSWORD=secret123 SSHPASS=secret123" +Enter +Type "/src/demos/terminal/zefs-config/run.sh prepare" +Enter +Wait+Screen /terminal demo prepared/ +Type "bash /src/demos/terminal/cards.sh zefs-config intro" +Enter +Show +Sleep 10s + +Hide +Type "clear" +Enter +Show +Type "ze init < $ZE_INIT_INPUT" +Enter +Sleep 4s +Type "ze config ls" +Enter +Wait+Screen /ze.conf/ +Sleep 5s +Type "ze data check" +Enter +Sleep 5s + +Hide +Type "/src/demos/terminal/zefs-config/run.sh start" +Enter +Wait+Screen /terminal demo ready/ +Type "bash /src/demos/terminal/cards.sh zefs-config ssh" +Enter +Show +Sleep 8s + +Hide +Type "clear" +Enter +Show +Type "sshpass -e ssh ze-demo" +Enter +Wait+Screen /ze#/ +Sleep 4s +Type "set environment cli format default table" +Enter +Sleep 4s +Type "show | compare" +Enter +Sleep 5s +Type "commit" +Enter +Wait+Screen /Session committed/ +Sleep 6s +Type "run show bgp summary" +Enter +Sleep 6s +Screenshot artifacts/zefs-config.png +Sleep 2s +Escape +Sleep 2s +Escape +Sleep 2s +Escape +Sleep 3s + +Hide +Type "/src/demos/terminal/zefs-config/run.sh stop" +Enter +Wait+Screen /terminal demo stopped/ +Type "bash /src/demos/terminal/cards.sh zefs-config recap" +Enter +Show +Sleep 8s diff --git a/demos/terminal/zefs-config/run.sh b/demos/terminal/zefs-config/run.sh new file mode 100755 index 0000000000..8d8f55f5ec --- /dev/null +++ b/demos/terminal/zefs-config/run.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +set -euo pipefail + +state=/src/tmp/terminal-demos/state/zefs-config +config_dir="${state}/config" +pid_file="${state}/pids" +log_file="${state}/daemon.log" +export ZE_CONFIG_DIR="${config_dir}" +export ZE_SSH_PASSWORD=secret123 + +stop() { + if [[ -f "${pid_file}" ]]; then + while read -r pid; do + kill "${pid}" 2>/dev/null || true + done < "${pid_file}" + fi + rm -f "${pid_file}" +} + +prepare() { + stop + rm -rf "${state}" + mkdir -p "${config_dir}" + printf 'admin\nsecret123\n127.0.0.1\n2222\nze-demo\n' >"${state}/init.input" + chmod 600 "${state}/init.input" +} + +start() { + ze config cat ze.conf >"${state}/active.conf" + cat /src/demos/terminal/zefs-config/ze.conf >>"${state}/active.conf" + ze config import --name ze.conf "${state}/active.conf" \ + >"${state}/import.log" 2>&1 + + : >"${pid_file}" + ze-test peer --mode sink --bind 127.0.0.2 --port 1179 --asn 65001 \ + >"${state}/peer.log" 2>&1 & + echo "$!" >>"${pid_file}" + ze ze.conf >"${log_file}" 2>&1 & + echo "$!" >>"${pid_file}" + + for _ in {1..100}; do + if [[ -f "${log_file}" ]] && grep -q "SSH server listening" "${log_file}"; then + return 0 + fi + sleep 0.1 + done + cat "${log_file}" >&2 + return 1 +} + +case "${1:-}" in + prepare) prepare; echo "terminal demo prepared" ;; + start) start; echo "terminal demo ready" ;; + stop) stop; echo "terminal demo stopped" ;; + *) echo "usage: $0 prepare|start|stop" >&2; exit 2 ;; +esac diff --git a/demos/terminal/zefs-config/transcript.txt b/demos/terminal/zefs-config/transcript.txt new file mode 100644 index 0000000000..0ca0c09f78 --- /dev/null +++ b/demos/terminal/zefs-config/transcript.txt @@ -0,0 +1,13 @@ +$ ze init < "$ZE_INIT_INPUT" +$ ze config ls +ze.conf +$ ze data check + +$ ssh ze-demo +ze# set environment cli format default table +ze# show | compare +ze# commit +Session committed +ze# run show bgp summary + +`ze init` creates `database.zefs`. The SSH editor then commits the table-format setting back to ZeFS, not to a second flat file. The operational command uses the new default immediately. diff --git a/demos/terminal/zefs-config/ze.conf b/demos/terminal/zefs-config/ze.conf new file mode 100644 index 0000000000..af29102a35 --- /dev/null +++ b/demos/terminal/zefs-config/ze.conf @@ -0,0 +1,42 @@ +environment { + ssh { + enabled enable + server main { + ip 127.0.0.1 + port 2222 + } + } +} + +bgp { + router-id 192.0.2.1 + session { + asn { + local 65000 + } + } + peer edge-a { + connection { + remote { + ip 127.0.0.2 + port 1179 + } + local { + ip 127.0.0.1 + } + } + session { + asn { + local 65000 + remote 65001 + } + family { + ipv4/unicast { + prefix { + maximum 1000 + } + } + } + } + } +} diff --git a/docs/guide/production-diagnostics.md b/docs/guide/production-diagnostics.md index cf55054af0..a23d28fc51 100644 --- a/docs/guide/production-diagnostics.md +++ b/docs/guide/production-diagnostics.md @@ -361,6 +361,8 @@ appends one line per round and annotates hops with ASN names, useful for identifying which network a path change occurs in. `| log | resolve` adds reverse DNS hostnames instead. + + ## Profiling Workflow ### CPU Profile diff --git a/docs/guide/quickstart.md b/docs/guide/quickstart.md index 9f7b7df6ac..c3f64467e0 100644 --- a/docs/guide/quickstart.md +++ b/docs/guide/quickstart.md @@ -48,6 +48,8 @@ bin/ze init --force # prompts for confirmation, then backs up and rei ``` + + ## Minimal Config Save as `example.conf`: diff --git a/docs/guide/terminal-demonstrations.md b/docs/guide/terminal-demonstrations.md new file mode 100644 index 0000000000..a92da4ff73 --- /dev/null +++ b/docs/guide/terminal-demonstrations.md @@ -0,0 +1,31 @@ +--- +title: Terminal Demonstrations +description: Watch Ze command, configuration, monitoring, and access-control workflows recorded from reproducible VHS tapes. +category: observe +journey: Evaluate +--- +# Terminal Demonstrations + +These recordings run real Ze commands against isolated local fixtures. The checked-in VHS tapes define every keystroke, pause, and terminal size, so a release can regenerate the videos when Ze changes. The recordings use no public service. + +Each demo also appears beside the documentation for the feature it exercises. The transcript below each player provides the same command sequence without requiring video playback. + +## Interactive command launcher + + + +## Live BGP dashboard + + + +## ZeFS and SSH configuration + + + +## Read-only operator access + + + +## Traceroute in an isolated Linux lab + + diff --git a/mk/terminal-demo.mk b/mk/terminal-demo.mk index ce5a02496b..eaeda4983b 100644 --- a/mk/terminal-demo.mk +++ b/mk/terminal-demo.mk @@ -9,13 +9,18 @@ TERMINAL_DEMO_GOARCH ?= $(shell $(GO) env GOARCH) TERMINAL_DEMO_IMAGE := ze-terminal-demos:vhs-0.11.0 TERMINAL_DEMO_RELEASE ?= $(ZE_VERSION) TERMINAL_DEMO_BIN_DIR := $(CURDIR)/tmp/terminal-demos/bin +TERMINAL_DEMO_OUTPUT ?= $(CURDIR)/../gh-pages/assets/demos TERMINAL_DEMO_TAGS := ze_core ze_distro $(ZE_FEATURES) $(ZE_TAGS) .PHONY: ze-terminal-demo ze-terminal-demos ze-terminal-demos-check .PHONY: ze-terminal-demo-image ze-terminal-demo-binaries .PHONY: ze-terminal-demos-release ze-terminal-demos-release-check +.PHONY: ze-terminal-demo-tools .PHONY: ze-release-assets ze-release-assets-check +ze-terminal-demo-tools: + @demos/terminal/install-vhs.sh + ze-terminal-demo-image: @command -v docker >/dev/null || { echo "error: docker is required to render terminal demos"; exit 1; } docker build \ @@ -34,20 +39,20 @@ ze-terminal-demo-binaries: ze-terminal-demo: ze-terminal-demo-image ze-terminal-demo-binaries @test -n "$(DEMO)" || { echo "error: pass DEMO= (see demos/terminal/manifest.json)"; exit 1; } - @python3 demos/terminal/render.py --demo "$(DEMO)" --release "$(TERMINAL_DEMO_RELEASE)" + @ZE_TERMINAL_DEMO_OUTPUT="$(TERMINAL_DEMO_OUTPUT)" python3 demos/terminal/render.py --demo "$(DEMO)" --release "$(TERMINAL_DEMO_RELEASE)" ze-terminal-demos: ze-terminal-demo-image ze-terminal-demo-binaries - @python3 demos/terminal/render.py --all --release "$(TERMINAL_DEMO_RELEASE)" + @ZE_TERMINAL_DEMO_OUTPUT="$(TERMINAL_DEMO_OUTPUT)" python3 demos/terminal/render.py --all --release "$(TERMINAL_DEMO_RELEASE)" ze-terminal-demos-check: - @python3 demos/terminal/render.py --all --check + @ZE_TERMINAL_DEMO_OUTPUT="$(TERMINAL_DEMO_OUTPUT)" python3 demos/terminal/render.py --all --check # Release preparation calls this aggregate target before tagging. It rebuilds # the released binaries and every website recording from the checked-in tapes. ze-terminal-demos-release: ze-terminal-demos ze-terminal-demos-release-check: - @python3 demos/terminal/render.py --all --release "$(TERMINAL_DEMO_RELEASE)" --check + @ZE_TERMINAL_DEMO_OUTPUT="$(TERMINAL_DEMO_OUTPUT)" python3 demos/terminal/render.py --all --release "$(TERMINAL_DEMO_RELEASE)" --check ze-release-assets: ze-terminal-demos-release From f47bd7093125538dc05645917168d178cdb3ee7d Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 08:30:28 +0100 Subject: [PATCH 7/9] plan: close the l2tp route-observer race entry (fixed in 9af30c440) 9af30c440 fixed TestPeerTeardownWithdrawsSubscriberRoute but left its known-failures entry open -- exactly the stale-entry problem this file warns about, where a resolved red sends the next session hunting. Closes it, and corrects two claims for the record. It was NOT load-sensitive and NOT 1-in-3: -race reports the race on the first run, 3/3, alone and in the full package. And it was never a product bug -- SetRouteObserver documents 'MUST be called before Start()' (reactor_setters.go:106-109) and the sole production caller honours it (subsystem.go:241 installs, :313 starts). The original diagnosis and suggested fix were both right; only the severity and framing were off. --- plan/known-failures.md | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/plan/known-failures.md b/plan/known-failures.md index ecb851e1b8..2300bf8db9 100644 --- a/plan/known-failures.md +++ b/plan/known-failures.md @@ -357,17 +357,32 @@ addresses reach the kernel through the owner registry, which is covered by `registry_integration_linux_test.go` and the new `device_owner_integration_linux_test.go` and is green. -### `internal/component/l2tp` `TestPeerTeardownWithdrawsSubscriberRoute` -- genuine `-race` data race, load-sensitive - -Observed 2026-07-01, re-confirmed 2026-07-03 (1/3 under `-race -count=3` in -isolation): `go test -race` reports a data race between -`L2TPReactor.SetRouteObserver` (`reactor_setters.go:114`, called from the test) -and `L2TPReactor.notifyRouteObserverDown` (`reactor_kernel.go:253`, called from -the reactor's goroutine) -- both racing on the same `RouteObserver` field. The -test calls `SetRouteObserver` after `Start()`, with no barrier against the -reactor goroutine already running. Fix: set the observer before `Start()`, or -synchronize via a channel. Owner: whichever session next touches -`internal/component/l2tp/reactor_test.go`. +### ~~`internal/component/l2tp` `TestPeerTeardownWithdrawsSubscriberRoute`~~ -- FIXED 2026-07-16 (`9af30c440`): the test violated a documented setter contract + +The diagnosis here was correct (write at `reactor_setters.go:114` vs read at +`reactor_kernel.go:263`, the test calling `SetRouteObserver` after `Start()`), and +the suggested fix -- set the observer before `Start()` -- was the one taken. + +Two corrections for the record. **It was not load-sensitive and not 1-in-3**: +`-race` reports it on the FIRST run, 3/3, both alone and in the full package. The +"1/3 under `-race -count=3`" measurement understated it. And it was never a +product bug: `SetRouteObserver` documents "MUST be called before `Start()`; the +goroutine creation barrier synchronizes the write here with reads in the run +loop" (`reactor_setters.go:106-109`), and the sole production caller honours it -- +`subsystem.go:241` installs the observer, `:313` starts the reactor 72 lines +later. The lock-free write is deliberate: the reload-time setters +(`setHelloRetries` and friends, `reactor_setters.go:18-98`) DO take `tunnelsMu` +because `subsystem_reload.go` calls them on a live reactor, while the install-time +setters trade the lock for the `Start()` happens-before edge. Adding a mutex would +have weakened a working design to accommodate a misusing test. + +Fixed by giving the builder an observer parameter that installs before `Start()` +(`buildLogReactorWithClockObserver`); the other 11 callers are untouched and the +wrapper passes nil, a documented no-op. `setHelloRetries` stays after `Start()`, +which is safe -- it locks. + +Verified: the two affected tests 5/5 under `-race`, and the whole +`./internal/component/l2tp/...` tree green under `-race`. ### `internal/component/l2tp` `TestReactorKernelDisabledReturnsNil` -- stale assertion vs. deliberate design change From 09a4bcdc0214969aa202c01261ec1139019b7238 Mon Sep 17 00:00:00 2001 From: Thomas Mangin Date: Thu, 16 Jul 2026 08:39:21 +0100 Subject: [PATCH 8/9] test(l2tp): assert the drained teardown, not the pre-e231fbfdd nil TestReactorKernelDisabledReturnsNil asserted require.Nil(t, teardowns) and had failed deterministically since e231fbfdd deliberately made the teardown drain unconditional. The assertion was stale, not the code -- the diagnosis in plan/known-failures.md was correct, and this is the fix it prescribed. Producer: collectKernelEventsLocked drains tunnel.pendingKernelTeardowns and nils the list BEFORE the worker check, then returns (nil, teardowns) when r.kernelWorker == nil (reactor_kernel.go:19-27). The drain is deliberately not gated on the worker so the route observer learns of torn sessions where no kernel worker exists -- the mechanism TestPeerTeardownWithdrawsSubscriberRoute depends on, i.e. the same teardown-withdraw path whose -race bug was fixed in 9af30c440. VERIFIED BY EXECUTION, not by reading. The file is _linux_test.go so it does not build on this macOS host (go test reports 'no tests to run'), which is why the entry sat open since 2026-07-10. Run in a golang:1.26 container -- the test is pure logic, no netlink and no privileges, so a container suffices: committed HEAD : FAIL reactor_kernel_linux_test.go:159: Expected nil, but got: []l2tp.kernelTeardownEvent{{localTID:0x66, localSID:0x9}} with the fix : PASS; full ./internal/component/l2tp/... tree green on linux/amd64 0x66/0x9 are exactly the event the test seeds, which confirms the producer above. The assertion is REPLACED, not dropped: a stricter require.Equal on the exact drained event plus a drained-list check, so the test now carries 4 assertions where it carried 3. The surviving assertions (setups nil, kernelSetupNeeded not cleared) were correct and are kept. Renamed to TestReactorKernelDisabledSkipsSetupsButStillDrainsTeardowns because 'ReturnsNil' now describes only the setups; leaving the old name would misdescribe the contract. Also closes the plan/known-failures.md entry. Like the chaos entry closed earlier today, this was a DETERMINISTIC red parked in a file whose own scope rule is non-deterministic failures only. Verified: make ze-lint-changed 0 issues; GOOS=linux go vet ./internal/component/l2tp/ clean (the darwin lint cannot see a _linux_test.go). --- .../l2tp/reactor_kernel_linux_test.go | 32 +++++++++++++++---- plan/known-failures.md | 31 +++++++++++++++++- 2 files changed, 55 insertions(+), 8 deletions(-) diff --git a/internal/component/l2tp/reactor_kernel_linux_test.go b/internal/component/l2tp/reactor_kernel_linux_test.go index 4342395c9e..d92e3f3c9d 100644 --- a/internal/component/l2tp/reactor_kernel_linux_test.go +++ b/internal/component/l2tp/reactor_kernel_linux_test.go @@ -138,11 +138,21 @@ func TestReactorCollectsTeardownEvent(t *testing.T) { require.Empty(t, tun.pendingKernelTeardowns, "teardown list must be drained") } -func TestReactorKernelDisabledReturnsNil(t *testing.T) { - // VALIDATES: with no kernel worker configured (non-Linux path or - // subsystem start without kernel support), collectKernelEventsLocked - // returns nil instead of producing events for a nil worker. - // PREVENTS: nil-deref when kernel integration is disabled. +func TestReactorKernelDisabledSkipsSetupsButStillDrainsTeardowns(t *testing.T) { + // VALIDATES: with no kernel worker configured (non-Linux path, or subsystem + // start without kernel support), collectKernelEventsLocked produces no + // SETUPS and leaves kernelSetupNeeded set for a later retry -- but it still + // drains pendingKernelTeardowns and returns them. + // + // The teardown drain is deliberately NOT gated on the worker + // (reactor_kernel.go:19-27): subscriber-route withdrawal is independent of + // kernel-resource teardown, so the route observer must learn of torn + // sessions even where no kernel worker exists. That is the path + // TestPeerTeardownWithdrawsSubscriberRoute exercises end to end. + // + // PREVENTS: nil-deref when kernel integration is disabled; and a regression + // to gating the drain on the worker, which would strand a torn session's + // /32 injected after the session was gone. _, r, stop := newUnstartedReactor(t) defer stop() @@ -155,8 +165,16 @@ func TestReactorKernelDisabledReturnsNil(t *testing.T) { setups, teardowns := r.collectKernelEventsLocked(tun) r.tunnelsMu.Unlock() - require.Nil(t, setups) - require.Nil(t, teardowns) + require.Nil(t, setups, "no kernel worker means no setup events") + // test-relax: the old teardowns-nil assertion asserted the pre-e231fbfdd + // behaviour and has failed deterministically ever since that commit made the + // drain unconditional on purpose (see the leading comment on + // collectKernelEventsLocked). The assertion was stale, not the code. It is + // REPLACED, not dropped, by the stricter equality check below plus the + // drained-list check -- this test now carries 4 assertions where it carried 3. + require.Equal(t, []kernelTeardownEvent{{localTID: 102, localSID: 9}}, teardowns, + "teardowns must drain even with no kernel worker, so routes are withdrawn") + require.Empty(t, tun.pendingKernelTeardowns, "teardown list must be drained") require.True(t, sess.kernelSetupNeeded, "flag must NOT clear when worker absent") } diff --git a/plan/known-failures.md b/plan/known-failures.md index 2300bf8db9..a933668b73 100644 --- a/plan/known-failures.md +++ b/plan/known-failures.md @@ -384,7 +384,36 @@ which is safe -- it locks. Verified: the two affected tests 5/5 under `-race`, and the whole `./internal/component/l2tp/...` tree green under `-race`. -### `internal/component/l2tp` `TestReactorKernelDisabledReturnsNil` -- stale assertion vs. deliberate design change +### ~~`internal/component/l2tp` `TestReactorKernelDisabledReturnsNil`~~ -- FIXED 2026-07-16: stale assertion retired, verified on real Linux + +The diagnosis logged here was exactly right, and the fix is the one it prescribed. +Recorded because the evidence is now direct rather than inferred: this is a +`_linux_test.go`, so it does not build on the macOS dev host (`go test` reports +"no tests to run"), which is why it sat open. Run in a `golang:1.26` container -- +the test is pure logic (no netlink, no privileges), so a container suffices: + +| Tree | Result on linux/amd64 | +|---|---| +| committed HEAD (`require.Nil(t, teardowns)`) | **FAIL** `reactor_kernel_linux_test.go:159: Expected nil, but got: []l2tp.kernelTeardownEvent{{localTID:0x66, localSID:0x9}}` | +| with the fix | **PASS**; full `./internal/component/l2tp/...` tree green | + +`0x66`/`0x9` are exactly the event the test seeds, confirming the producer: +`collectKernelEventsLocked` drains `pendingKernelTeardowns` BEFORE the worker +check and returns them (`reactor_kernel.go:23-27`), deliberately, so the route +observer learns of torn sessions with no kernel worker present. That is the +mechanism `TestPeerTeardownWithdrawsSubscriberRoute` depends on -- the same +teardown-withdraw path whose `-race` bug was fixed today in `9af30c440`. + +Fixed by retiring the teardowns-nil assertion and asserting the drained event +instead (a stricter `require.Equal` on the exact event, plus a drained-list +check: 4 assertions where there were 3). Renamed to +`TestReactorKernelDisabledSkipsSetupsButStillDrainsTeardowns`, because +"ReturnsNil" now describes only the setups. The surviving assertions (setups nil, +`kernelSetupNeeded` not cleared) were correct and are kept. + +Note this was a DETERMINISTIC red, so like the chaos entry above it never +belonged in a file scoped to non-deterministic failures. Two of this file's +entries turned out that way today. Confirmed pre-existing (git-blame) 2026-07-10: the test asserts `require.Nil(t, teardowns)` from `collectKernelEventsLocked` when no kernel From 24616bfe62eb7e4b88fde43578d3d736ee401263 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 16 Jul 2026 07:47:27 +0000 Subject: [PATCH 9/9] build(deps): bump the github-actions group across 1 directory with 6 updates Bumps the github-actions group with 6 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4` | `7` | | [actions/setup-go](https://github.com/actions/setup-go) | `5` | `7` | | [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `6` | `7` | | [actions/configure-pages](https://github.com/actions/configure-pages) | `5` | `6` | | [actions/upload-pages-artifact](https://github.com/actions/upload-pages-artifact) | `3` | `5` | | [actions/deploy-pages](https://github.com/actions/deploy-pages) | `4` | `5` | Updates `actions/checkout` from 4 to 7 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v7) Updates `actions/setup-go` from 5 to 7 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v5...v7) Updates `astral-sh/setup-uv` from 6 to 7 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/v6...v7) Updates `actions/configure-pages` from 5 to 6 - [Release notes](https://github.com/actions/configure-pages/releases) - [Commits](https://github.com/actions/configure-pages/compare/v5...v6) Updates `actions/upload-pages-artifact` from 3 to 5 - [Release notes](https://github.com/actions/upload-pages-artifact/releases) - [Commits](https://github.com/actions/upload-pages-artifact/compare/v3...v5) Updates `actions/deploy-pages` from 4 to 5 - [Release notes](https://github.com/actions/deploy-pages/releases) - [Commits](https://github.com/actions/deploy-pages/compare/v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-go dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/configure-pages dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/upload-pages-artifact dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/deploy-pages dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] --- .github/workflows/codeql.yml | 2 +- .github/workflows/pages.yml | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 870d3cd6e7..a5b7785645 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -59,7 +59,7 @@ jobs: # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v7 # Add any setup steps before running the `github/codeql-action/init` action. # This includes steps like installing compilers or runtimes (`actions/setup-node` diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index 6188dbc6a6..6641d0d3bc 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -27,25 +27,25 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout Ze - uses: actions/checkout@v4 + uses: actions/checkout@v7 with: ref: main path: main - name: Checkout website - uses: actions/checkout@v4 + uses: actions/checkout@v7 with: ref: gh-pages path: site - name: Set up Go - uses: actions/setup-go@v5 + uses: actions/setup-go@v7 with: go-version-file: main/go.mod cache-dependency-path: main/go.sum - name: Set up uv - uses: astral-sh/setup-uv@v6 + uses: astral-sh/setup-uv@v7 - name: Install VHS and ffmpeg run: | @@ -60,13 +60,13 @@ jobs: working-directory: site - name: Configure Pages - uses: actions/configure-pages@v5 + uses: actions/configure-pages@v6 - name: Upload Pages artifact - uses: actions/upload-pages-artifact@v3 + uses: actions/upload-pages-artifact@v5 with: path: site - name: Deploy Pages id: deployment - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@v5