Skip to content

devnet-7: tysm beacon canary on prysm-geth-1 wired to bad-tysm#45

Merged
parithosh merged 1 commit into
masterfrom
devnet-7/tysm-canary-prysm-geth-1
Jul 21, 2026
Merged

devnet-7: tysm beacon canary on prysm-geth-1 wired to bad-tysm#45
parithosh merged 1 commit into
masterfrom
devnet-7/tysm-canary-prysm-geth-1

Conversation

@samcm

@samcm samcm commented Jul 21, 2026

Copy link
Copy Markdown
Member

Runs a tysm beacon canary on prysm-geth-1 and registers it with the production bad-tysm control plane. Only the beacon image changes to ethpandaops/tysm:glamsterdam-devnet-7; the validator and geth EL stay on their stock devnet-7 images. epbs-mutator ships disabled and bad-tysm activates it at runtime.

Verified live: the beacon is synced and stable, appears in bad-tysm as a push-discovered node with fresh heartbeats, its control API is reachable on 8675, and epbs-mutator is present with runtime_reconfigurable: true.

What the tysm build needs beyond stock prysm

  • a hidden --secret-access-key gate flag — value stored in sops (tysm_secret_access_key), not in this repo; without it the beacon exits with a panda banner and no message.
  • --xatu-config-file — mandatory even with the xatu hook disabled. A no-op local output satisfies it; devnet events keep flowing through the existing xatu-sentry sidecar.
  • --tysm-hook-config-file — carries the api (control API on 0.0.0.0:8675) and discovery (bad-tysm URL + shared token) blocks. TYSM_TOKEN is injected from the tysm_secret_key sops value.

Image requirement

Needs a tysm:glamsterdam-devnet-7 image built on/after 2026-07-20 (the epbs_mutator merge). The 07-17 image predated the hook. Ansible compares by tag not digest, so a rebuild under the same tag shows changed=0 — watchtower handles the container recreate on these hosts.

Deploying it

The prysm role has no template/copy task, so the two config files are pushed out-of-band into the datadir (they surface at /data in the container via the existing mount):

ansible prysm-geth-1 -b -m copy -a "src=inventories/devnet-7/files/tysm/prysm-geth-1-hooks.yaml dest=/data/prysm/tysm-hooks.yaml owner=prysm group=prysm mode=0644"
ansible prysm-geth-1 -b -m copy -a "src=inventories/devnet-7/files/tysm/prysm-geth-1-xatu-config.yaml dest=/data/prysm/tysm-xatu-config.yaml owner=prysm group=prysm mode=0644"
ansible-playbook playbook.yaml --limit prysm-geth-1 --tags ethereum_node_cl

The 8675/tcp firewall rule (DO + Hetzner, mirroring the 8961 snooper rule) is included. Extending to more prysm nodes is a copy of the host_vars + the two files per host (fix the self_url/instance_id/name).

@samcm

samcm commented Jul 21, 2026

Copy link
Copy Markdown
Member Author

Live and verified. The tysm:glamsterdam-devnet-7 image had to be rebuilt first — the published image was from 07-17, before epbs_mutator merged onto the branch (07-20), so the hook wasn't in the binary. Rebuilt from current HEAD (tysm run 29798151390); watchtower auto-pulled and recreated the beacon on the new digest.

prysm-geth-1 now: synced, registered + heartbeating in production bad-tysm, control API reachable on 8675, and epbs-mutator present with runtime_reconfigurable: true (shows in the bad-tysm activation picker).

Note for future rolls: this needs an image built on/after 2026-07-20. Ansible compares by tag not digest, so a rebuild under the same tag shows changed=0 — watchtower handles the container recreate on these hosts.

Swaps only the prysm-geth-1 beacon to ethpandaops/tysm:glamsterdam-devnet-7
(validator + geth EL stay stock) and points it at the production bad-tysm
control plane via push discovery. epbs-mutator ships disabled; bad-tysm
activates it at runtime.

The tysm build needs a xatu config path even with the xatu hook disabled (a
no-op local output satisfies it), a hook-config file with the api + discovery
blocks, and a gate flag whose value is sourced from sops. TYSM_TOKEN and the
gate value both come from sops, not plaintext.

Publishes 8675 and opens it on the shared devnet-7 firewall so bad-tysm can
reach the control API. The prysm role has no file-push task, so the two config
files are delivered with ad-hoc ansible copy into the datadir.

Claude-Session: https://claude.ai/code/session_01ATKD5DcMqmzXj3DLymX9ZY
@samcm
samcm force-pushed the devnet-7/tysm-canary-prysm-geth-1 branch from d56bd9b to 25695b9 Compare July 21, 2026 04:10
@parithosh
parithosh merged commit 8a20465 into master Jul 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants