Skip to content

feat(spartan): add the inbox bot as a fourth bot flavour - #190

Draft
spalladino wants to merge 1 commit into
spl/a-2019-consolidate-fast-inbox-e2e-coveragefrom
spl/a-2020-deploy-the-inbox-bot-to-next-net-and-staging
Draft

spalladino wants to merge 1 commit into
spl/a-2019-consolidate-fast-inbox-e2e-coveragefrom
spl/a-2020-deploy-the-inbox-bot-to-next-net-and-staging

Conversation

@spalladino

@spalladino spalladino commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Deploy the inbox bot to next-net and staging

Fixes A-2020. Follows A-1986, which built the inbox bot itself.

This adds BOT_MODE=inbox to the spartan stack as a fourth bot flavour, alongside
transfers, swaps and cross-chain. It is deliberately a no-op deploy: every
BOT_INBOX_REPLICAS default is 0, so the Terraform release is absent on every
environment and no environment file is touched here. Turning the bot on is the next
three PRs, one per network, in rollout order.

Rollout

The plan is to land this wiring at replicas 0, then enable next-net, then
staging-internal, then staging-public, each as its own deploy with a 24h soak
before the next. This PR is the replicas-0 landing step. The plan document
explicitly keeps the three spartan/environments/*.env enablement blocks out of the
no-op wiring change, so they are not in this diff even though the issue lists those
files in scope; each enable ships with the deploy it belongs to.

For reference, the block each of next-net.env, staging-internal.env and
staging-public.env will get, one at a time:

BOT_INBOX_REPLICAS=1
BOT_INBOX_TX_INTERVAL_SECONDS=300
BOT_INBOX_FOLLOW_CHAIN=PROPOSED
BOT_INBOX_MESSAGES_PER_BATCH=4
BOT_INBOX_CONSUME_MODE=mixed
BOT_INBOX_SATURATION_INTERVAL_SECONDS=0
BOT_INBOX_L1_TO_L2_TIMEOUT_SECONDS=3600

The saturation batch ships disabled

BOT_INBOX_SATURATION_INTERVAL_SECONDS=0 on every path that can set it. This is not
a default being inherited: the bot code defaults inboxSaturationIntervalSeconds to
86400, so spartan/aztec-bot/values.yaml, values/bot-inbox.yaml, the Terraform
variable and the deploy_network.sh default all set 0 explicitly.

A saturation run sends 257 sendL2Message calls in one atomic Multicall3
aggregate3 transaction to force a bucket rollover. It cannot partially land.
Enabling it is a separate exercise that starts by measuring the batch's L1 gas
against Sepolia's block gas limit, and every daily-run alert must be gated on
saturation_enabled == 1 so a deliberately disabled schedule never pages.

Two things worth a reviewer's attention

Dedicated gas limits. The inbox release takes BOT_INBOX_DA_GAS_LIMIT and
BOT_INBOX_L2_GAS_LIMIT, not the shared BOT_DA_GAS_LIMIT / BOT_L2_GAS_LIMIT the
other three bots inherit. Staging pins those shared values to 100000 and 6540000,
tuned for a token transfer. A private consumption of an L1 to L2 message is a
different circuit; too low a fixed limit fails the transaction, too high overpays,
and either way the probe stops measuring what it exists to measure. Both inbox
variables default to empty, so the wallet estimates.

Funding. calculate_publisher_indices.sh now reports mnemonic index 7300 once
the bot is enabled. That list is what ensure_funded_environment tops up, and all
three target networks run with CREATE_ETH_DEVNET=false, so PREFUNDED_MNEMONIC_INDICES
never reaches them. Without this the bot's L1 account is never funded and it fails on
its first aggregate3.

Index 7300 keeps the existing 100-wide spacing after cross-chain's 7200, and each of
the three networks has its own mnemonic, so the index is free and independent in all
of them.

Configuration

One replica; two would each keep their own store and their own saturation schedule
with no coordination between them. Four messages per batch every 300 seconds, in
mixed consume mode so one bot covers both the public and private domains.
followChain: PROPOSED — inbox mode rejects NONE outright, and PROPOSED keeps
the outstanding-message cap draining while still recording first-proposed inclusion
separately. maxErrors: 3 with stopIfUnhealthy: false: maxErrors: 0 would
disable the unhealthy state rather than merely prevent a restart, so the threshold
stays and only the exit is disabled, which keeps the measuring process exporting
failure and timeout counts. A 2Gi PVC, because restart reconciliation reads the
durable store and an emptyDir would make every pod restart look like a fresh bot.

Verification

No deploy, no terraform apply, no live cluster.

  • helm template on the three existing bot flavours before and after the chart
    change: all three configmaps render byte-identical, so their pods will not roll.
  • helm template on the inbox flavour renders
    BOT_INBOX_SATURATION_INTERVAL_SECONDS: "0" and omits BOT_DA_GAS_LIMIT /
    BOT_L2_GAS_LIMIT entirely, which is the "estimate" path.
  • helm lint on the chart with the inbox values: clean.
  • terraform validate: success. terraform console with BOT_DA_GAS_LIMIT=100000
    and BOT_L2_GAS_LIMIT=6540000 set, as staging sets them, evaluates
    bot_transfers with those limits and bot_inbox with "" for both, which is the
    contrast the dedicated variables exist to produce.
  • terraform fmt -check on the new blocks: clean.
  • bash -n on both scripts: clean. calculate_publisher_indices.sh emits 7300 with
    BOT_INBOX_REPLICAS=1 and is unchanged from its previous output at 0.
  • shellcheck is not installed in this environment, so the scripts were not
    shellchecked.

Recorded, not resolved

The shared BOT_DA_GAS_LIMIT / BOT_L2_GAS_LIMIT never reach Terraform today.
staging-internal.env and staging-public.env set them, but deploy_network.sh
does not write either into terraform.tfvars and nothing else exports them, so the
Terraform variables sit at their "" defaults and all four bots estimate regardless.
The dedicated inbox variables are still correct and are plumbed end to end, but the
"staging gas-limit trap" is latent rather than live. Left alone — repairing the
shared path would change the transfer and swap bots' behaviour on staging, which is
out of scope here.

Two pre-existing failures on this base, untouched by this change.
terraform fmt -check flags main.tf:401 (logLevel alignment), and
spartan/scripts/check_env_vars.sh reports 7 undefined variables
(AZTEC_EPOCHS_LAG, DEVNET_NAMESPACES_REGEX, MAINNET_NAMESPACES_REGEX,
NEXT_NET_REGEX, NEXT_SCENARIO_REGEX, STAGING_REGEX,
TESTNET_NAMESPACES_REGEX). None appear in this diff. spartan/bootstrap.sh runs
that check, so it is presumably already failing.

Sepolia funding is an operational prerequisite for each enable, not a repo change.
The inbox bot is the only bot with a continuous L1 burn — one transaction every 300
seconds, forever, on three networks at once, against roughly 38.6M gas/day per
network. ensure_funded_environment defaults to a 0.5 ETH low watermark and a 1.0
ETH top-up, which is about three days of headroom at 10 gwei and under a day through
a spike, and no scheduled job runs it. Before enabling each network, seed index 7300
to around 5 ETH with a targeted call rather than raising the environment-wide
watermarks, which apply to every index:

spartan/scripts/ensure_eth_balances.sh "$ETHEREUM_HOST" "$FUNDING_PRIVATE_KEY" "$LABS_INFRA_MNEMONIC" 7300 5

and put either scheduled replenishment or a balance alert on 7300 in place first. A
funding gap makes the probe report L1 submission failures that look like network
faults, which is the worst failure mode for a measuring instrument.

A dependency gate before the first enable. The image selected for deployment has
to contain BOT_MODE=inbox. Rendering valid Helm and Terraform proves nothing about
whether a deployed binary can start the new mode.

@spalladino
spalladino added this pull request to stack #188 September 12, 2026 06:48
@spalladino spalladino added the ci-draft Run CI on this draft PR label Sep 12, 2026
Wire BOT_MODE=inbox into the Helm chart and the deploy-aztec-infra Terraform
stack beside the transfers, swaps and cross-chain bots. Every BOT_INBOX_REPLICAS
default is 0, so this is a no-op on every environment: the Terraform release is
simply absent until a network opts in.

The bot chart gains BOT_L1_TO_L2_SEED_COUNT, BOT_L1_TO_L2_TIMEOUT_SECONDS,
BOT_INBOX_MESSAGES_PER_BATCH, BOT_INBOX_CONSUME_MODE and
BOT_INBOX_SATURATION_INTERVAL_SECONDS. The three inbox-only keys are gated on
botMode so the other flavours render a byte-identical configmap and their pods
do not roll; the two seed/timeout keys are gated on being set, which keeps the
cross-chain bot on the code default it runs today.

The daily saturation batch is off. It sends 257 sendL2Message calls in a single
atomic Multicall3 aggregate3 transaction to force a bucket rollover, so it wants
its own measured rollout rather than riding along with steady-state traffic.
The bot code defaults the interval to 86400, so both values.yaml and
bot-inbox.yaml set it to 0 explicitly rather than relying on a default.

The release takes dedicated BOT_INBOX_DA_GAS_LIMIT and BOT_INBOX_L2_GAS_LIMIT
rather than the shared BOT_DA_GAS_LIMIT and BOT_L2_GAS_LIMIT the other three
bots inherit. Staging pins those shared limits to values tuned for a token
transfer; a private consumption of an L1 to L2 message is a different circuit,
and a fixed limit that is too low fails the transaction while one that is too
high overpays. Both inbox variables default to empty so the wallet estimates.

calculate_publisher_indices.sh reports mnemonic index 7300 once the bot is
enabled. That list is what ensure_funded_environment tops up on Sepolia, where
CREATE_ETH_DEVNET is false, so without it the bot's L1 account never receives
ETH and fails on its first aggregate3.
@spalladino
spalladino force-pushed the spl/a-2020-deploy-the-inbox-bot-to-next-net-and-staging branch from 36ecf14 to 67d262f Compare September 15, 2026 22:39
@spalladino
spalladino removed this pull request from stack #188 September 15, 2026 22:42
@spalladino
spalladino added this pull request to stack #217 September 15, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on this draft PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant