eth/sequencer, ci: trip read breaker on a frozen store tail; add kurtosis sequencer e2e leg

[cffls]

Summary

Two related changes to keep the sequence store off the block-production critical path, and to make CI prove it against a bor built from the PR.

1. eth/sequencer: trip the read breaker on a frozen store tail

Fixes the "frozen tail" liveness fault: a sequence-store gateway cut off from its broker keeps answering reads from an in-memory window that no longer advances, and the producer — reading that same held height on every build — sees what looks like a rival already holding the height, refuses to seal, re-reads, and refuses again until the fault clears. A store outage that is supposed to pause preconfirmations and nothing else instead slows block production.

The gap is that #2406's read breaker keys on silence: isSilent counts only timeout / Unavailable / Canceled, and a frozen tail answers (fast, successful, stale), so the breaker never trips; writeDown covers only the write path, which is healthy during a gateway↔broker partition. This adds staleness detection on the read path:

• readBreaker.observeLiveTail(head) — the live-tail head (resp.GetNext() at a Live page in walk) that does not advance across staleReadsToTrip (6) reads opens the breaker with reason stale. Any advancement resets the counter and clears the hold.
• answered() returns early when the hold is stale: a frozen tail still answers, so a bare successful round trip must not clear a staleness hold — only real advancement does. The silence path is unchanged.
• Once open, reads return errReadPathDown, so the pre-seal barrier and seal gate take their existing "unreadable → proceed" branch (SealUnknown) and production continues at full cadence. The 5 s probe re-reads once per interval and clears the hold the moment the tail moves.

This also subsumes the degraded-but-answering (too-slow-to-advance) store that the silence-only breaker missed. The companion gateway-readiness fix (fail /ready when the broker offset can't be sampled) belongs in the sequence-store service; this makes the producer self-protecting regardless of gateway behavior.

fault	gateway serves	before	with this PR
gateway↔broker partition, ingress healthy (run 07)	frozen tail (answers)	contending, throughput ~halved	full cadence
gateway stopped (run 01)	timeout (silent)	proceeds	proceeds (unchanged)
broker/ingress down (run 08)	write error	proceeds via writeDown	proceeds (unchanged)

2. ci: add the Kurtosis sequencer e2e leg

Adds Kurtosis Sequencer E2E Tests, running the pos-workflows sequence-store suites (functional ~25 m, rpc ~5 m, chaos ~10 m) against a bor built from this checkout, on pushes and PRs to cffls/sequence-publisher plus workflow_dispatch. pos-workflows runs the same suites but builds bor from a fixed ref and is not a workflow_call workflow, so it cannot gate a bor PR. Mirrors bor's kurtosis-e2e.yml (build bor:local from the PR, heimdall-v2:local from develop, pos-workflows' configs/kurtosis-sequencer-e2e.yml, kurtosis-pos v1.4.2, polycli v0.1.103).

• Scoped to the sequencing branch with no path filter — every PR into it is sequencer work; path-scoping is the follow-up when this graduates to develop.
• The e2e job runs for trusted refs only: the sequence-store image is a private ghcr package fork pull_request runs cannot pull. Build jobs still run for forks.
• Artifacts are namespaced (sequencer-*-image) to avoid colliding with the other kurtosis legs on the same ref.

Note: the chaos suite covers broker stop, gateway pause, and gateway latency, but not the gateway↔broker partition that produces a frozen tail — so it exercises the store-outage paths but does not yet regression-test change 1. A dedicated episode (100 % loss scoped to the broker IP on the gateway container) is a follow-up in pos-workflows.

Executed tests

Change 1

• Unit (eth/sequencer/readbreaker_stale_test.go, confirmed to fail without the fix): TestReadBreakerTripsOnFrozenTail (frozen tail opens the breaker; a successful-but-frozen read does not clear it; an advancing tail does) and TestReadBreakerIgnoresAdvancingTail.

• Full eth/sequencer suite green on the combined tree (on top of sequencer, rawdb, ethapi: audit the store for the window a node did not watch #2388), gofmt clean.

• Reproduced end to end on a Kurtosis devnet (3 producers + 4 consumers + 1 RPC, single-instance seqstore, ~1 s cadence). Fault = pumba netem loss 100% on gateway→redpanda only, ingress→redpanda healthy, 90 s hold:

  • Unfixed (bor:2406): sequencer_publish_state=5 (contending) for the entire partition; throughput fell from 8 to ~2–6 blocks / 8 s, recovering only on heal.
  • Fixed: sequencer_read_breakeropened=1, 131 stale reads refused, breakerclosed=1 on heal; producer log Sequencer store tail is frozen (answering but not advancing) → Sequencer store tail is advancing again; throughput held 8 blocks / 8 s throughout.

  (On this 1 s-cadence devnet the fault is a sustained ~50 % slowdown rather than the 4 s-cadence incident's ~123 s stall; mechanism and publish_state=5 match.)

Change 2

• Workflow YAML parsed and structurally checked (jobs, needs graph, every steps.<id>.outcome referenced in an if: is a defined step id).
• Exercised by this PR itself: same-repo branch targeting cffls/sequence-publisher, so the new leg runs here against this PR's bor build — treat a green e2e-tests job as the acceptance test. First-run watch item: Pull sequence-store image relies on this repo's read grant on ghcr.io/0xpolygon/sequence-store (a SEQUENCE_STORE_GHCR_TOKEN fallback is wired).

Rollout notes

Not consensus-affecting; no coordinated upgrade. Change 1 is confined to the sequence-store publisher (off by default), no new knobs, no wire or database changes, no safety-property change — every site touched already returned "proceed" once its budget expired; this changes when the producer stops waiting on a store that cannot inform it, not what it does about it. Reuses the existing sequencer/read/* metrics. Change 2 is CI only; not wired as a required status check — that is a branch-protection setting to flip once the leg is stable.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.12%. Comparing base (44ec7db) to head (f620a4b).

Files with missing lines	Patch %	Lines
eth/sequencer/readbreaker.go	90.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@                     Coverage Diff                      @@
##           cffls/sequence-publisher    #2423      +/-   ##
============================================================
- Coverage                     57.17%   57.12%   -0.06%     
============================================================
  Files                           953      953              
  Lines                        174875   174893      +18     
============================================================
- Hits                          99990    99903      -87     
- Misses                        69219    69310      +91     
- Partials                       5666     5680      +14     

Files with missing lines	Coverage Δ
eth/sequencer/reader.go	88.58% <100.00%> (+0.03%)	⬆️
eth/sequencer/readbreaker.go	96.66% <90.00%> (-3.34%)	⬇️

... and 31 files with indirect coverage changes

Files with missing lines	Coverage Δ
eth/sequencer/reader.go	88.58% <100.00%> (+0.03%)	⬆️
eth/sequencer/readbreaker.go	96.66% <90.00%> (-3.34%)	⬇️

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}