[Prototype] Bigtable

[Kbhat1]

Describe your changes and provide context

Testing performed to validate your change

---

Note

Medium Risk
Adds an optional fallback path for pruned Get/Has state-store reads to query external Scylla/Cassandra or Bigtable, plus new ingestion tooling and cloud/db dependencies; misconfiguration or backend issues could affect historical query behavior.

Overview
Adds prototype historical-state offload support so SS point reads (Get/Has) for versions below local retention can fall back to an external store (Scylla/Cassandra or Bigtable), wrapped via historical.FallbackStateStore and enabled through new [state-store] TOML/flag config (mutually exclusive backends).

Introduces a new Kafka-based offload consumer (sei-db/state_db/ss/offload/consumer) with configurable sinks for Scylla and Bigtable, including schema/examples and retry/batching/backpressure behavior, and exports offload.NewSASLMechanism so external consumers share the producer auth path.

Extends SS config structs/templates/tests for the new historical-offload settings and adds new dependencies (cloud.google.com/go/bigtable, github.com/gocql/gocql, golang.org/x/oauth2).

^{Reviewed by Cursor Bugbot for commit 66f8c9f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	May 18, 2026, 5:59 PM

[codecov]

Codecov Report

❌ Patch coverage is 40.75630% with 705 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.14%. Comparing base (654d40b) to head (a406321).
⚠️ Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/ss/offload/historical/bigtable.go	38.95%	184 Missing and 15 partials ⚠️
sei-db/state_db/ss/offload/consumer/consumer.go	0.00%	177 Missing ⚠️
sei-db/state_db/ss/offload/historical/scylla.go	49.29%	71 Missing and 1 partial ⚠️
sei-db/state_db/ss/offload/consumer/scylla.go	66.85%	48 Missing and 11 partials ⚠️
sei-db/state_db/ss/store.go	22.64%	39 Missing and 2 partials ⚠️
sei-db/state_db/ss/offload/historical/store.go	39.34%	32 Missing and 5 partials ⚠️
sei-db/state_db/ss/offload/consumer/config.go	39.28%	29 Missing and 5 partials ⚠️
sei-db/state_db/ss/offload/consumer/bigtable.go	69.72%	27 Missing and 6 partials ⚠️
sei-db/state_db/ss/offload/consumer/kafka.go	44.82%	31 Missing and 1 partial ⚠️
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00%	21 Missing ⚠️

❌ Your patch check has failed because the patch coverage (40.75%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3437      +/-   ##
==========================================
- Coverage   59.25%   59.14%   -0.11%     
==========================================
  Files        2110     2120      +10     
  Lines      174181   176194    +2013     
==========================================
+ Hits       103210   104216    +1006     
- Misses      62044    62969     +925     
- Partials     8927     9009      +82     

Flag	Coverage Δ
sei-chain-pr	48.70% <40.75%> (?)
sei-db	70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
app/seidb.go	74.39% <100.00%> (+2.39%)	⬆️
sei-db/config/ss_config.go	100.00% <ø> (ø)
sei-db/state_db/ss/offload/kafka.go	59.37% <100.00%> (ø)
...ad/consumer/cmd/historical-scylla-consumer/main.go	0.00% <0.00%> (ø)
sei-db/state_db/ss/offload/consumer/kafka.go	44.82% <44.82%> (ø)
sei-db/state_db/ss/offload/consumer/bigtable.go	69.72% <69.72%> (ø)
sei-db/state_db/ss/offload/consumer/config.go	39.28% <39.28%> (ø)
sei-db/state_db/ss/offload/historical/store.go	39.34% <39.34%> (ø)
sei-db/state_db/ss/store.go	24.07% <22.64%> (-75.93%)	⬇️
sei-db/state_db/ss/offload/consumer/scylla.go	66.85% <66.85%> (ø)
... and 3 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

PR Summary

Medium Risk
Adds new external backends (Scylla/Cassandra and Google Bigtable) and changes SS Get/Has behavior to optionally read from them when local history is pruned, which could impact query correctness/latency and introduces new dependency/config surface.

Overview
Adds a prototype historical-state offload path for SeiDB SS. Nodes can now optionally fallback point reads (Get/Has) to an external historical store (Scylla/Cassandra or Bigtable) when the requested version is older than local SS retention; iteration and writes remain on local SS via a new historical.FallbackStateStore wrapper.

Introduces a Kafka-based offload consumer (sei-db/state_db/ss/offload/consumer) that ingests changelog entries and writes MVCC mutation rows + version markers into Scylla or Bigtable (including schema/config examples), plus new SS config/TOML fields and CLI flags to configure the fallback reader. Also exports offload.NewSASLMechanism so the consumer shares the producer’s Kafka auth path, and adds new deps for gocql and Bigtable/gRPC OAuth.

^{Reviewed by Cursor Bugbot for commit a406321. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ecdf5cc. Configure here.}