Unified ingestion workflow: streaming daemon — Slice 2 (events)#803
Closed
chowbao wants to merge 2 commits into
Closed
Unified ingestion workflow: streaming daemon — Slice 2 (events)#803chowbao wants to merge 2 commits into
chowbao wants to merge 2 commits into
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
chowbao
force-pushed
the
streaming-slice1-ledgers
branch
from
9611dfb to
c031951
Compare
chowbao
force-pushed
the
streaming-slice2-events
branch
2 times, most recently
from
cb5c1e3 to
a0935aa
Compare
chowbao
force-pushed
the
streaming-slice1-ledgers
branch
from
c031951 to
62d69a7
Compare
chowbao
force-pushed
the
streaming-slice2-events
branch
from
a0935aa to
9c92206
Compare
This was referenced Jun 23, 2026
chowbao
force-pushed
the
streaming-slice1-ledgers
branch
from
62d69a7 to
f4ca90f
Compare
chowbao
force-pushed
the
streaming-slice2-events
branch
2 times, most recently
from
de84d1d to
e823bc7
Compare
chowbao
force-pushed
the
streaming-slice1-ledgers
branch
from
f4ca90f to
cb57955
Compare
chowbao
force-pushed
the
streaming-slice2-events
branch
from
e823bc7 to
a5d2543
Compare
Stacked on slice 1 (the ledgers skeleton); this commit's diff is only the
events additions on top of it. Adds the EVENTS data type to the streaming
daemon:
- events column families in the per-chunk hot RocksDB (hotchunk), so one
atomic synced WriteBatch per ledger now carries ledgers + events;
- the events cold-segment writer in processChunk;
- the chunk:{c}:events catalog key + its sweeps;
- events coverage in the audit (INV-3 disk<->catalog) and in the
crash-injection/convergence suite and lifecycle E2E.
Events is a per-chunk artifact, like ledgers — no window/index subsystem
(that is tx-hash, deferred to slice 3).
Composes the events store (pkg/stores/eventstore, #740/#756) and the events
design (getevents-full-history-design.md, #635), already on
feature/full-history.
Built against RocksDB 10.9.1 (grocksdb 1.10.7); fullhistory tree green on the
non-short suite incl. the lifecycle E2E.
doc.go: events adds no new streaming/*.go files (woven into existing seams); update the coverage line, data-model artifact note, and 'Later slices' pointer (now only tx-hash). golangci-lint (this slice's own new findings, in the files events modifies): - eventstore/hot_store.go: //nolint:nilnil on the three documented idempotent-duplicate (nil hook/ledger, nil err) returns; wrap the IngestLedgerToBatch signature (lll); drop a dead //nolint:cyclop - hotchunk: preallocate the columnFamilies slice (prealloc)
chowbao
force-pushed
the
streaming-slice2-events
branch
from
a5d2543 to
f808a54
Compare
This was referenced Jun 24, 2026
Contributor
Author
|
Superseded by a new 2-phase stacked series that re-slices this work by phase (backfill → live ingestion + lifecycle), with the MVP scope cuts (recovery / audit / convergence / retention-reconfiguration dropped) and the folded-in fixes (cold+hot ingest service +
Each layer builds + |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice 2 of the unified ingestion workflow (design #722 / roadmap #777), stacked on slice 1 (#802 — the ledgers skeleton). The base of this PR is
streaming-slice1-ledgers, so the diff shown here is only the events additions on top of the ledger skeleton. Review after slice 1.What this adds (the events data type)
pkg/stores/hotchunk), so one atomic syncedWriteBatchper ledger now carries ledgers + events.processChunk.chunk:{c}:eventscatalog key + its key-driven sweeps.auditcommand (INV-3 disk↔catalog) and in the crash-injection/convergence suite + the in-process lifecycle E2E.Events is a per-chunk artifact, like ledgers — there is no window/index subsystem here (that is tx-hash, slice 3). The events wiring is small and additive on top of the skeleton.
Composes (already on
feature/full-history)pkg/stores/eventstore(#740/#756) and the events design docgetevents-full-history-design.md(#635).Deferred to slice 3
The tx-hash CF,
.bin/.idx, and the per-window rolling-index subsystem (the design's hardest-to-review component).Testing
Built against RocksDB 10.9.1 (grocksdb 1.10.7). Full
fullhistorytree green on the non-short suite incl. the lifecycle E2E:go test ./cmd/stellar-rpc/internal/fullhistory/....