Skip to content

Streaming daemon: Slice 1 — Layer 4 (daemon + ops + validation)#802

Closed
chowbao wants to merge 4 commits into
slice1cfrom
streaming-slice1-ledgers
Closed

Streaming daemon: Slice 1 — Layer 4 (daemon + ops + validation)#802
chowbao wants to merge 4 commits into
slice1cfrom
streaming-slice1-ledgers

Conversation

@chowbao

@chowbao chowbao commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Slice 1, Layer 4 of 4 — Daemon assembly, operability & validation. Stacked on Layer 3 (orchestration). This completes the ledgers skeleton — with Layers 1–4 applied, the daemon is whole.

  • startup orchestration (startStreaming: catalog → validate → catch-up → serve+ingest handoff) and validateConfig (the network-dependent earliest-ledger resolution — it lives here, not in foundations, because it calls networkTip);
  • the runnable daemon/CLI entrypoint (main.go wiring);
  • surgical recovery, the audit command (INV-2/3/4 for ledgers), retention ops;
  • the crash-injection/convergence suite and the in-process lifecycle E2E.

Compiles + full -short/E2E green. (This PR was previously the single "Slice 1 (ledgers skeleton)" PR #802 — now retargeted to sit atop Layers 1–3, so its diff is just the daemon-assembly layer.)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@chowbao chowbao force-pushed the streaming-slice1-ledgers branch 2 times, most recently from c031951 to 62d69a7 Compare June 23, 2026 17:34
@chowbao chowbao changed the title Unified ingestion workflow: streaming daemon — Slice 1 (ledgers skeleton) Streaming daemon: Slice 1 — Layer 4 (daemon + ops + validation) Jun 23, 2026
@chowbao chowbao changed the base branch from feature/full-history to slice1c June 23, 2026 17:34
@chowbao chowbao force-pushed the streaming-slice1-ledgers branch from 62d69a7 to f4ca90f Compare June 23, 2026 22:32
chowbao added 3 commits June 23, 2026 23:35
doc.go: add the Daemon (startup/daemon/config_validate) and Operability
(recovery/audit/audit_invariants) groups; this completes Slice 1, so the
note now points only at Slices 2 & 3 (events, tx-hash).

golangci-lint (this layer's own new findings) — auto-fixable ones via
golangci-lint --fix (protogetter, intrange, modernize, gofumpt, misspell,
perfsprint, some testifylint) plus:
- exhaustive: handle StateFrozen in the INV-2 switch (the quiescent state)
- testifylint: assert.Equal(float64,...) -> assert.InDelta(...,0) (14)
- revive: rename unused params (buildProductionBoundaries ctx,
  newCaptiveCoreOpener logger) to _
- unconvert: drop redundant uint32(c0First)
- lll: move long t.Parallel() trailing comments above the call; wrap a
  ProcessConfig literal
- nonamedreturns: drop names on snapshotLag; //nolint on the test helpers
  whose named outputs document the returned pair/handles
- //nolint for intentional/forward/complex spots: nilerr (ctx-canceled is
  a clean shutdown), nestif x2, contextcheck (test), unparam
  (newCaptiveCoreOpener stub until #772; test-helper params that vary in
  later slices), gocognit/cyclop (auditDiskMatchesMeta)
TestRunDaemon_StoragePathOverridesHonored now sets one [immutable_storage]
cold override and asserts ledgers resolve under {cold}/ledgers (the cold
tier's fixed subdir). Daemon code is unchanged — ResolvePaths/LockRoots/
NewLayoutFromPaths keep the same API.
The full-history-streaming wiring imports the streaming daemon, which is
Unix-only (flock via x/sys/unix; cgo RocksDB), so the windows-latest build
fails once it is wired in. Document the limitation and the fix direction
(build-constrain the daemon path off Windows) at the wiring point; the fix
itself is a follow-up.
@chowbao

chowbao commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by a new 2-phase stacked series that re-slices this work by phase (backfill → live ingestion + lifecycle), with the MVP scope cuts (recovery / audit / convergence / retention-reconfiguration dropped) and the folded-in fixes (cold+hot ingest service + NewPrometheusSink metrics wiring, exponential withRetries backoff, deletion of the dead RunHot/RunCold stream-drain orchestration):

Each layer builds + go vet + go test -short green; the capstone (#821) also passes the lifecycle E2E. Leaving this open for now — can be closed once the new stack is reviewed.

@chowbao chowbao closed this Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant