Skip to content

Streaming daemon: Slice 1 — Layer 3 (orchestration)#807

Closed
chowbao wants to merge 4 commits into
slice1bfrom
slice1c
Closed

Streaming daemon: Slice 1 — Layer 3 (orchestration)#807
chowbao wants to merge 4 commits into
slice1bfrom
slice1c

Conversation

@chowbao

@chowbao chowbao commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Slice 1, Layer 3 of 4 — Orchestration. Stacked on Layer 2 (storage). Diff is the orchestration brain.

Driving production:

  • derived progress (resume recomputed from durable state, never stored);
  • the postcondition resolver + executor (pure catalog diff → bounded worker pool, success-signalling done-channels);
  • the hot-DB ingestion loop (captive-core → hot DB, boundary handoff);
  • the lifecycle tick (plan → discard → prune) + the retention-floor arithmetic and observability sink.

Compiles + -short-green on Layers 1–2.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

chowbao added 4 commits June 23, 2026 23:35
doc.go: add the Storage group (process.go, hotsource.go) to the file map
and switch the map header to 'by concern' now that the package spans two
layers; shrink the 'Later layers' note accordingly.

golangci-lint (this layer's own new findings):
- hotchunk.DB.MaxCommittedSeq: drop named returns (nonamedreturns)
- process.go / ingest driver.go: wrap two >120-char lines (lll)
doc.go: add Planner (resolve/execute/eligibility), Ingestion (ingest),
Orchestration (progress/lifecycle/retention), and Observability groups;
update the coverage line and 'Later layers' note.

golangci-lint (this layer's own new findings):
- misspell: cancelled -> canceled (comments across ingest/lifecycle +
  their tests)
- modernize: if-guard -> max() in IngestionLag and lastCommittedLedger
- staticcheck QF1008: cfg.ExecConfig.WithDefaults() -> cfg.WithDefaults()
- revive: rename unused refineWithHotDB(cat) parameter to _
- errname: assertErr/errStr -> errSyntheticOp/stringError (test)
- lll: wrap/shorten 4 long lines
- godoclint: Metrics doc starts with the symbol name
- drop two dead //nolint:gosec directives; //nolint:gosec on uint32(floor)
  (now needed after the max() rewrite)
- //nolint for forward-/test-only or inherently-complex symbols:
  runIngestionLoop metrics (unparam), seqWithinRetention (unused),
  Metrics (interfacebloat), runLifecycleTick (gocognit/gocyclo/cyclop/
  funlen), the plan-and-execute guard (nestif)
@chowbao

chowbao commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by a new 2-phase stacked series that re-slices this work by phase (backfill → live ingestion + lifecycle), with the MVP scope cuts (recovery / audit / convergence / retention-reconfiguration dropped) and the folded-in fixes (cold+hot ingest service + NewPrometheusSink metrics wiring, exponential withRetries backoff, deletion of the dead RunHot/RunCold stream-drain orchestration):

Each layer builds + go vet + go test -short green; the capstone (#821) also passes the lifecycle E2E. Leaving this open for now — can be closed once the new stack is reviewed.

@chowbao chowbao closed this Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant