Skip to content

fix(ingest): exact WAL backlog and org queue counters, no metering for refused batches - #2440

Merged
Makisuo merged 15 commits into
fix/audit-issue-lifecyclefrom
fix/audit-ingest-wal
Nov 10, 2026
Merged

Makisuo merged 15 commits into
fix/audit-issue-lifecyclefrom
fix/audit-ingest-wal

Merge branch 'fix/audit-issue-lifecycle' into fix/audit-ingest-wal

4434067
Select commit
Loading
Failed to load commit list.
Maple Review Bot / Maple / review succeeded Nov 10, 2026 in 2m 34s

Confidence 10/10 · No issues found

🟢 Confidence 10/10 · safe to merge. Backlog and org-byte arithmetic verified by reading the lane sync/delivery ordering; the Rust test job has not finished at this head.
no findings · tests covered · risk high

Fixes five WAL/metering defects in the ingest gateway plus two encoder mismatches in the Go collector exporter, and drops the unused INGEST_TINYBIRD_CONCURRENCY_PER_SHARD. The counter and billing changes hold up; safe to merge.

  • while_caller_waits subtracts a skipped frame's bytes from committed_bytes, so backlog returns to zero
  • export_and_mark releases org queue bytes before marked?, covering a failed cursor write
  • batch_was_stored skips metering and key use when a relay was refused and nothing was stored
  • read_wal_frame grows the payload buffer instead of allocating the declared length up front

Before merge

  • Infra · Check every environment has what this config now expects · apps/ingest/Dockerfile.prebuilt
  • Infra · Check every environment has what this config now expects · apps/ingest/alchemy.run.ts
  • Manual · Drop INGEST_TINYBIRD_CONCURRENCY_PER_SHARD from the prd and dev deploy configs; nothing reads it at this head
What was checked
  • Skip subtraction matches the add exactly: end - start equals the added accumulated into SyncBatch.bytes (telemetry.rs:1754, :1854)
  • mpsc::channel(0) check sits in validate_for_pipeline, which both write-mode branches call at boot (main.rs:463-467)
  • uses_native_pipeline_for is the same predicate that gates the native accept in process_decoded_payload (main.rs:7955), so the dual-mode predicate cannot over- or under-count

4434067 · Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.