Skip to content

fix(mq): one tenant's failed queue join ends ingest delivery for every tenant and restarts the process #675

Description

@EricAndrechek

Area: mq / ingest · footgun (multi-tenant blast radius) · found via #612's "Follow-ups"

Expected: a tenant whose queue cannot be opened or joined at runtime is refused on its own; the other tenants keep ingesting.

Actual: when apply opens a tenant's ingest queue and a registered consumer cannot join it, it calls f.fail(fmt.Errorf("tenant %s: %w: join its queue: %w", id, ErrDeliveryEnded, err)) — internal/mq/embedded.go:445-446. For the ingest worker's consumer that fail is the one CreateConsumer installs (embedded.go:703-712), which pushes the error onto c.failed; dispatchLoop reads it as deliveryEnded (internal/ingest/worker.go:262) and fails the worker, so the process stops and restarts. The stream hub's consumer takes the other path and only logs (embedded.go:662-667: "its events reach handler from the next boot").

Impact: adding one tenant folder at runtime whose queue open or join fails takes ingest down for every tenant on the process, and restarts it. The failure is one tenant's; the blast radius is the whole process.

Scope: #612's follow-up names the direction — leave such a queue unrecorded instead, so that tenant's publishes answer 503 and the next publish or reload retries the join. The broker already refuses a queue it has not recorded open (embedded.go:502, :546).

Related: #612, #653, #658, #583


From the merged-PR follow-up sweep (#612 "Follow-ups"); validated by code-read against 4ff50745 on 2026-09-28.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/ingestIngest pipeline (Bento, batching, DLQ)area/tenantTenant id, header resolution, per-tenant settings (internal/tenant)bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions