Skip to content

bug(app): a tenant moved to another ClickHouse database keeps the old schema until its next refresh #638

Description

@EricAndrechek

Area: app · query — bug · found via #610's follow-ups (PR body)

Expected: a tenant whose clickhouse.database or clickhouse.addr changes in a reload reads the new database's tables from the next request on. #610 already computes exactly who moved: Pools.Reconcile returns a stale list of the tenants whose data moved (internal/chconn/chconn.go:445-512).

Actual: that list drives cache invalidation only. wire.go:334-345 walks it and calls Cache.InvalidateTenant on each; nothing asks the tenant's discovery.SchemaRegistry to refresh. The registry keeps serving the previous database's schema until its own loop fires at schema.refresh_interval, or until someone calls POST /v1/ops/schema/refresh?tenant=.

Impact (inferred from the code path; not run): inside that window the tenant's queries and inserts are validated against one database and executed against another. A table that exists only in the new database is rejected as unknown; one that exists only in the old passes validation and then fails at ClickHouse; a column whose type differs is coerced to the old type. The window is a whole schema.refresh_interval long and nothing logs that the schema is behind.

Scope: #610's own follow-up names the fix — refresh the tenants Reconcile reports stale, beside the InvalidateTenant that already runs for them.

Related: #610, #583 (story 6), #391


From #610's "Follow-ups" section (taitelee), no story owns it — story 6 is closed; validated by code-read against 93d80198 on 2026-09-25. Filed by the pm-triage routine.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/appProcess wiring (internal/app): component build, run, releasearea/queryStructured query AST, SQL builderarea/tenantTenant id, header resolution, per-tenant settings (internal/tenant)bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions