Skip to content

fix(converge): reconcile Pangolin resources instead of add-only (#309) - #326

Merged
ChrisonSimtian merged 1 commit into
mainfrom
fix/pangolin-reconcile-resources
Jul 28, 2026
Merged

fix(converge): reconcile Pangolin resources instead of add-only (#309)#326
ChrisonSimtian merged 1 commit into
mainfrom
fix/pangolin-reconcile-resources

Conversation

@ChrisonSimtian

Copy link
Copy Markdown
Collaborator

Closes #309.

The Pangolin resource reconciler found-or-created by fullDomain and, if the resource already
existed, left it entirely alone — target included. Editing a declared target updated desired
state and never reached Pangolin.

That failed silently, which is the real defect. During #303 the Pulse UI moved from CT 4000 to
CT 4001 and stacks/Core was updated; converge reported 0 to create, 0 drifted, 3 up-to-date
while the live target still pointed at 10.10.0.40 — a container that had just been stopped.
pulse.lab.chrison.dev was down and converge said everything was fine.

Now

Drift Call
target ip / port / method / enabled POST /target/{targetId}
the ssl and sso gates POST /resource/{resourceId}

siteId is required on the target update even when unchanged, or it 400s with
expected number, received undefined at "siteId" — as the issue found.

Verified live against CT 2013, end to end

1. No false positives — live in sync:

NOCHANGE: … 15 resource(s) declared, 0 created, 0 retargeted, 0 re-gated

2. Drift injected on traefik.lab's target via the API (80808081), then converge:

APPLIED: … 15 resource(s) declared, 0 created, 1 retargeted, 0 re-gated
  traefik.lab.chrison.dev: target http://localhost:8081 → http://localhost:8080

3. Restored and idempotenttraefik.lab.chrison.dev serves again (302 to Pangolin auth), and
a re-run returns to NOCHANGE.

Scoped with --only pangolin from #324, which is what made testing this against a live stack
reasonable at all.

Three implementation notes

One call covers the common case. GET /org/{org}/resources already embeds ssl, sso and the
targets, so a no-drift run still costs a single call. But that embedded form omits method, so
detail comes from GET /resource/{id}/targets — one extra call per declared-and-existing resource.
Comparing only the embedded fields would have left method drift undetectable, i.e. reproduced this
very bug in a narrower form.

Pangolin returns booleans inconsistently. ssl arrives as a JSON bool, sso as the integer
1
. GetBoolean() throws on the latter, and treating a non-bool as false would have re-gated
all 15 resources on every single run. Both shapes are now read — there's a test pinning it.

Multi-target resources are reported and left alone. Pangolin supports several targets per
resource for load balancing; our shapes only ever declare one, and rewriting the first of several
would silently destroy a hand-built config — which the add-only guardrail in CLAUDE.md exists to
prevent.

Slightly beyond the ask

The issue is about the target; I also reconcile ssl/sso. Same silent-drift bug, and the
data was already fetched — a resource created before sso defaulted ON (#238), or flipped off by
hand in the UI, would otherwise stay publicly open forever. Reported as its own counter (re-gated)
so it's visible rather than folded into the target count.

Also: a read failure on the resource list now fails the member instead of reporting a clean run.
Reporting success when the API couldn't be read is the exact failure mode this issue is about.

Tests

11 new (174 total). In-sync writes nothing; ip/port/method/disabled drift each detected;
siteId present in the update payload; integer sso not mistaken for drift; sso: false honoured
without fighting it every run; live sso: 0 re-gated; multi-target left alone; zero-target gets one
added; unreachable API fails loudly.

Docs: ADR-0007's "add-only, idempotent by fullDomain" line and the matching comment in
stacks/Core/pangolin.lxc.yaml both corrected — they were describing the bug as the design.

Unblocks #314.

The Pangolin resource reconciler found-or-created by fullDomain and, if the
resource already existed, left it entirely alone — target included. Editing a
declared resource's target therefore updated desired state and never reached
Pangolin.

That failed silently, which is the worst part. During the monitoring migration
(#303) the Pulse UI moved from CT 4000 to CT 4001 and stacks/Core was updated to
match; converge then reported "0 to create, 0 drifted, 3 up-to-date" — a
completely clean plan — while the live target still pointed at 10.10.0.40, a
container that had just been stopped. pulse.lab.chrison.dev was down and
converge said everything was fine. It was fixed by hand with a curl.

Now an existing resource is reconciled:

  - target ip / port / method / enabled  → POST /target/{targetId}
  - the ssl and sso gates                → POST /resource/{resourceId}

siteId is required on the target update even when unchanged, or it 400s with
'expected number, received undefined at "siteId"'.

Three things worth knowing about the implementation:

  - GET /org/{org}/resources already embeds ssl, sso and the targets, so the
    common no-drift case still costs a single call. But that embedded target
    form omits `method`, so detail comes from GET /resource/{id}/targets — one
    extra call per declared-and-existing resource. Comparing only the embedded
    fields would have left method drift undetectable, i.e. it would reproduce
    this very bug in a narrower form.

  - Pangolin returns SQLite-backed booleans inconsistently: `ssl` arrives as a
    JSON bool, `sso` as the integer 1. GetBoolean() throws on the latter, and
    treating a non-bool as false would have re-gated all 15 resources on every
    run. Both shapes are now read.

  - A resource with MORE THAN ONE target is reported and left alone. Pangolin
    supports several for load balancing; our shapes only ever declare one, and
    rewriting the first of several would silently destroy a hand-built config,
    which the add-only guardrail in CLAUDE.md exists to prevent.

Reconciling ssl/sso is slightly beyond the issue's ask, but it is the same
silent-drift bug and the data was already fetched: a resource created before sso
defaulted ON (#238), or flipped off by hand in the UI, would otherwise stay
publicly open forever.

A read failure on the resource list now fails the member rather than reporting a
clean run — reporting success when the API could not be read is the exact
failure mode this issue is about.

Verified live against CT 2013 end to end: in-sync run reports 0 changes; port
drift injected on traefik.lab's target via the API is detected and corrected
(http://localhost:8081 -> http://localhost:8080), the hostname serves again, and
re-running returns to NOCHANGE.

11 new tests (174 total).
@ChrisonSimtian
ChrisonSimtian merged commit ba489e3 into main Jul 28, 2026
6 checks passed
@ChrisonSimtian
ChrisonSimtian deleted the fix/pangolin-reconcile-resources branch July 28, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pangolin resource reconciler is add-only — it can't repoint an existing resource's target

1 participant