fix(converge): reconcile Pangolin resources instead of add-only (#309) - #326
Merged
Conversation
The Pangolin resource reconciler found-or-created by fullDomain and, if the resource already existed, left it entirely alone — target included. Editing a declared resource's target therefore updated desired state and never reached Pangolin. That failed silently, which is the worst part. During the monitoring migration (#303) the Pulse UI moved from CT 4000 to CT 4001 and stacks/Core was updated to match; converge then reported "0 to create, 0 drifted, 3 up-to-date" — a completely clean plan — while the live target still pointed at 10.10.0.40, a container that had just been stopped. pulse.lab.chrison.dev was down and converge said everything was fine. It was fixed by hand with a curl. Now an existing resource is reconciled: - target ip / port / method / enabled → POST /target/{targetId} - the ssl and sso gates → POST /resource/{resourceId} siteId is required on the target update even when unchanged, or it 400s with 'expected number, received undefined at "siteId"'. Three things worth knowing about the implementation: - GET /org/{org}/resources already embeds ssl, sso and the targets, so the common no-drift case still costs a single call. But that embedded target form omits `method`, so detail comes from GET /resource/{id}/targets — one extra call per declared-and-existing resource. Comparing only the embedded fields would have left method drift undetectable, i.e. it would reproduce this very bug in a narrower form. - Pangolin returns SQLite-backed booleans inconsistently: `ssl` arrives as a JSON bool, `sso` as the integer 1. GetBoolean() throws on the latter, and treating a non-bool as false would have re-gated all 15 resources on every run. Both shapes are now read. - A resource with MORE THAN ONE target is reported and left alone. Pangolin supports several for load balancing; our shapes only ever declare one, and rewriting the first of several would silently destroy a hand-built config, which the add-only guardrail in CLAUDE.md exists to prevent. Reconciling ssl/sso is slightly beyond the issue's ask, but it is the same silent-drift bug and the data was already fetched: a resource created before sso defaulted ON (#238), or flipped off by hand in the UI, would otherwise stay publicly open forever. A read failure on the resource list now fails the member rather than reporting a clean run — reporting success when the API could not be read is the exact failure mode this issue is about. Verified live against CT 2013 end to end: in-sync run reports 0 changes; port drift injected on traefik.lab's target via the API is detected and corrected (http://localhost:8081 -> http://localhost:8080), the hostname serves again, and re-running returns to NOCHANGE. 11 new tests (174 total).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #309.
The Pangolin resource reconciler found-or-created by
fullDomainand, if the resource alreadyexisted, left it entirely alone — target included. Editing a declared target updated desired
state and never reached Pangolin.
That failed silently, which is the real defect. During #303 the Pulse UI moved from CT 4000 to
CT 4001 and
stacks/Corewas updated; converge reported0 to create, 0 drifted, 3 up-to-datewhile the live target still pointed at
10.10.0.40— a container that had just been stopped.pulse.lab.chrison.devwas down and converge said everything was fine.Now
ip/port/method/enabledPOST /target/{targetId}sslandssogatesPOST /resource/{resourceId}siteIdis required on the target update even when unchanged, or it 400s withexpected number, received undefined at "siteId"— as the issue found.Verified live against CT 2013, end to end
1. No false positives — live in sync:
2. Drift injected on
traefik.lab's target via the API (8080→8081), then converge:3. Restored and idempotent —
traefik.lab.chrison.devserves again (302 to Pangolin auth), anda re-run returns to
NOCHANGE.Scoped with
--only pangolinfrom #324, which is what made testing this against a live stackreasonable at all.
Three implementation notes
One call covers the common case.
GET /org/{org}/resourcesalready embedsssl,ssoand thetargets, so a no-drift run still costs a single call. But that embedded form omits
method, sodetail comes from
GET /resource/{id}/targets— one extra call per declared-and-existing resource.Comparing only the embedded fields would have left method drift undetectable, i.e. reproduced this
very bug in a narrower form.
Pangolin returns booleans inconsistently.
sslarrives as a JSON bool,ssoas the integer1.GetBoolean()throws on the latter, and treating a non-bool asfalsewould have re-gatedall 15 resources on every single run. Both shapes are now read — there's a test pinning it.
Multi-target resources are reported and left alone. Pangolin supports several targets per
resource for load balancing; our shapes only ever declare one, and rewriting the first of several
would silently destroy a hand-built config — which the add-only guardrail in
CLAUDE.mdexists toprevent.
Slightly beyond the ask
The issue is about the target; I also reconcile
ssl/sso. Same silent-drift bug, and thedata was already fetched — a resource created before
ssodefaulted ON (#238), or flipped off byhand in the UI, would otherwise stay publicly open forever. Reported as its own counter (
re-gated)so it's visible rather than folded into the target count.
Also: a read failure on the resource list now fails the member instead of reporting a clean run.
Reporting success when the API couldn't be read is the exact failure mode this issue is about.
Tests
11 new (174 total). In-sync writes nothing; ip/port/method/disabled drift each detected;
siteIdpresent in the update payload; integerssonot mistaken for drift;sso: falsehonouredwithout fighting it every run; live
sso: 0re-gated; multi-target left alone; zero-target gets oneadded; unreachable API fails loudly.
Docs: ADR-0007's "add-only, idempotent by
fullDomain" line and the matching comment instacks/Core/pangolin.lxc.yamlboth corrected — they were describing the bug as the design.Unblocks #314.