Retire unsound NQE removal; consolidate mutations behind one guarded gateway

.github/RELEASE_NOTES_TEMPLATE.md

@@ -1,5 +1,79 @@
 ## aws-sync {{VERSION}}
 
+### Breaking changes
+
+Read this section before upgrading. These changes fail closed, so an automated
+deployment that does not act on them will stop working rather than degrade.
+
+**1. NQE-based account removal has been removed.**
+
+`--prune-missing` remains a recognized option so existing automation receives
+an actionable error, but it always refuses before credentials, NQE, planning,
+or PATCH work. NQE is observed snapshot inventory, not an authoritative account
+manifest, so an absent row cannot prove an account should be deleted. Replace
+NQE prune workflows with `sync-accounts` and a complete human-reviewed manifest.
+Manifest removals still require `--allow-removals` and both nonzero removal
+ceilings.
+
+**2. `serve-webhook --apply` now requires authentication and an explicit network.**
+
+The server previously accepted unauthenticated requests when no webhook
+credentials were configured, and let an event select any network or setup. It
+now refuses to start in apply mode without all three of:
+
+```bash
+awssync serve-webhook --apply --yes \
+  --webhook-basic-username <USER> \
+  --webhook-basic-password <PASSWORD> \
+  --network-id <NETWORK_ID>
+```
+
+Configure Forward to send matching credentials (`awssync configure-webhook`).
+Event scope is now intersected with configured scope: an event naming a
+different network, or a setup outside `--setup-id`, is rejected with `403`
+instead of being honored.
+
+The server also persists dedupe and snapshot-ordering state to
+`$UserConfigDir/awssync/webhook-state.json`. Ensure the service user can write
+that directory, or set `--webhook-state-file`.
+
+**3. Destructive applies in unattended contexts now require an explicit flag.**
+
+Forward's API exposes no compare-and-swap token, so a concurrent edit in the UI
+cannot be detected before a full-list PATCH overwrites it. Removals and disables
+requested without a human present now require `--allow-unattended-destructive`:
+
+```bash
+awssync sync-accounts --apply --yes --allow-unattended-destructive ...
+awssync apply-plan --allow-unattended-destructive ...   # when removing/disabling
+awssync serve-webhook --apply --yes --allow-unattended-destructive ...
+```
+
+`--yes` counts as unattended even in a terminal. The flag does not bypass
+`--allow-removals`, evidence rules, or either removal ceiling — it is an
+additional acknowledgement, not a replacement. `safe-sync` is unaffected,
+being additive-only. Non-destructive applies are unaffected.
+
+### Safety changes
+
+- NQE reconciliation is unconditionally additive. Pagination completeness checks remain to diagnose truncated observed data, but completeness no longer authorizes absence-based deletion.
+- A malformed account ID now fails the plan instead of being silently skipped, since skipping rows is how a partial inventory becomes a deletion. Use `--allow-malformed-rows` to skip and report them; doing so marks the inventory incomplete and therefore blocks removals.
+- Setting an account to `enabled: false` is now classified as destructive. It consumes the same authorization and removal ceilings as deletion, closing a path where `apply-plan` could disable every account in a setup without tripping any removal guard.
+- All account-list writes go through one guarded apply path, enforced by a test that fails if any other caller appears.
+- External ID rotation now writes a pre-change rollback artifact, re-reads before PATCH, and binds confirmation to the computed payload.
+- A partial multi-setup apply reports per-setup disposition (applied, pending, conflicted, failed) and a result-journal path instead of a bare error.
+- Planning is deterministic: preview and apply produce identical digests for identical inputs.
+- Cross-setup account moves are refused. Sequential per-setup PATCHes cannot guarantee an account ends up in exactly one setup if the run fails midway.
+
+### Known limitation
+
+Forward's cloud-account API provides no ETag, version field, or other
+compare-and-swap token. A concurrent edit made in the Forward UI between this
+tool's final read and its PATCH will be overwritten, and this is deterministic
+rather than a narrow race. The pre-PATCH re-read narrows the window but does not
+close it. Prefer `safe-sync` for routine work, and avoid unattended destructive
+runs on setups that people also edit by hand.
+
 ### Highlights
 
 - New `awssync safe-sync` command provides a one-command routine workflow: 24-hour snapshot freshness, preflight, compact preview, additive-only enforcement, one confirmation, rollback, and apply.
@@ -8,12 +82,12 @@
 - The README is now novice-first, with the routine workflow, count definitions, expected output, common stop conditions, and a short decision diagram before expert features.
 - A one-page routine operator handoff is available at `docs/routine-safe-sync.md`.
 - NQE reconciliation is additive by default: configured accounts missing from the current NQE result remain in the setup, while discovered disabled accounts are re-enabled.
-- NQE-based deletion now requires `--prune-missing`, `--allow-removals`, and both nonzero `--max-removals` and `--max-removal-percent` bounds.
-- Every apply writes a complete pre-change `.rollback.json` payload and verifies that the selected setup state has not changed before the first PATCH.
+- NQE-based deletion is retired; `--prune-missing` returns an actionable refusal and reviewed manifest removal remains available through `sync-accounts`.
+- Every apply writes a pre-change `.rollback.json` PATCH payload containing the account list and PATCHable setup fields, not a full setup backup, and verifies that the selected setup state has not changed before the first PATCH.
 - CLI runs pin the latest processed snapshot so planning and apply use one immutable NQE inventory.
 - Invalid NQE account-ID placeholders are ignored and reported instead of becoming AWS accounts.
 - Human-readable output is now the default; use `--json` or `--format json` for automation.
-- Regression coverage includes 0, 1, 10, half, and all-enabled account states; additive and explicit-prune paths; multi-setup isolation; concurrent setup changes; rollback; and snapshot pinning.
+- Regression coverage includes 0, 1, 10, half, and all-enabled account states; additive NQE and authoritative-manifest paths; multi-setup isolation; concurrent setup changes; rollback; and snapshot pinning.
 - Per-account External ID selection and CSV workflows from v2.3.0 remain supported.
 - Release assets remain available for Linux and macOS on amd64 and arm64 with SHA-256 checksums and GitHub build-provenance attestations.
 

.github/workflows/ci.yml

@@ -25,14 +25,45 @@ jobs:
         with:
           go-version-file: go.mod
           cache: true
-      - name: Check formatting
-        run: make fmt-check
+      - name: Check Go formatting
+        shell: bash
+        run: |
+          unformatted="$(gofmt -l .)"
+          if [[ -n "${unformatted}" ]]; then
+            echo "The following Go files are not gofmt-formatted:" >&2
+            printf '%s\n' "${unformatted}" >&2
+            exit 1
+          fi
       - name: Vet
-        run: make vet
+        run: go vet ./...
+      - name: Guard phase 0 failure-test switches
+        shell: bash
+        run: |
+          guards=(
+            "internal/api/architecture_failure_test.go:runP0APIFailureTests"
+            "internal/app/architecture_failure_test.go:runP0ArchitectureFailureTests"
+            "internal/webhook/architecture_failure_test.go:runP0WebhookFailureTests"
+          )
+
+          status=0
+          for entry in "${guards[@]}"; do
+            file="${entry%%:*}"
+            guard="${entry#*:}"
+            expected="^[[:space:]]*const[[:space:]]+${guard}([[:space:]]+bool)?[[:space:]]*=[[:space:]]*false[[:space:]]*(//.*)?$"
+            if ! grep -Eq "${expected}" "${file}"; then
+              actual="$(grep -En "^[[:space:]]*const[[:space:]]+${guard}([[:space:]]+bool)?[[:space:]]*=" "${file}" || true)"
+              printf '::error file=%s::%s must remain false; found: %s\n' \
+                "${file}" "${guard}" "${actual:-<missing>}"
+              status=1
+            fi
+          done
+          exit "${status}"
       - name: Test
         run: make test
-      - name: Race detector
-        run: make race
+      - name: Race detector (full suite)
+        run: go test -race ./...
+      - name: Race detector (webhook, 10 runs)
+        run: go test -race ./internal/webhook/ -count=10
       - name: Vulnerability scan
         run: make vuln
       - name: Build

README.md

@@ -4,6 +4,8 @@
 
 Most operators should use `safe-sync`. It runs the safety checks, shows a short preview, and asks before changing Forward. It can add or re-enable accounts, but it cannot remove them.
 
+Upgrading an existing deployment? Read [Upgrading `awssync`](docs/upgrading.md) before replacing the binary; this release intentionally breaks retired prune automation, applying webhook receivers without fixed authentication and network scope, and unattended destructive applies without an additional acknowledgement.
+
 ## Routine Safe Sync
 
 ### 1. Download and verify
@@ -59,6 +61,7 @@ For two AWS setups:
 6. Otherwise, prompts for the word `apply`.
 7. Confirms that the reviewed payload has not changed.
 8. Writes a rollback file before PATCHing Forward.
+9. Updates a durable per-setup result journal as the apply proceeds.
 
 Example preview:
 
@@ -99,7 +102,7 @@ flowchart TD
     D -->|Yes| F[Forward Terraform provider]
     D -->|No or incomplete GovCloud inventory| G[Reviewed account manifest]
     B --> H[Preflight, preview, confirm, rollback, apply]
-    C --> I[Verify lifecycle outside Forward, then use explicit removal guards]
+    C --> I[Review a complete manifest, then use sync-accounts]
 ```
 
 Use `safe-sync` for ordinary account additions and unchecked accounts. The remaining commands are expert workflows:
@@ -108,7 +111,7 @@ Use `safe-sync` for ordinary account additions and unchecked accounts. The remai
 | --- | --- |
 | Routine existing-setup sync | `safe-sync` |
 | Scheduled or JSON automation | Standard `awssync` command |
-| Independently verified account removal | Standard command with the reviewed removal workflow |
+| Independently verified account removal | `sync-accounts` with a complete reviewed manifest |
 | New commercial AWS Organization | Forward Terraform provider; `discover-org` is the manual fallback |
 | No Organizations access | `onboard-accounts` or `sync-accounts` with a complete manifest |
 | GovCloud | [GovCloud workflow](docs/govcloud-workflow.md) |
@@ -131,23 +134,15 @@ Fix the reported condition and run the same command again. Do not add removal ov
 
 ## Account Removal Is a Separate Expert Workflow
 
-`safe-sync` has no removal switches. Removing an account requires an operator to confirm outside Forward that the AWS account was closed, retired, or removed from the intended Organization.
-
-The standard NQE workflow requires all of the following before a removal can be applied:
+`safe-sync` and the standard NQE workflow cannot remove accounts. NQE reports observed snapshot inventory, which combines successfully collected accounts with accounts visible through Organizations metadata; absence is not proof of deletion. The recognized `--prune-missing` flag now fails with an explanation instead of producing a plan.
 
-- `--prune-missing`
-- `--allow-removals`
-- a nonzero `--max-removals`
-- a nonzero `--max-removal-percent`
-- additional Organizations-evidence overrides when applicable
+Use `sync-accounts` with a complete, human-reviewed manifest for lifecycle removals. Applying a manifest removal requires `--allow-removals` plus nonzero `--max-removals` and `--max-removal-percent` ceilings. A destructive run using `--yes`, CI, or another unattended context also requires `--allow-unattended-destructive`. Never remove an account only because its collection fails.
 
-Prefer `sync-accounts` with a complete authoritative manifest for lifecycle removals. Never remove an account only because its collection fails.
-
-See [AWS account sync procedure](docs/aws-account-sync-procedure.md#apply-the-sync) for the reviewed removal commands and rollback procedure.
+See [AWS account sync procedure](docs/aws-account-sync-procedure.md#reviewed-manifest-removal) for the reviewed removal commands and rollback procedure.
 
 ## Automation
 
-For scheduled additive-only operation, use the standard command without any prune or removal flags:
+For scheduled additive-only operation, use the standard command without removal flags:
 
 ```bash
 ./awssync-linux-amd64 \
@@ -158,17 +153,19 @@ For scheduled additive-only operation, use the standard command without any prun
   --apply --yes --json
 ```
 
-The standard command is additive by default, pins one processed snapshot, writes the payload before PATCH, verifies current setup state, and writes `<output>.rollback.json`.
+The standard command is additive by default, pins one processed snapshot, writes the payload before PATCH, verifies current setup state, and writes `<output>.rollback.json`. Every apply also maintains `<output>.result.json`, whose per-setup status distinguishes applied, conflicted, and failed work after a partial or ambiguous run.
 
 For event-driven operation, `serve-webhook` accepts Forward `SNAPSHOT_READY` events and serializes jobs through a bounded queue.
 
+An applying receiver requires `--yes`, an explicit `--network-id`, and inbound Basic Auth credentials. Configure Forward to send the same credentials, and keep the receiver's durable state file on service-owned storage. Failed events are attempted at most five times and then remain dead-lettered for operator recovery.
+
 Do not pass Forward or AWS secrets on command lines in shared process environments. Use protected environment injection or a service-manager secret facility.
 
 ## External IDs, Onboarding, and GovCloud
 
 These are separate from routine synchronization:
 
-- [External ID procedure](docs/aws-account-sync-procedure.md#customer-defined-external-id-with-an-iam-user)
+- [External ID procedure](docs/aws-account-sync-procedure.md#add-a-customer-defined-external-id-to-an-existing-setup)
 - [New AWS Organizations onboarding](docs/aws-account-sync-procedure.md#onboard-from-aws-organizations-directly)
 - [Account-manifest workflow](docs/architecture-flow.md)
 - [AWS GovCloud workflow](docs/govcloud-workflow.md)
@@ -178,19 +175,23 @@ Existing per-account External IDs are preserved during ordinary synchronization.
 ## Safety Guarantees
 
 - Routine NQE synchronization is additive; accounts missing from NQE remain configured.
+- NQE-derived plans cannot select `CompleteInventory` removal semantics; `--prune-missing` is retained only to return an actionable refusal.
 - `safe-sync` cannot remove accounts.
 - Human-readable output is the default; `--json` is for standard-command automation.
 - The latest processed snapshot is pinned before planning.
-- Invalid NQE account-ID placeholders are ignored and reported.
-- Every apply writes a complete pre-change rollback payload.
+- Malformed NQE account IDs fail by default; `--allow-malformed-rows` skips and reports them only for incomplete additive runs.
+- Every apply writes a pre-change rollback payload containing the complete `assumeRoleInfos` account list and the PATCHable setup fields (`type`, `name`, `regions`, `regionToProxyServerId`, and `proxyServerId`). It does not capture `collect`, `connectionTimeoutSeconds`, `requestTimeoutSeconds`, `numVirtualizedDevices`, or `useForwardAccountToAssumeRole`. Forward PATCH leaves absent top-level fields unchanged, so the artifact safely restores the fields `awssync` changes without overwriting those settings; it is not a full setup backup or a setup-creation payload.
+- Every apply writes a durable per-setup result journal.
 - The reviewed target payload and current Forward setup are revalidated before PATCH.
+- Forward exposes no atomic compare-and-swap token; unattended destructive applies require a separate explicit acknowledgement.
 - Generated payloads use atomic owner-only `0600` files.
 - Idempotent reads and full-state updates use bounded transient retries.
 
 ## Documentation
 
 | Guide | Use it for |
 | --- | --- |
+| [Upgrade guide](docs/upgrading.md) | Breaking changes and migration steps for existing automation |
 | [Routine safe sync](docs/routine-safe-sync.md) | One-page operator handoff |
 | [Quick start](docs/quick-start.md) | Standard CLI examples and troubleshooting |
 | [AWS account sync procedure](docs/aws-account-sync-procedure.md) | IAM prerequisites, automation, removals, and rollback |