Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/RELEASE_NOTES_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

### Highlights

- Added `--max-removals` and `--max-removal-percent` blast-radius ceilings across NQE sync, manifest sync, saved-plan apply, preflight, and webhook workflows.
- Added release installation, checksum, provenance verification, automation audit handling, and explicit External ID rollback guidance.
- Fixed the release checksum manifest so downloaded assets verify directly with `sha256sum -c sha256sums.txt`.
- Added reversible one-time External ID migration for existing AWS setups with `external-id --value` and `external-id --clear`.
- Added AWS GovCloud workflows for both regular Forward Organizations/NQE discovery and reviewed standalone-account manifests.
Expand Down
38 changes: 37 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ An existing setup can add, replace, or clear its per-account External ID without
--yes
```

The value is written to every existing `assumeRoleInfos` entry for that setup. Review and apply the Forward payload first, test a representative account, and then update the target-role trust policies to require the identical value. After the migration PATCH, normal syncs preserve the stored External ID without rerunning this command. Use `external-id --clear` for an intentional rollback to null. Stored IAM access keys and secrets are not included in or changed by the PATCH.
The value is written to every existing `assumeRoleInfos` entry for that setup. Review and apply the Forward payload first, test a representative account, and then update the target-role trust policies to require the identical value. After the migration PATCH, normal syncs preserve the stored External ID without rerunning this command. For rollback, relax or remove the mandatory `sts:ExternalId` trust-policy condition first, confirm the role can still be assumed, and only then apply `external-id --clear`. Stored IAM access keys and secrets are not included in or changed by the PATCH.

## Procedure

Expand All @@ -63,6 +63,30 @@ For GovCloud Organizations and standalone-account workflows, including collector
make build
```

## Install a Release

Prefer the tarball because it preserves the executable bit. Download the tarball and checksum manifest for the required platform, verify both the checksum and GitHub build provenance, then extract it:

```bash
VERSION=v2.1.2
PLATFORM=linux-amd64

gh release download "$VERSION" \
--repo forwardnetworks/aws-sync \
--pattern "awssync-${PLATFORM}.tar.gz" \
--pattern sha256sums.txt

grep " awssync-${PLATFORM}.tar.gz$" sha256sums.txt | sha256sum -c -
gh attestation verify "awssync-${PLATFORM}.tar.gz" \
--repo forwardnetworks/aws-sync \
--signer-workflow forwardnetworks/aws-sync/.github/workflows/release.yml

tar -xzf "awssync-${PLATFORM}.tar.gz"
./"awssync-${PLATFORM}" --help
```

On macOS, use `PLATFORM=darwin-amd64` or `PLATFORM=darwin-arm64` and replace `sha256sum -c -` with `shasum -a 256 -c -`. A raw binary downloaded directly from GitHub may need `chmod +x`; the tarball does not.

## Usage

Set common inputs through environment variables:
Expand Down Expand Up @@ -160,10 +184,22 @@ Type 'apply' to continue:
```

If the plan removes accounts from a Forward setup, `--apply` fails unless `--allow-removals` is also provided.
Use `--max-removals` to cap the aggregate removal count across all selected setups and `--max-removal-percent` to cap each setup independently. Both are optional, apply-time safety ceilings; a value of `0` disables that limit.
If removals are included and no uncollected candidate rows are visible, add `--allow-no-candidates` only after confirming AWS Organizations discovery.
If removals are included and there is no candidate or Organizational Unit signal, add `--allow-no-org-evidence` only after independent discovery verification.
In a run with multiple `--setup-id` values, this is enforced per setup and the check output includes the setup IDs that are missing signals.

For example, an approved removal run can still be limited to no more than 10 accounts overall and no more than 5% of any setup:

```bash
./bin/awssync \
--apply \
--yes \
--allow-removals \
--max-removals 10 \
--max-removal-percent 5
```

Apply a reviewed payload file without recomputing the plan:

```bash
Expand Down
86 changes: 64 additions & 22 deletions cmd/awssync/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ func newRootCommand() *cobra.Command {
Insecure: v.GetBool("insecure"),
Timeout: v.GetDuration("timeout"),
AllowRemovals: flagBool(cmd, v, "allow-removals"),
MaxRemovals: flagInt(cmd, v, "max-removals"),
MaxRemovalPercent: flagFloat64(cmd, v, "max-removal-percent"),
AllowNoCandidates: flagBool(cmd, v, "allow-no-candidates"),
AllowNoOrgEvidence: flagBool(cmd, v, "allow-no-org-evidence"),
MaxSnapshotAge: flagDuration(cmd, v, "max-snapshot-age"),
Expand Down Expand Up @@ -112,6 +114,8 @@ func newRootCommand() *cobra.Command {
Timeout: v.GetDuration("timeout"),
Apply: apply,
AllowRemovals: flagBool(cmd, v, "allow-removals"),
MaxRemovals: flagInt(cmd, v, "max-removals"),
MaxRemovalPercent: flagFloat64(cmd, v, "max-removal-percent"),
AllowNoCandidates: flagBool(cmd, v, "allow-no-candidates"),
AllowNoOrgEvidence: flagBool(cmd, v, "allow-no-org-evidence"),
MaxSnapshotAge: flagDuration(cmd, v, "max-snapshot-age"),
Expand Down Expand Up @@ -230,12 +234,16 @@ func bindPreflightFlags(v *viper.Viper, flags *pflag.FlagSet) {
flags.String("query-setup-param", "", "optional saved-query String parameter name to receive the single selected --setup-id")
flags.StringSlice("setup-id", nil, "optional Forward AWS setup ID to sync; repeatable")
flags.Bool("allow-no-org-evidence", false, "allow removals when no AWS Organizations evidence is visible in NQE")
flags.Int("max-removals", 0, "maximum aggregate account removals allowed during apply; 0 disables the limit")
flags.Float64("max-removal-percent", 0, "maximum removal percentage allowed per setup during apply; 0 disables the limit")
flags.Duration("max-snapshot-age", 0, "fail if latest processed snapshot is older than this duration; 0 disables the check")
mustBind(v, flags, "snapshot-id")
mustBind(v, flags, "query-id")
mustBind(v, flags, "query-setup-param")
mustBind(v, flags, "setup-id")
mustBind(v, flags, "allow-no-org-evidence")
mustBind(v, flags, "max-removals")
mustBind(v, flags, "max-removal-percent")
mustBind(v, flags, "max-snapshot-age")
}

Expand All @@ -248,6 +256,8 @@ func bindProcessingFlags(v *viper.Viper, flags *pflag.FlagSet) {
flags.Bool("apply", false, "PATCH the generated setup payloads back into Forward")
flags.Bool("yes", false, "skip apply confirmation prompt")
flags.Bool("allow-removals", false, "allow planned account removals during apply")
flags.Int("max-removals", 0, "maximum aggregate account removals allowed during apply; 0 disables the limit")
flags.Float64("max-removal-percent", 0, "maximum removal percentage allowed per setup during apply; 0 disables the limit")
flags.Bool("allow-no-candidates", false, "allow removals when no uncollected candidate accounts are visible")
flags.Bool("allow-no-org-evidence", false, "allow removals when no AWS Organizations evidence is visible in NQE")
flags.Duration("max-snapshot-age", 0, "fail if latest processed snapshot is older than this duration; 0 disables the check")
Expand All @@ -259,6 +269,8 @@ func bindProcessingFlags(v *viper.Viper, flags *pflag.FlagSet) {
mustBind(v, flags, "apply")
mustBind(v, flags, "yes")
mustBind(v, flags, "allow-removals")
mustBind(v, flags, "max-removals")
mustBind(v, flags, "max-removal-percent")
mustBind(v, flags, "allow-no-candidates")
mustBind(v, flags, "allow-no-org-evidence")
mustBind(v, flags, "max-snapshot-age")
Expand Down Expand Up @@ -296,6 +308,8 @@ func newPreflightCommand(v *viper.Viper) *cobra.Command {
Insecure: v.GetBool("insecure"),
Timeout: v.GetDuration("timeout"),
AllowNoOrgEvidence: flagBool(cmd, v, "allow-no-org-evidence"),
MaxRemovals: flagInt(cmd, v, "max-removals"),
MaxRemovalPercent: flagFloat64(cmd, v, "max-removal-percent"),
MaxSnapshotAge: flagDuration(cmd, v, "max-snapshot-age"),
})
if err != nil {
Expand Down Expand Up @@ -361,15 +375,17 @@ func newApplyPlanCommand(v *viper.Viper) *cobra.Command {
return err
}
summary, err := app.ApplyPlan(cmd.Context(), app.ApplyPlanConfig{
Host: v.GetString("host"),
Username: v.GetString("username"),
Password: password,
NetworkID: networkID,
PlanPath: flagString(cmd, v, "plan"),
APIPrefix: v.GetString("api-prefix"),
Insecure: v.GetBool("insecure"),
Timeout: v.GetDuration("timeout"),
AllowRemovals: flagBool(cmd, v, "allow-removals"),
Host: v.GetString("host"),
Username: v.GetString("username"),
Password: password,
NetworkID: networkID,
PlanPath: flagString(cmd, v, "plan"),
APIPrefix: v.GetString("api-prefix"),
Insecure: v.GetBool("insecure"),
Timeout: v.GetDuration("timeout"),
AllowRemovals: flagBool(cmd, v, "allow-removals"),
MaxRemovals: flagInt(cmd, v, "max-removals"),
MaxRemovalPercent: flagFloat64(cmd, v, "max-removal-percent"),
})
if err != nil {
return err
Expand All @@ -381,9 +397,13 @@ func newApplyPlanCommand(v *viper.Viper) *cobra.Command {
cmd.Flags().String("plan", "aws_sync_payload.json", "reviewed payload file to apply")
cmd.Flags().Bool("yes", false, "confirm applying the reviewed payload file")
cmd.Flags().Bool("allow-removals", false, "allow reviewed commercial-partition account removals; GovCloud removals must use their source workflow")
cmd.Flags().Int("max-removals", 0, "maximum aggregate account removals allowed; 0 disables the limit")
cmd.Flags().Float64("max-removal-percent", 0, "maximum removal percentage allowed per setup; 0 disables the limit")
mustBind(v, cmd.Flags(), "plan")
mustBind(v, cmd.Flags(), "yes")
mustBind(v, cmd.Flags(), "allow-removals")
mustBind(v, cmd.Flags(), "max-removals")
mustBind(v, cmd.Flags(), "max-removal-percent")
return cmd
}

Expand Down Expand Up @@ -656,18 +676,20 @@ func newSyncAccountsCommand(v *viper.Viper) *cobra.Command {
return err
}
summary, err := app.SyncAWSAccountManifest(cmd.Context(), app.Config{
Host: v.GetString("host"),
Username: v.GetString("username"),
Password: password,
NetworkID: networkID,
SetupIDs: setupIDs,
Output: flagString(cmd, v, "output"),
ManualOutput: flagString(cmd, v, "manual-output"),
APIPrefix: v.GetString("api-prefix"),
Insecure: v.GetBool("insecure"),
Timeout: v.GetDuration("timeout"),
Apply: apply,
AllowRemovals: flagBool(cmd, v, "allow-removals"),
Host: v.GetString("host"),
Username: v.GetString("username"),
Password: password,
NetworkID: networkID,
SetupIDs: setupIDs,
Output: flagString(cmd, v, "output"),
ManualOutput: flagString(cmd, v, "manual-output"),
APIPrefix: v.GetString("api-prefix"),
Insecure: v.GetBool("insecure"),
Timeout: v.GetDuration("timeout"),
Apply: apply,
AllowRemovals: flagBool(cmd, v, "allow-removals"),
MaxRemovals: flagInt(cmd, v, "max-removals"),
MaxRemovalPercent: flagFloat64(cmd, v, "max-removal-percent"),
}, accounts)
if err != nil {
return err
Expand All @@ -683,7 +705,9 @@ func newSyncAccountsCommand(v *viper.Viper) *cobra.Command {
cmd.Flags().Bool("apply", false, "PATCH the generated setup payload into Forward")
cmd.Flags().Bool("yes", false, "skip apply confirmation prompt")
cmd.Flags().Bool("allow-removals", false, "allow reviewed manifest entries to remove accounts from the setup")
for _, name := range []string{"accounts-file", "setup-id", "output", "manual-output", "apply", "yes", "allow-removals"} {
cmd.Flags().Int("max-removals", 0, "maximum aggregate account removals allowed; 0 disables the limit")
cmd.Flags().Float64("max-removal-percent", 0, "maximum removal percentage allowed for the setup; 0 disables the limit")
for _, name := range []string{"accounts-file", "setup-id", "output", "manual-output", "apply", "yes", "allow-removals", "max-removals", "max-removal-percent"} {
mustBind(v, cmd.Flags(), name)
}
return cmd
Expand Down Expand Up @@ -722,6 +746,8 @@ func newServeWebhookCommand(v *viper.Viper) *cobra.Command {
Timeout: v.GetDuration("timeout"),
Apply: flagBool(cmd, v, "apply"),
AllowRemovals: flagBool(cmd, v, "allow-removals"),
MaxRemovals: flagInt(cmd, v, "max-removals"),
MaxRemovalPercent: flagFloat64(cmd, v, "max-removal-percent"),
AllowNoCandidates: flagBool(cmd, v, "allow-no-candidates"),
AllowNoOrgEvidence: flagBool(cmd, v, "allow-no-org-evidence"),
MaxSnapshotAge: flagDuration(cmd, v, "max-snapshot-age"),
Expand Down Expand Up @@ -1043,6 +1069,22 @@ func flagBool(cmd *cobra.Command, v *viper.Viper, name string) bool {
return v.GetBool(name)
}

func flagInt(cmd *cobra.Command, v *viper.Viper, name string) int {
if flag := cmd.Flags().Lookup(name); flag != nil && flag.Changed {
value, _ := cmd.Flags().GetInt(name)
return value
}
return v.GetInt(name)
}

func flagFloat64(cmd *cobra.Command, v *viper.Viper, name string) float64 {
if flag := cmd.Flags().Lookup(name); flag != nil && flag.Changed {
value, _ := cmd.Flags().GetFloat64(name)
return value
}
return v.GetFloat64(name)
}

func flagDuration(cmd *cobra.Command, v *viper.Viper, name string) time.Duration {
if flag := cmd.Flags().Lookup(name); flag != nil && flag.Changed {
value, _ := cmd.Flags().GetDuration(name)
Expand Down
19 changes: 13 additions & 6 deletions docs/architecture-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ flowchart TD
removals{"Plan contains removals?"}
review["Review exact account IDs"]
approve["Explicit --allow-removals\nall removal paths"]
blast{"Within --max-removals\nand --max-removal-percent?"}
apply["PATCH Forward setup"]
block["BLOCK\nno empty/unproven inventory apply"]

Expand All @@ -32,7 +33,9 @@ flowchart TD
removals -- "no" --> apply
removals -- "yes, NQE evidence present" --> review
removals -- "yes, authoritative manifest" --> review
review --> approve --> apply
review --> approve --> blast
blast -- "yes" --> apply
blast -- "no" --> block
removals -- "yes, NQE evidence absent" --> block

classDef neutral fill:#F1EFE8,stroke:#5F5E5A,color:#2C2C2A;
Expand All @@ -41,7 +44,7 @@ flowchart TD
classDef blocked fill:#FCEBEB,stroke:#A32D2D,color:#501313;

class start,snapshot,org_check,nqe,manifest,removals,review neutral;
class approve,apply safe;
class approve,blast,apply safe;
class block blocked;
```

Expand All @@ -63,6 +66,7 @@ flowchart TB
subgraph plan_apply["awssync — plan and apply"]
plan["plan / dry-run\nPOST /nqe + GET /cloudAccounts"]
disk["payload.json\nwritten to disk before any change"]
safety["Removal gates\nexplicit approval + count/% ceilings"]
apply["--apply\nPATCH /cloudAccounts/{setupId}"]
apply_plan["apply-plan\nreload current state + validate\nGovCloud removals refused"]
end
Expand All @@ -81,8 +85,9 @@ flowchart TB

preflight -- "read-only" --> fwd
plan --> disk
disk --> apply
disk --> safety --> apply
disk --> apply_plan
apply_plan --> safety
apply --> patch_accts
apply_plan --> patch_accts
plan --> nqe
Expand All @@ -94,7 +99,7 @@ flowchart TB
classDef artifact fill:#FAEEDA,stroke:#854F0B,color:#412402;

class cli,cron,preflight neutral;
class plan,apply,apply_plan neutral;
class plan,safety,apply,apply_plan neutral;
class nqe,get_accts,patch_accts,get_snap fwdnode;
class disk artifact;
```
Expand Down Expand Up @@ -170,13 +175,14 @@ flowchart TB
removal{"Any removals?"}
patch["--apply --yes\nPATCH /cloudAccounts/{setupId}"]
approved["--allow-removals\nexplicit approval"]
blast["--max-removals\n--max-removal-percent"]
end

manifest --> validate
validate --> onboard --> create_files --> post
validate --> sync --> diff --> removal
removal -- "no" --> patch
removal -- "yes" --> approved --> patch
removal -- "yes" --> approved --> blast --> patch

classDef neutral fill:#F1EFE8,stroke:#5F5E5A,color:#2C2C2A;
classDef fwdnode fill:#E6F1FB,stroke:#185FA5,color:#042C53;
Expand All @@ -186,7 +192,7 @@ flowchart TB
class manifest,validate,onboard,sync,diff,removal neutral;
class create_files artifact;
class post,patch fwdnode;
class approved safe;
class approved,blast safe;
```

For GovCloud, use `--partition aws-us-gov` when onboarding. Existing-setup sync derives and preserves the partition from the current role ARNs. Mixed partitions or a mismatch between role ARNs and configured regions fail before a payload can be applied.
Expand Down Expand Up @@ -404,6 +410,7 @@ flowchart LR
- Static-key collector secrets are only included in the create payload when explicitly supplied. Without the secret, the file contains a placeholder and is marked not POST-ready.
- Removals require explicit `--allow-removals` flag; `awssync` will not silently
remove accounts from a Forward setup.
- Optional `--max-removals` and `--max-removal-percent` ceilings limit aggregate and per-setup removal blast radius and are rechecked immediately before apply.
- GovCloud NQE removals additionally require positive Organizations evidence. Generic no-evidence flags cannot override this gate.
- Manifest removals require an authoritative complete manifest plus `--allow-removals`.
- `apply-plan` reloads current state and refuses GovCloud removals, so a saved payload cannot bypass the source workflow's safety checks.
Expand Down
Loading