From b5014dd614ac2735930bda77bbb7c4f2d2ba8fe0 Mon Sep 17 00:00:00 2001 From: Dusty <42273218+DustyStudy@users.noreply.github.com> Date: Tue, 22 Sep 2026 19:26:16 -0500 Subject: [PATCH] docs(proof): security-baseline-new-accounts tested against a real org Ran the member-baseline StackSet for real: applied via Terraform from the management account against a test OU, verified GuardDuty/Security Hub/Config came up in the target account, then tore it down and reverified everything was gone. Found and fixed two gaps the proof surfaced: - README's Prerequisites was missing cloudformation:ActivateOrganizations Access; CreateStackSet fails without it even with Organizations trusted access already enabled. - aws_cloudformation_stack_set_instance's region argument is deprecated in AWS provider v6; switched to stack_set_instance_region. See docs/PROOF.md for the claims/evidence table, what running it for real found, and what this run does not prove. --- README.md | 8 ++ docs/PROOF.md | 73 +++++++++++++++++++ docs/proof/member-account-state.json | 43 +++++++++++ docs/proof/prerequisite-gap.json | 15 ++++ docs/proof/stackset-instance-status.json | 17 +++++ .../member-baseline/README.md | 16 +++- .../member-baseline/main.tf | 12 ++- 7 files changed, 180 insertions(+), 4 deletions(-) create mode 100644 docs/PROOF.md create mode 100644 docs/proof/member-account-state.json create mode 100644 docs/proof/prerequisite-gap.json create mode 100644 docs/proof/stackset-instance-status.json diff --git a/README.md b/README.md index 3868b0a..377ced9 100644 --- a/README.md +++ b/README.md @@ -299,6 +299,14 @@ destination URL, no custom headers). See the module README before relying on this in production — it needs a short tuning pass against a real Wiz payload first. +## Proof + +`security-baseline-new-accounts` (the `member-baseline` StackSet) has been +run for real against a real AWS Organization, verified against AWS's own +records, and torn down cleanly. See [`docs/PROOF.md`](docs/PROOF.md). The +rest of the toolbox has not yet been tested this way — treat that +distinction as real, not a formality. + ## CI GitHub Actions on every push/PR: diff --git a/docs/PROOF.md b/docs/PROOF.md new file mode 100644 index 0000000..b98a6c7 --- /dev/null +++ b/docs/PROOF.md @@ -0,0 +1,73 @@ +# Proof that security-baseline-new-accounts works + +Run for real against a real AWS Organization on **2026-09-22**, then checked +against AWS's own records (GuardDuty, Security Hub, and Config APIs read +directly in the target account) rather than only against Terraform's own +output. Account IDs and the OU ID are masked below and in the evidence +files; they're not secret, just not worth publishing. + +Only `security-baseline-new-accounts` / `member-baseline` has been tested +this way so far. The rest of this toolbox has not — see the README's +top-level Proof section. + +## What was tested + +| | | +|---|---| +| **Org** | One AWS Organization: a management account, and a dedicated single-account OU already used to prove out `aws-orgseed` | +| **Target** | `member-baseline`'s StackSet, applied via this repo's Terraform module (not the standalone CloudFormation template) | +| **Method** | `terraform apply` from the Organizations management account; GuardDuty/Security Hub/Config APIs read directly in the target member account before, during, and after; teardown re-verified the same way | + +## 1. Claims and evidence + +| # | Claim | Result | Evidence | +|---|---|---|---| +| 1 | The StackSet deploys via `SERVICE_MANAGED` auto-deployment to every account in a target OU | **Proven** | [`stackset-instance-status.json`](proof/stackset-instance-status.json): instance status `CURRENT` / `SUCCEEDED` for the target account, `OrganizationalUnitId` matching the target OU | +| 2 | GuardDuty, Security Hub, and AWS Config all come up in the target account | **Proven** | [`member-account-state.json`](proof/member-account-state.json): all three unset before, all three enabled and recording after | +| 3 | The Config recorder actually records (not just created) | **Proven** | Same file: `describe-configuration-recorder-status` shows `"recording": true, "lastStatus": "SUCCESS"` | +| 4 | The Config S3 bucket is created with the documented name and is reachable | **Proven** | Same file: `HEAD` on `aws-config--us-east-1` returns 200 before teardown | +| 5 | Tearing down removes everything, including the S3 bucket | **Proven** | Same file, `after_destroy`: all three services unset again, bucket `HEAD` returns 404, StackSet gone from `list-stack-sets` in the management account | +| 6 | The module's documented prerequisites are sufficient to deploy | **Disproven, then fixed** | [`prerequisite-gap.json`](proof/prerequisite-gap.json): `CreateStackSet` failed with only the README's original single prerequisite applied; needed a second, undocumented one | + +## 2. What running it for real found + +Neither `terraform validate`, `tflint`, nor Checkov catch either of these — they're runtime/environment gaps, not code defects visible from the plan. + +| Found | By | Fixed | +|---|---|---| +| `CreateStackSet` requires `cloudformation:ActivateOrganizationsAccess` in the calling account in addition to the Organizations-side trusted access the README already documented. Without it: `ValidationError: You must enable organizations access to operate a service managed stack set` | The first real `terraform apply` | README Prerequisites (this proof's commit); `docs/proof/prerequisite-gap.json` has the exact error and fix | +| `aws_cloudformation_stack_set_instance`'s `region` argument is deprecated in AWS provider v6 (`Use stack_set_instance_region instead`) — still worked, but every apply printed a warning | `terraform plan` output during this test | `main.tf`, this proof's commit | +| The Config bucket isn't empty by the time you'd tear it down — Config writes a `ConfigWritabilityCheckFile` on first recorder start within seconds, and the bucket has versioning on. A plain `terraform destroy` (or `aws s3 rb`) fails against a non-empty versioned bucket | Reading the bucket's object versions before attempting teardown | Not a code fix — documented in section 3 below, since retaining the bucket is the module's own documented default (`retain_stacks_on_account_removal`) and emptying it is an operator step, not something the module should do silently | + +## 3. What this does not prove + +- **New-account auto-deployment.** `auto_deployment.enabled = true` is meant to catch accounts added to the OU *after* the StackSet exists. This run created the StackSet against an OU that already contained its one account — the "add an account later and watch it get baselined automatically" path was not exercised. +- **Multi-region.** Only `regions = ["us-east-1"]` was tested. The `for_each` over multiple regions, and the `IncludeGlobalResourceTypes: false` reasoning for avoiding duplicate global-resource recording, rest on reading the code, not on a multi-region run. +- **`call_as = "DELEGATED_ADMIN"`.** Applied as `SELF` from the management account only; the delegated-administrator path is untested. +- **GovCloud.** Not exercised; the README's GovCloud region example was not run. +- **Clean teardown of a bucket with real Config history.** This test's bucket held one empty check file. A bucket with weeks of Config snapshots would need the same emptying step at a larger scale; not measured here. +- **Failure/rollback behavior.** `failure_tolerance_percentage = 20` was never exercised — every target account in this test succeeded, so partial-failure handling across an OU with multiple accounts is unverified. +- **Cost.** Real GuardDuty/Security Hub/Config charges accrued for the ~1 hour the stack was live in one account; not measured or reported here, and would scale with account count and Config's per-configuration-item pricing in real use. +- **One account, one OU.** Everything above is shown in one member account. Nothing here says the auto-deployment behavior holds at OU sizes beyond one account. + +## 4. Reproduce it + +Prerequisites: an AWS Organization; management-account credentials; a +throwaway OU with at least one member account you're fine enabling +GuardDuty/Security Hub/Config in temporarily. + +1. `aws organizations enable-aws-service-access --service-principal member.org.stacksets.cloudformation.amazonaws.com` +2. `aws cloudformation activate-organizations-access` (see section 2 — this is the gap the README was missing) +3. Apply the `member-baseline` module against your test OU: + ```hcl + module "security_baseline" { + source = "github.com/DustyStudy/aws-cloud-security-toolbox//terraform/security-baseline-new-accounts/member-baseline" + target_organizational_unit_ids = [""] + regions = ["us-east-1"] + } + ``` +4. In the target member account, confirm: `aws guardduty list-detectors`, `aws securityhub describe-hub`, `aws configservice describe-configuration-recorder-status` — all three should show enabled/recording within a couple of minutes. +5. Tear down: empty the `aws-config--` bucket's object versions first (`aws s3api list-object-versions` / `delete-object --version-id`), then `terraform destroy`. +6. Re-check step 4's three commands — all should be back to unset, and the S3 bucket should 404. + +`docs/proof/` holds the machine-readable evidence from the run above. diff --git a/docs/proof/member-account-state.json b/docs/proof/member-account-state.json new file mode 100644 index 0000000..4283654 --- /dev/null +++ b/docs/proof/member-account-state.json @@ -0,0 +1,43 @@ +{ + "note": "State of the target member account, read directly (not from the tool's own output), before and after the StackSet instance was applied. Account ID masked.", + "before_apply": { + "guardduty_list_detectors": { "DetectorIds": [] }, + "securityhub_describe_hub": { + "error": "InvalidAccessException: Account is not subscribed to AWS Security Hub" + }, + "configservice_describe_configuration_recorders": { "ConfigurationRecorders": [] } + }, + "after_apply": { + "guardduty_list_detectors": { "DetectorIds": ["0df571ef8a87497ca40e5c9e92f6d193"] }, + "securityhub_describe_hub": { + "HubArn": "arn:aws:securityhub:us-east-1::hub/default", + "SubscribedAt": "2026-09-23T00:17:29.068Z", + "AutoEnableControls": true, + "ControlFindingGenerator": "SECURITY_CONTROL" + }, + "configservice_describe_configuration_recorders": { + "ConfigurationRecorders": [ + { + "name": "StackSet-security-baseline-member-baseline-2d9f10e2-5a53-4714-9867-503ca242ad1e-ConfigRecorder-WY42O1VYTSAS", + "roleARN": "arn:aws:iam:::role/StackSet-security-baseline-member-baseli-ConfigRole-xczAQoxGWl15", + "recordingGroup": { "allSupported": true, "includeGlobalResourceTypes": false } + } + ] + }, + "configservice_describe_configuration_recorder_status": { + "recording": true, + "lastStatus": "SUCCESS", + "lastStartTime": "2026-09-22T19:17:51.588000-05:00" + }, + "s3_head_bucket_aws_config_bucket": { "result": "200 OK - bucket exists" } + }, + "after_destroy": { + "guardduty_list_detectors": { "DetectorIds": [] }, + "securityhub_describe_hub": { + "error": "InvalidAccessException: Account is not subscribed to AWS Security Hub" + }, + "configservice_describe_configuration_recorders": { "ConfigurationRecorders": [] }, + "s3_head_bucket_aws_config_bucket": { "error": "404 Not Found" }, + "cloudformation_list_stack_sets_in_management_account": { "Summaries": [] } + } +} diff --git a/docs/proof/prerequisite-gap.json b/docs/proof/prerequisite-gap.json new file mode 100644 index 0000000..6794024 --- /dev/null +++ b/docs/proof/prerequisite-gap.json @@ -0,0 +1,15 @@ +{ + "note": "The module's Prerequisites section (before this proof) listed only one of the two settings CreateStackSet actually requires. This is the failure that surfaced the gap, and the fix.", + "first_apply_attempt": { + "prerequisite_applied": "aws organizations enable-aws-service-access --service-principal member.org.stacksets.cloudformation.amazonaws.com", + "result": "terraform apply failed", + "error": "Error: creating CloudFormation StackSet (security-baseline-member-baseline): operation error CloudFormation: CreateStackSet, https response error StatusCode: 400, ValidationError: You must enable organizations access to operate a service managed stack set" + }, + "fix_applied": "aws cloudformation activate-organizations-access", + "verification": { + "before": { "describe-organizations-access": { "Status": "DISABLED (implicit - not yet activated)" } }, + "after": { "describe-organizations-access": { "Status": "ENABLED" } } + }, + "second_apply_attempt": "succeeded - see stackset-instance-status.json and member-account-state.json", + "readme_and_module_fixed_in": "this proof's commit - see README.md Prerequisites and main.tf" +} diff --git a/docs/proof/stackset-instance-status.json b/docs/proof/stackset-instance-status.json new file mode 100644 index 0000000..d3e82e8 --- /dev/null +++ b/docs/proof/stackset-instance-status.json @@ -0,0 +1,17 @@ +{ + "note": "Account IDs and the OU ID are masked (see docs/PROOF.md). Captured via: aws cloudformation describe-stack-instance --stack-set-name security-baseline-member-baseline --stack-instance-account --stack-instance-region us-east-1", + "StackInstance": { + "StackSetId": "security-baseline-member-baseline:e7d97ea0-9f81-4f31-92a2-4d806471514f", + "Region": "us-east-1", + "Account": "", + "StackId": "arn:aws:cloudformation:us-east-1::stack/StackSet-security-baseline-member-baseline-2d9f10e2-5a53-4714-9867-503ca242ad1e/2688bc10-b6e4-11f1-ba9d-0affec771efb", + "ParameterOverrides": [], + "Status": "CURRENT", + "StackInstanceStatus": { + "DetailedStatus": "SUCCEEDED" + }, + "OrganizationalUnitId": "", + "DriftStatus": "NOT_CHECKED", + "LastOperationId": "terraform-j5bI2sn3gDcq5Z0Za22ldYGyYd" + } +} diff --git a/terraform/security-baseline-new-accounts/member-baseline/README.md b/terraform/security-baseline-new-accounts/member-baseline/README.md index 3d881f7..8a2d9b6 100644 --- a/terraform/security-baseline-new-accounts/member-baseline/README.md +++ b/terraform/security-baseline-new-accounts/member-baseline/README.md @@ -23,7 +23,14 @@ there if you want to see or modify exactly what gets deployed per account. aws organizations enable-aws-service-access \ --service-principal member.org.stacksets.cloudformation.amazonaws.com ``` -2. Apply from the **Organizations management account**, or from an +2. CloudFormation's own Organizations access activated in the account you + apply from. This is separate from step 1 and easy to miss - without it, + `CreateStackSet` fails with `You must enable organizations access to + operate a service managed stack set`: + ```bash + aws cloudformation activate-organizations-access + ``` +3. Apply from the **Organizations management account**, or from an account registered as a delegated administrator for CloudFormation StackSets (set `call_as = "DELEGATED_ADMIN"` in that case). @@ -90,3 +97,10 @@ For GovCloud: enrolled through AWS Control Tower, or set up by hand), creating the baseline's recorder fails in that account and region. Exclude those OUs, or remove the existing recorder first. + +## Proof + +Run for real against a real AWS Organization - deployed via this Terraform +module through the StackSet mechanism it's actually designed for (not just +the standalone CFN template), then verified and torn down. See +[`docs/PROOF.md`](../../../docs/PROOF.md). diff --git a/terraform/security-baseline-new-accounts/member-baseline/main.tf b/terraform/security-baseline-new-accounts/member-baseline/main.tf index 157f776..119a170 100644 --- a/terraform/security-baseline-new-accounts/member-baseline/main.tf +++ b/terraform/security-baseline-new-accounts/member-baseline/main.tf @@ -4,6 +4,12 @@ # CloudFormation StackSets enabled for the Organization: # aws organizations enable-aws-service-access \ # --service-principal member.org.stacksets.cloudformation.amazonaws.com +# +# That alone is not enough - CloudFormation also needs its own Organizations +# access activated in the calling account, or CreateStackSet fails with +# "You must enable organizations access to operate a service managed stack +# set" (see docs/PROOF.md, section 4): +# aws cloudformation activate-organizations-access resource "aws_cloudformation_stack_set" "member_baseline" { name = "${var.name_prefix}-member-baseline" @@ -31,9 +37,9 @@ resource "aws_cloudformation_stack_set" "member_baseline" { resource "aws_cloudformation_stack_set_instance" "member_baseline" { for_each = toset(var.regions) - stack_set_name = aws_cloudformation_stack_set.member_baseline.name - call_as = var.call_as - region = each.value + stack_set_name = aws_cloudformation_stack_set.member_baseline.name + call_as = var.call_as + stack_set_instance_region = each.value deployment_targets { organizational_unit_ids = var.target_organizational_unit_ids