Skip to content

feat(helpers): add managed AWS Config S3 notifications #653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
edersonbrilhante wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat(helpers): add managed AWS Config S3 notifications #653

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
244b9b2
feat(helpers): add managed AWS Config S3 notifications
edersonbrilhante Aug 13, 2026
5e77db6
refactor(storage): remove AWS Config delivery policy
edersonbrilhante Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 29 additions & 16 deletions docs/configurations/deployments/helpers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,20 @@ ______________________________________________________________________

## Helper Modules

| Module | Example directory | Use when |
| -------------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------- |
| `modules/helpers/aws_config_recording` | `environments/prod/regions/eu-west-1/aws_config_recording` | You need configuration history for selected AWS resource types. |
| `modules/helpers/ami_policy` | `environments/prod/ami_policy` | Forge owns AMI usage policy support. |
| `modules/helpers/ami_sharing` | `environments/prod/regions/eu-west-1/ami_sharing` | Runner AMIs must be shared across accounts or regions. |
| `modules/helpers/cloud_custodian` | `environments/prod/cloud_custodian` | You run cleanup or governance policies from Forge. |
| `modules/helpers/cloud_formation` | `environments/prod/cloud_formation` | Integrations need CloudFormation admin/execution roles. |
| `modules/helpers/dedicated_mac_hosts` | `environments/prod/regions/eu-west-1/dedicated_mac_hosts` | Forge owns EC2 Mac Dedicated Host capacity. |
| `modules/helpers/ecr` | `environments/prod/regions/eu-west-1/ecr` | Forge owns ECR repositories for runner or helper images. |
| `modules/helpers/forge_subscription` | `environments/prod/forge_subscription` | Tenant accounts need Forge-managed IAM, Packer, S3, or ECR access. |
| `modules/helpers/microvm` | `environments/prod/regions/eu-west-1/microvm` | Provides regional MicroVM publishing/runtime foundations and egress. |
| `modules/helpers/opt_in_regions` | `environments/prod/opt_in_regions` | You need to enable AWS opt-in regions before regional deploys. |
| `modules/helpers/service_linked_roles` | `environments/prod/service_linked_roles` | The account needs the EC2 Spot service-linked role. |
| `modules/helpers/storage` | `environments/prod/storage` | Forge owns operational S3 buckets for logs, artifacts, or templates. |
| Module | Example directory | Use when |
| -------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------- |
| `modules/helpers/aws_config_recording` | `environments/prod/regions/eu-west-1/aws_config_recording` | You need AWS Config history with a Splunk-compatible S3/SQS producer. |
| `modules/helpers/ami_policy` | `environments/prod/ami_policy` | Forge owns AMI usage policy support. |
| `modules/helpers/ami_sharing` | `environments/prod/regions/eu-west-1/ami_sharing` | Runner AMIs must be shared across accounts or regions. |
| `modules/helpers/cloud_custodian` | `environments/prod/cloud_custodian` | You run cleanup or governance policies from Forge. |
| `modules/helpers/cloud_formation` | `environments/prod/cloud_formation` | Integrations need CloudFormation admin/execution roles. |
| `modules/helpers/dedicated_mac_hosts` | `environments/prod/regions/eu-west-1/dedicated_mac_hosts` | Forge owns EC2 Mac Dedicated Host capacity. |
| `modules/helpers/ecr` | `environments/prod/regions/eu-west-1/ecr` | Forge owns ECR repositories for runner or helper images. |
| `modules/helpers/forge_subscription` | `environments/prod/forge_subscription` | Tenant accounts need Forge-managed IAM, Packer, S3, or ECR access. |
| `modules/helpers/microvm` | `environments/prod/regions/eu-west-1/microvm` | Provides regional MicroVM publishing/runtime foundations and egress. |
| `modules/helpers/opt_in_regions` | `environments/prod/opt_in_regions` | You need to enable AWS opt-in regions before regional deploys. |
| `modules/helpers/service_linked_roles` | `environments/prod/service_linked_roles` | The account needs the EC2 Spot service-linked role. |
| `modules/helpers/storage` | `environments/prod/storage` | Forge owns operational S3 buckets for logs, artifacts, or templates. |

______________________________________________________________________

Expand Down Expand Up @@ -78,6 +78,20 @@ cd examples/deployments/helpers/terragrunt/environments/prod/regions/eu-west-1/a
cd examples/deployments/helpers/terragrunt/environments/prod/regions/eu-west-1/dedicated_mac_hosts
```

After applying `aws_config_recording`, inspect the values for a Splunk Data
Manager custom S3 input:

```bash
cd examples/deployments/helpers/terragrunt/environments/prod/regions/eu-west-1/aws_config_recording
terragrunt output -json splunk_s3_logs
terragrunt output -json splunk_s3_logs | jq -r '.sqs.url'
terragrunt output -json splunk_s3_logs | jq -r '.bucket_arn'
```

These outputs establish only the AWS producer contract. Configure the custom
Splunk parser and validate a real AWS Config `.json.gz` object through to
indexed events before calling the integration operational.

Plan the full helper environment only after the individual helper plans are
understood:

Expand All @@ -92,8 +106,7 @@ ______________________________________________________________________

1. `service_linked_roles` when the account needs the EC2 Spot service-linked role.
1. `opt_in_regions` before deploying into opt-in regions.
1. `storage` before `aws_config_recording` so the long-term bucket is available as its delivery dependency.
1. `aws_config_recording` before allocating Dedicated Hosts so their full history is captured.
1. `aws_config_recording` before allocating Dedicated Hosts so their full history is captured; it creates its own delivery bucket and notification queue.
1. `dedicated_mac_hosts` only after reviewing Mac host cost and minimum allocation periods.
1. `ecr` if Forge builds or stores runner/helper images.
1. `ami_sharing` if runner AMIs live in a central image account.
Expand Down
37 changes: 37 additions & 0 deletions docs/configurations/deployments/integrations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,40 @@ environments/prod/regions/eu-west-1/splunk_dependency_monitor
Deploy it once per Forge AWS region, after `splunk_o11y_conf_shared` and
`splunk_secrets`.

The Splunk Cloud Data Manager example includes a disabled custom S3 input for
AWS Config history. The helper and integration examples have separate
Terragrunt roots, so transfer the producer values explicitly:

1. Apply the regional helper:

```bash
cd examples/deployments/helpers/terragrunt/environments/prod/regions/eu-west-1/aws_config_recording
terragrunt apply
terragrunt output -json splunk_s3_logs
terragrunt output -json splunk_s3_logs | jq -r '.sqs.url'
terragrunt output -json splunk_s3_logs | jq -r '.bucket_arn'
```

1. In
`examples/deployments/integrations/terragrunt/environments/prod/splunk_cloud_data_manager/config.yml`,
copy `sqs.url` into `sqs_urls` and `bucket_arn` into
`s3_bucket_patterns` for `forge-aws-config-prod`.

1. Leave `kms_key_arns` empty because `bucket_kms_key_arn` is `null`; the
helper bucket uses SSE-S3.

1. Confirm the helper resources and Data Manager deployment use the same AWS
account, then set the input's `enabled` value to `true`.

1. Apply `environments/prod/splunk_cloud_data_manager_common`, followed by
`environments/prod/splunk_cloud_data_manager`.

The input name, index, IAM region, and `source_type` remain owned by the
integration example. Before enabling it, deploy the custom Splunk parser and
validate one real AWS Config `.json.gz` object from S3 notification through to
the expected indexed events. The AWS producer shape alone is not end-to-end
validation.

Skip both pages if your observability stack is not Splunk.

______________________________________________________________________
Expand All @@ -58,6 +92,7 @@ ______________________________________________________________________
| `environments/prod/_environment_wide_settings/_environment.yml` | AWS account, default region, AWS profile, and remote state. |
| `environments/prod/github_webhook_relay_destination/config.yml` | Destination EventBridge and reader role settings. |
| `environments/prod/github_webhook_relay_destination_receivers/config.yml` | Receiver bundle configuration. |
| `environments/prod/splunk_cloud_data_manager/config.yml` | Splunk URL, index, AWS Config source type, and input name. |
| `environments/prod/regions/eu-west-1/teleport/config.yml` | Teleport cluster, namespace, chart, and EKS access values. |
| `release_versions.yml` | Integration module sources, refs, and `module_path` values. |

Expand Down Expand Up @@ -96,3 +131,5 @@ ______________________________________________________________________
- Splunk modules can depend on `splunk_secrets`, helper buckets, and
CloudFormation helper roles. Use externally managed resources if your company
already provides them.
- The AWS Config SQS queue is dedicated to Data Manager. Do not attach another
consumer that can remove its notifications before Splunk reads them.
24 changes: 24 additions & 0 deletions docs/integrations/splunk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,30 @@ Splunk contract.

The full secret list is in [Splunk Secrets](splunk-secrets.md).

## AWS Config Custom S3 Input

The `aws_config_recording` helper creates an AWS-side producer compatible with
the Data Manager `s3-custom-logs` fields. Apply it and inspect the handoff
values:

```bash
cd examples/deployments/helpers/terragrunt/environments/prod/regions/eu-west-1/aws_config_recording
terragrunt apply
terragrunt output -json splunk_s3_logs
terragrunt output -json splunk_s3_logs | jq -r '.sqs.url'
terragrunt output -json splunk_s3_logs | jq -r '.bucket_arn'
```

Copy the SQS URL and bucket ARN into the disabled `forge-aws-config-prod`
custom-log item, leave `kms_key_arns` empty because the bucket uses SSE-S3,
and confirm Data Manager runs in the same AWS account. The example source type
is `forgecicd:aws:config:s3`.

This is producer compatibility, not proof of successful ingestion. Configure
the source type's parser for the AWS Config `configurationItems` envelope,
then validate a real `.json.gz` object from S3 notification through queue
consumption to the expected indexed events before enabling production use.

## Apply Secrets First

```bash
Expand Down
28 changes: 14 additions & 14 deletions docs/reference/module-catalog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,20 +29,20 @@ EC2-only deployments can skip `examples/deployments/infra`.

## Helper Modules

| Module | Role | Required? | Example root | Validation |
| -------------------------------------- | ------------------------------------------------------------ | --------- | ------------------------------ | ----------------------------------------------- |
| `modules/helpers/aws_config_recording` | AWS Config history for caller-selected AWS resource types. | Optional | `examples/deployments/helpers` | Recorder is active for the configured types. |
| `modules/helpers/ami_policy` | AMI policy support for approved runner images. | Optional | `examples/deployments/helpers` | Policy plan plus AMI usage review. |
| `modules/helpers/ami_sharing` | Shares runner AMIs across accounts or regions. | Optional | `examples/deployments/helpers` | Target account can describe and launch AMI. |
| `modules/helpers/cloud_custodian` | Cleanup and policy jobs for stale resources. | Optional | `examples/deployments/helpers` | Custodian dry run and scheduled job output. |
| `modules/helpers/cloud_formation` | CloudFormation admin/execution roles for setup paths. | Optional | `examples/deployments/helpers` | Stack role assumption check. |
| `modules/helpers/dedicated_mac_hosts` | Mac Dedicated Hosts, host groups, and license configuration. | Optional | `examples/deployments/helpers` | Host allocation and group membership review. |
| `modules/helpers/ecr` | ECR repositories for runner and CI helper containers. | Optional | `examples/deployments/helpers` | Push/pull smoke for configured repositories. |
| `modules/helpers/forge_subscription` | Tenant-side access for ForgeMT jobs and artifacts. | Optional | `examples/deployments/helpers` | Tenant role can access intended S3/ECR/secrets. |
| `modules/helpers/microvm` | Regional MicroVM publishing and VPC egress foundation. | Optional | `examples/deployments/helpers` | Artifact plan and connector `ACTIVE` state. |
| `modules/helpers/opt_in_regions` | Enables AWS opt-in regions. | Sometimes | `examples/deployments/helpers` | AWS account region status is enabled. |
| `modules/helpers/service_linked_roles` | Creates the EC2 Spot service-linked role. | Sometimes | `examples/deployments/helpers` | Role exists before EC2 runner launch. |
| `modules/helpers/storage` | S3 buckets for artifacts, templates, logs, and integrations. | Optional | `examples/deployments/helpers` | Bucket policy, encryption, and access checks. |
| Module | Role | Required? | Example root | Validation |
| -------------------------------------- | ------------------------------------------------------------ | --------- | ------------------------------ | ------------------------------------------------ |
| `modules/helpers/aws_config_recording` | AWS Config history with a Data Manager-compatible producer. | Optional | `examples/deployments/helpers` | Recorder, queue, DLQ, and `.json.gz` event plan. |
| `modules/helpers/ami_policy` | AMI policy support for approved runner images. | Optional | `examples/deployments/helpers` | Policy plan plus AMI usage review. |
| `modules/helpers/ami_sharing` | Shares runner AMIs across accounts or regions. | Optional | `examples/deployments/helpers` | Target account can describe and launch AMI. |
| `modules/helpers/cloud_custodian` | Cleanup and policy jobs for stale resources. | Optional | `examples/deployments/helpers` | Custodian dry run and scheduled job output. |
| `modules/helpers/cloud_formation` | CloudFormation admin/execution roles for setup paths. | Optional | `examples/deployments/helpers` | Stack role assumption check. |
| `modules/helpers/dedicated_mac_hosts` | Mac Dedicated Hosts, host groups, and license configuration. | Optional | `examples/deployments/helpers` | Host allocation and group membership review. |
| `modules/helpers/ecr` | ECR repositories for runner and CI helper containers. | Optional | `examples/deployments/helpers` | Push/pull smoke for configured repositories. |
| `modules/helpers/forge_subscription` | Tenant-side access for ForgeMT jobs and artifacts. | Optional | `examples/deployments/helpers` | Tenant role can access intended S3/ECR/secrets. |
| `modules/helpers/microvm` | Regional MicroVM publishing and VPC egress foundation. | Optional | `examples/deployments/helpers` | Artifact plan and connector `ACTIVE` state. |
| `modules/helpers/opt_in_regions` | Enables AWS opt-in regions. | Sometimes | `examples/deployments/helpers` | AWS account region status is enabled. |
| `modules/helpers/service_linked_roles` | Creates the EC2 Spot service-linked role. | Sometimes | `examples/deployments/helpers` | Role exists before EC2 runner launch. |
| `modules/helpers/storage` | S3 buckets for artifacts, templates, logs, and integrations. | Optional | `examples/deployments/helpers` | Bucket policy, encryption, and access checks. |

Helpers are not runtime platform modules. Deploy them only when ForgeMT owns
that operating concern.
Expand Down
7 changes: 5 additions & 2 deletions docs/reference/module-dependencies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ ______________________________________________________________________

| Module | Deploy before platform? | Why |
| -------------------------------------- | ------------------------------ | ---------------------------------------------------------------- |
| `modules/helpers/storage` | Before AWS Config recording | Supplies the long-term S3 delivery bucket used by the example. |
| `modules/helpers/aws_config_recording` | Before recorded resources | Captures configuration history from resource creation. |
| `modules/helpers/storage` | Optional | Creates shared artifact buckets; it has no Config delivery role. |
| `modules/helpers/aws_config_recording` | Before recorded resources | Creates its delivery bucket and queue, then captures history. |
| `modules/helpers/opt_in_regions` | Yes, for opt-in regions | Regional resources cannot deploy until the region is enabled. |
| `modules/helpers/service_linked_roles` | Usually, for EC2 Spot | Some accounts need the EC2 Spot service-linked role first. |
| `modules/helpers/ami_policy` | Optional | Account policy support for AMI usage. |
Expand All @@ -73,6 +73,9 @@ ______________________________________________________________________
events to it.
- Deploy `splunk_secrets` and `splunk_o11y_conf_shared` before each regional
`splunk_dependency_monitor` instance.
- For AWS Config ingestion, apply the regional `aws_config_recording` helper,
copy its `splunk_s3_logs` queue URL and bucket ARN into the disabled Data
Manager custom S3 input, validate parsing, and then enable the input.

______________________________________________________________________

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/module-layout.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ ______________________________________________________________________

| Module | Purpose |
| -------------------------------------- | ------------------------------------------------------------------------------------------- |
| `modules/helpers/aws_config_recording` | Records caller-selected AWS resource types with AWS Config. |
| `modules/helpers/aws_config_recording` | Records selected AWS resources with a Data Manager-compatible S3/SQS producer. |
| `modules/helpers/ami_policy` | IAM policy support for Forge AMI usage. |
| `modules/helpers/ami_sharing` | Shares runner AMIs across accounts or regions. |
| `modules/helpers/cloud_custodian` | Runs cleanup and policy automation. |
Expand Down
Loading
Loading