diff --git a/README.md b/README.md index 499e1a7..77b596f 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,50 @@ Terraform for an AWS EKS platform with private worker nodes, Prometheus, public HTTPS Grafana, and GitHub OIDC deployments. +[Deploy dev](docs/deployment.md) · [GitHub Actions](docs/github-actions.md) · [Costs](docs/costs.md) · [Cleanup](docs/cleanup.md) · [Verification](docs/verification.md) + ## What gets built -```text -GitHub Actions -- OIDC --> plan / apply IAM roles - | - S3 state + lock files - | - Terraform platform root - | - VPC across availability zones - | | - public subnets private subnets - NAT gateways EKS managed nodes - Grafana ALB -- HTTP ----> Grafana + Prometheus - ^ | - | gp3 EBS - HTTPS / ACM certificate - | - Route 53: grafana..hudson.tf - -Terraform monitoring root --> Helm / Kubernetes API -EKS control-plane logs --> CloudWatch -CNI / EBS / ALB controller --> individual IRSA roles -``` +![EKS platform with public Grafana ingress and private worker nodes across availability zones](docs/architecture.svg) + +[Open full size](docs/architecture.svg) · [Editable draw.io source](docs/eks-platform-reference.drawio) · [Deployment and state flow](docs/github-actions.md) Dev uses two zones, two small nodes, and one NAT gateway. Prod uses three -zones, three larger nodes, and a NAT gateway in each zone. Neither cluster -places worker nodes on the public internet. The public ALB forwards only -Grafana traffic; Prometheus stays inside the cluster. +zones, three larger nodes, and a NAT gateway in each zone. Worker nodes have +private IPs. The public ALB forwards Grafana traffic; Prometheus stays inside +the cluster. Grafana and Prometheus each use an encrypted gp3 volume. + +The drawing shows logical tiers across those zones. A managed node group spans +the private subnets; monitoring runs on those nodes and is not replicated. +The ALB is created by the AWS Load Balancer Controller from the Kubernetes +Ingress. Route 53 provides DNS and ACM provides the ALB's TLS certificate. ## Repository layout ```text -bootstrap/ state bucket and GitHub OIDC identities -modules/network/ VPC, subnets, routing and NAT -modules/cluster/ EKS, managed nodes, add-ons and IRSA -modules/observability/ monitoring, ALB controller, TLS and DNS -modules/pipeline/ scoped state access and GitHub IAM roles -environments/ shared platform root - terraform.tfvars platform settings keyed by dev/prod workspace - observability/ shared monitoring root - terraform.tfvars monitoring settings keyed by dev/prod workspace -.github/workflows/terraform.yml validation, PR plans and manual applies -scripts/ local checks and workflow deployment +eks-platform-reference/ +├── .github/workflows/ +│ └── terraform.yml # PR plans; manual applies +├── bootstrap/ # State bucket and GitHub OIDC roles +├── environments/ # Shared platform root +│ ├── terraform.tfvars # Dev/prod settings selected by workspace +│ └── observability/ # Monitoring root and its own tfvars +├── modules/ +│ ├── network/ # VPC, subnet routing and NAT +│ ├── cluster/ # EKS, managed nodes, add-ons and IRSA +│ ├── observability/ # Helm monitoring, public ingress and DNS +│ └── pipeline/ # Plan/apply role trust and permissions +├── scripts/ # Checks, CI deployment and cleanup audit +├── tests/ # Deployment, PR-comment and audit tests +├── docs/ +│ ├── eks-platform-reference.drawio # Editable, two-page architecture +│ ├── architecture.svg # AWS topology preview +│ ├── delivery.svg # Deployment and state preview +│ ├── deployment.md # Local setup and dev/prod operations +│ ├── github-actions.md # Environments, approvals and retries +│ ├── costs.md # Hourly/daily estimates and assumptions +│ ├── cleanup.md # Teardown and AWS deletion audit +│ └── verification.md # Test coverage and live checks +└── README.md ``` Both roots use Terraform CLI workspaces named `dev` and `prod`. All environment @@ -72,7 +73,7 @@ The cluster does not grant implicit administrator access to its creator. The public `hudson.tf` hosted zone must already exist in the same account, with its nameservers delegated correctly. Set its exact `hosted_zone_id` in bootstrap -and both monitoring tfvars files. +and in both environment entries in `environments/observability/terraform.tfvars`. Terraform selects that ID and checks the zone name and public-zone setting before adding Grafana alias records and ACM validation records. It does not create or replace the zone or register the domain. Use unused Grafana hostnames; existing records @@ -87,242 +88,40 @@ the end of standard support; schedule explicit upgrades before that happens. ## Run dev first -Start each shell block below at the repository root. The commands enter the -relevant Terraform folder and return to the root when finished. They provision -billable AWS resources. Read the [cost breakdown](#cost) before leaving a -cluster running. - -### Bootstrap state and CI roles - -Authenticate your AWS profile first using your configured login method -(`aws login` or `aws sso login`), then select it for this shell: - -```bash -export AWS_PROFILE=YOUR_PROFILE -aws sts get-caller-identity -cp bootstrap/terraform.tfvars.example bootstrap/terraform.tfvars -``` - -Read the repository value GitHub actually uses in its OIDC subject: - -```bash -gh api repos/OWNER/REPOSITORY/actions/oidc/customization/sub \ - --jq '.sub_claim_prefix | sub("^repo:"; "")' -``` - -Set `github_repository` in the local tfvars file to that exact result. New -repositories use `OWNER@OWNER-ID/REPOSITORY@REPOSITORY-ID`; older repositories -may use `OWNER/REPOSITORY`. Do not infer the value from the repository URL. -The module adds `repo:` and the exact environment to build the trust condition. -An incorrect value causes `Not authorized to perform sts:AssumeRoleWithWebIdentity`. -After correcting it, plan and apply bootstrap again to update the role trusts, -then retry the workflow. This does not require reapplying the EKS platform. -See [GitHub's subject formats](https://docs.github.com/en/actions/reference/security/oidc#immutable-subject-claims). - -Choose a globally unique state bucket and confirm the cluster and role names -match the environment inputs. Confirm that `hosted_zone_id` identifies the zone whose -name servers are delegated for your domain; duplicate hosted-zone names are -allowed in AWS. If this account already has GitHub's OIDC provider, set -`github_oidc_provider_arn` to its ARN. There is only one provider per issuer per -account; this configuration can reuse it. - -```bash -cd bootstrap -terraform init -terraform plan -out=bootstrap.tfplan -terraform apply bootstrap.tfplan -terraform output -cd .. -``` - -Bootstrap starts with local state because the bucket does not exist yet. -Move that state into the bucket once it has been created: - -```bash -cp bootstrap/backend.tf.example bootstrap/backend.tf -cp bootstrap/backend.hcl.example bootstrap/backend.hcl -# Edit bootstrap/backend.hcl with the bucket output and region. -cd bootstrap -terraform init -migrate-state -backend-config=backend.hcl -cd .. -``` - -Confirm the migration prompt. Secure any local backup state left by the -migration. Bootstrap state lives at `bootstrap/terraform.tfstate`; environment -roles cannot read or modify it. Bucket deletion is blocked with `prevent_destroy`. -S3 versioning, encryption, TLS-only access, and public-access blocking are enabled. - -### Initialize the two roots - -```bash -cp environments/backend.hcl.example environments/backend.hcl -cp environments/observability/backend.hcl.example environments/observability/backend.hcl -# Set the same bucket and region in both files. Keep their distinct keys. -cd environments -terraform init -backend-config=backend.hcl -terraform workspace select -or-create dev -terraform workspace select -or-create prod -terraform workspace select dev -cd observability -terraform init -backend-config=backend.hcl -terraform workspace select -or-create dev -terraform workspace select -or-create prod -terraform workspace select dev -cd ../.. -``` - -Create both workspaces using your bootstrap credentials before enabling PR -plans. A read-only plan role cannot initialize an empty workspace state. The -backend uses native S3 locking with `use_lockfile = true`, without DynamoDB. -The [S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) -explains lock permissions and workspace paths. - -| Workspace | Platform state | Monitoring state | -| --- | --- | --- | -| dev | `environments/dev/platform.tfstate` | `environments/dev/observability.tfstate` | -| prod | `environments/prod/platform.tfstate` | `environments/prod/observability.tfstate` | - -### Plan and apply dev - -In `environments/terraform.tfvars`, set `administrator_arns` inside the -`environments.dev` object to your human administrator's persistent IAM ARN: +Use the [deployment guide](docs/deployment.md) to configure inputs, bootstrap the +state bucket and OIDC roles, and initialize both Terraform roots. The guide also +covers Grafana access and the checks to run after deployment. -```hcl -administrator_arns = { - operator = "REPLACE_WITH_YOUR_PERSISTENT_IAM_ROLE_ARN" -} -``` - -Then apply using that role's AWS session: +Once setup is complete, start at the repository root: ```bash +export AWS_PROFILE=dev cd environments terraform workspace select dev terraform plan -out=dev.tfplan terraform apply dev.tfplan -cd observability -terraform workspace select dev -terraform plan -out=dev.tfplan -terraform apply dev.tfplan -cd ../.. -``` -The second apply waits for Helm, certificate validation, and ALB reconciliation. -An access-denied error is a reason to check your IAM principal and its EKS access -entry. Do not rerun monitoring with an unrelated administrator identity. - -### Open Grafana and check the cluster - -```bash -cd environments -cluster_region=$(terraform output -raw region) -cluster_name=$(terraform output -raw cluster_name) -aws eks update-kubeconfig --region "$cluster_region" --name "$cluster_name" -kubectl get nodes -kubectl -n monitoring get pods,pvc,ingress cd observability -terraform output -raw grafana_url -kubectl -n monitoring get secret monitoring-grafana \ - -o jsonpath='{.data.admin-password}' | base64 --decode +terraform workspace select dev +terraform plan -out=dev-monitoring.tfplan +terraform apply dev-monitoring.tfplan cd ../.. ``` -Log in as `admin` with that generated password. The dev URL is -`https://grafana.dev.hudson.tf`; prod uses `https://grafana.prod.hudson.tf`. -HTTP redirects to HTTPS. Anonymous access and account sign-up are disabled. -Keep the password private and change it after first login. Prometheus provides -node, pod, and Kubernetes object metrics; Grafana ships the chart's dashboards. - -The API endpoint is public with IAM authentication, and private access is also -enabled. The sample API CIDRs allow GitHub-hosted runners with changing egress -addresses. For a fixed-egress runner, restrict `public_access_cidrs` in platform -tfvars. `ingress_cidrs` separately controls who can reach Grafana's ALB. -For a long-lived shared Grafana deployment, add SSO before giving more people access. - -### Run prod - -Select `prod` in both roots and configure `environments.prod.administrator_arns` -in the platform tfvars. Use the same plan and apply commands; the selected -workspace supplies the prod settings. Name the saved plans `prod.tfplan`. -Do not copy Terraform code or change modules to deploy prod. +These commands create billable resources. For a short dev verification run, +budget $2–3 for a few hours or $8–10 for a day, then follow the cleanup runbook. +The [full cost breakdown](docs/costs.md) explains the rates and usage charges. ## GitHub Actions -Create GitHub environments `dev-plan`, `prod-plan`, `dev`, and `prod`. Each needs -these environment variables; they are configuration, not stored AWS secrets: - -| Variable | Value | -| --- | --- | -| `AWS_ROLE_ARN` | Corresponding role from bootstrap's `github_role_arns` output | -| `AWS_ACCOUNT_ID` | Account allowed by the AWS credentials action | -| `AWS_REGION` | Region matching tfvars and the state bucket | -| `TF_STATE_BUCKET` | Bootstrap state bucket output | - -Restrict deployment branches on `dev` and `prod` to `main`. Require reviewers -for `prod`. Restrict who can approve same-repository PRs against `dev-plan` and -`prod-plan`: plan roles can read state and Kubernetes Secrets to refresh Helm -releases. The plan environments must allow PR refs, not just `main`. Protect -`main`, require the validate job, and require review for changes to workflows, -Terraform, and IAM policies. Environment branch restrictions are essential: -the OIDC environment subject does not itself prove which branch requested it. - -Every PR runs formatting, validation, and offline plan tests. Same-repository -PRs also plan both environments using the read-only roles. Fork PRs run the -credential-free checks only. No `pull_request_target` execution is used. - -For an environment without a cluster yet, the platform plan runs and the live -monitoring plan is explicitly deferred. Static validation and offline monitoring -plans still run. Once a cluster exists, both live plans run. PR plans are unlocked -because their roles cannot write lock files; they can be stale. Manual applies -create new saved plans under S3 locks and apply those exact files in the same job. -PR plan artifacts are never reused for deployment or published as artifacts. - -Each PR plan updates one bot comment per environment with platform and monitoring -resource counts: add, change, destroy and replace. The comment includes the -commit, job result and workflow link. Failed plans and deferred monitoring are -marked explicitly. Comments contain counts only; raw plan values and JSON are -not posted. Retries update the existing comment for that environment. - -A merge or push to `main` runs validation only. It does not request AWS -credentials or run a live plan or apply. This lets the demo be merged and -maintained without starting billable infrastructure. -Manual runs select one environment and either `plan` or `apply`; they default -to a dev plan. Manual applies require `main`. Deployment jobs serialize per environment and -are never canceled mid-apply. The prod environment approval sits before AWS -credentials are requested. This is not a sequential dev-to-prod promotion gate. - -After the workflow is merged into `main`, use Actions > Terraform > Run workflow -to select the environment and operation, or run: +Pull requests run formatting, validation, and offline tests. Approved plans for +same-repository PRs use separate read-only roles for dev and prod and update a +comment per environment with the results. Fork PRs run credential-free checks. -```bash -gh workflow run terraform.yml --ref main -f environment=dev -f operation=plan -gh workflow run terraform.yml --ref main -f environment=prod -f operation=apply -``` - -Manual plans use the corresponding `dev-plan` or `prod-plan` environment and -read-only role. Manual applies use `dev` or `prod`, including its approval rules. -To retry a failed PR job before merging, use GitHub's Re-run failed jobs button -or `gh run rerun RUN_ID --failed`. To retry just one environment job, find its -`databaseId` with `gh run view RUN_ID --json jobs --jq '.jobs[] | {name, databaseId}'`, -then run `gh run rerun RUN_ID --job JOB_ID` using that ID. -Plans do not take a state lock, so an active local apply does not block them; -wait until the apply finishes for a reliable result. Applies wait up to five -minutes for the S3 lock. Retry after the other apply finishes if that wait expires. - -CI refuses plans that delete or replace the EKS cluster. Those changes need -monitoring-first cleanup and an AWS audit while its controllers still run; -follow [the cleanup procedure](docs/cleanup.md) before changing the cluster's -identity or other settings that require replacement. Initial creation and -in-place updates run normally. - -The apply roles can manage EC2 and EKS resources in the configured region, -cluster-prefixed IAM roles, and records in the chosen hosted zone. Direct IAM grants target cluster-prefixed roles, and state-object deletion is -not granted; lock files can be deleted. This is not a permissions boundary: -creating roles and attaching policies can allow privilege escalation beyond -those direct grants, including access to bootstrap identities. Treat apply -workflows and their approvers as fully trusted within this AWS account. These -are privileged infrastructure roles, not least-privilege application roles. Workspaces separate state, not accounts or failure domains. At larger -scale, use separate accounts and backends and narrow IAM permissions further. +Merging to `main` runs validation only. Applies require a manual workflow run on +`main`, with the environment selected. Prod requires environment approval. +See the [pipeline guide](docs/github-actions.md) for role variables, environment +protection, plan comments, retry commands, and the deployment/state diagram. ## Design decisions @@ -381,69 +180,14 @@ and must be reviewed alongside the controller chart pin. ## Cost -Estimate checked on October 14, 2026, for Linux on-demand instances in the sample -region, 730 hours per month, and the desired node counts. No free-tier credits, -Savings Plans, taxes, or support plan are included. Monitoring runs on the listed -nodes; there are no extra EC2 monitoring servers. - -| Component | Dev / month | Prod / month | Basis | -| --- | ---: | ---: | --- | -| EKS control plane | $73.00 | $73.00 | One cluster at $0.10/hour | -| Worker nodes | $60.74 | $210.24 | Two at $0.0416/hour; three at $0.096/hour | -| Node root disks | $4.80 | $7.20 | 30 GiB gp3 per node at $0.08/GiB-month | -| Monitoring disks | $1.20 | $2.80 | 15 / 35 GiB total gp3 | -| NAT gateways | $32.85 | $98.55 | One / three at $0.045/hour | -| NAT public IPv4 | $3.65 | $10.95 | One / three at $0.005/IP-hour | -| Grafana ALB | $16.43 | $16.43 | One at $0.0225/hour | -| ALB public IPv4 | $7.30 | $10.95 | At least one address per enabled zone: two / three | -| ALB capacity allowance | $5.84 | $5.84 | Budget assumption: average one LCU at $0.008/LCU-hour | -| ACM certificate | $0 | $0 | Non-exportable public certificate used by ALB | -| Route 53 alias record | $0 additional | $0 additional | Existing zone; alias queries to ELB are free | -| **Estimated subtotal** | **$205.80** | **$435.96** | Before usage costs below | - -The existing `hudson.tf` hosted zone costs $0.50/month once, not per environment; -it is already being paid for. Domain registration and renewal are separate. -State storage and S3 requests are usually well under $1/month for this project. -CloudWatch control-plane log ingestion and storage depend on traffic; allow a -few dollars initially and measure it. A reasonable light-use budget is roughly -**$210–225/month for dev** and **$440–465/month for prod**, or **$650–690/month -for both**. These are budget estimates, not limits or a bill guarantee. - -For a short verification run, use these subtotal estimates: - -| Runtime | Dev | Prod | Both | -| --- | ---: | ---: | ---: | -| 3 hours | $0.85 | $1.79 | $2.64 | -| 6 hours | $1.69 | $3.58 | $5.27 | -| 24 hours | $6.77 | $14.33 | $21.10 | - -Allow **$2–3 for a short dev test** or **$8–10 for a day of dev**, including some -startup downloads and logs. Those allowances assume light traffic and successful -cleanup; they are not caps. Timing starts when AWS resources are provisioned, -including time spent waiting for a deployment or teardown. - -Public Grafana adds about **$29.57/month in dev** or **$33.22/month in prod**, -including the one-LCU allowance and minimum public IPs. Capacity and IP counts -can grow under load. Without the LCU allowance, the fixed ingress floor is about -$23.73 / $27.38 per month. A 24-hour dev run is about **$6.77** at the subtotal -rate; prod is about **$14.33**, before variable charges. There is no separate -one-time EKS provisioning fee. Deployment, idle time, and teardown all count. - -Add $0.045 for each GB processed by NAT, plus applicable internet and cross-zone -data transfer. Image pulls during first startup use NAT too. Dev's shared NAT -makes cross-zone charges more likely. Burstable CPU credit use can add EC2 -charges. Extra nodes cost their instance rate and root disk storage. Retained -PVCs, snapshots, or ALBs left after a failed teardown continue to cost money. -If a cluster enters extended Kubernetes support, EKS's price rises to -$0.60/hour, adding $365/month; this configuration opts into STANDARD support. - -Sources: [EKS](https://aws.amazon.com/eks/pricing/), -[EC2 Linux regional rate data](https://b0.p.awsstatic.com/pricing/2.0/meteredUnitMaps/ec2/USD/current/ec2-ondemand-without-sec-sel/US%20East%20(N.%20Virginia)/Linux/index.json), -[EBS](https://aws.amazon.com/ebs/pricing/), -[NAT and public IPv4](https://aws.amazon.com/vpc/pricing/), -[ALB](https://aws.amazon.com/elasticloadbalancing/pricing/), -[ACM](https://aws.amazon.com/certificate-manager/pricing/), and -[Route 53](https://aws.amazon.com/route53/pricing/). +The checked-in estimate is about **$206/month for dev** and **$436/month for +prod**, before usage charges. A 24-hour verification run is about **$6.77 for +dev** or **$14.33 for prod** at the estimated subtotal rates. Allow **$8–10 for +a day of dev** with light traffic and logs. Provisioning and teardown time count. + +See [AWS cost estimates](docs/costs.md) for every component, the cost of public +Grafana, short-run budgets, regional assumptions, and pricing sources. Successful +cleanup is part of the estimate: retained load balancers and disks keep billing. ## Checks @@ -461,33 +205,11 @@ be checked during the first real deployment. Static validation cannot prove thos ## Tear it down -Use the [full cleanup and API audit runbook](docs/cleanup.md) to capture an -inventory before teardown and verify that billable resources are gone afterward. -The brief commands below are only the normal destruction sequence. - -Select the same workspace in both roots. Save anything you want to keep from -Grafana and Prometheus, then destroy monitoring before the platform: - -```bash -cd environments/observability -terraform workspace select dev -terraform destroy -cd .. -terraform workspace select dev -terraform destroy -cd .. -``` - -The ingress is deleted before its controller. Namespace deletion removes any -Prometheus PVCs retained by Helm; the storage class uses a Delete reclaim policy. -Before removing the cluster, confirm that the ALB and monitoring EBS volumes -are gone. A failed controller cleanup can leave AWS resources behind. If that -happens, fix it while the cluster and controller still exist. Do not delete the -cluster first and assume the finalizers will finish later. - -The state bucket, bootstrap roles, and existing hosted zone remain. Keep state -versions until you no longer need recovery. Removing bootstrap protection is an -explicit cleanup task, not part of destroying dev or prod. +Follow the [cleanup and API audit runbook](docs/cleanup.md). Capture the AWS +inventory first, destroy monitoring before the platform, and verify that ALBs, +volumes, NAT gateways and public IPs are gone. The runbook covers failed cleanup +and resources that survive Terraform. Bootstrap state and the existing hosted +zone are retained unless explicitly retired. ## What this is not diff --git a/docs/architecture.svg b/docs/architecture.svg new file mode 100644 index 0000000..77e3cf4 --- /dev/null +++ b/docs/architecture.svg @@ -0,0 +1,4 @@ + + + +
EKS platform reference
EKS platform reference
One environment per workspace · public Grafana, private nodes · logical tiers across availability zones
One environment per workspace · public Grafana, private nodes · logical tiers across availability zones
AWS account
AWS account
Configured AWS region · logical subnet tiers
Configured AWS region · logical subnet tiers
VPC · public/private subnet pair in each availability zone
VPC · public/private subnet pair in each availability zone
Public subnets · load balancer and outbound NAT
Public subnets · load balancer and outbound NAT
Private subnets · managed node group spans all configured zones
Private subnets · managed node group spans all configured zones
Internet clients
Internet clients
Internet / AWS APIs
Internet / AWS APIs
DNS
DNS
Route 53
Existing hudson.tf zone
Route 53Existing hudson.tf zone
TLS
TLS
ACM
DNS-validated certificate
ACMDNS-validated certificate
Ingress
Ingress
Application LB
TLS 443 → HTTP 80
Application LBTLS 443 → HTTP 80
Egress
Egress
NAT gateway(s)
Elastic IP per gateway
NAT gateway(s)Elastic IP per gateway
Control plane
Control plane
Amazon EKS
AWS-managed Kubernetes
Amazon EKSAWS-managed Kubernetes
Compute
Compute
Managed nodes
Private IPs · on-demand
Managed nodesPrivate IPs · on-demand
Dashboard
Dashboard
Grafana
Password login
GrafanaPassword login
Metrics
Metrics
Prometheus
Internal ClusterIP only
PrometheusInternal ClusterIP only
Storage
Storage
Amazon EBS
Two encrypted gp3 PVCs
Amazon EBSTwo encrypted gp3 PVCs
Control-plane logs
Control-plane logs
Logs
Logs
CloudWatch
Retention set in tfvars
CloudWatchRetention set in tfvars
IRSA
Separate CNI, EBS CSI and
Load Balancer Controller roles
IRSASeparate CNI, EBS CSI andLoad Balancer Controller roles
Load Balancer Controller
reconciles Ingress → ALB
Load Balancer Controllerreconciles Ingress → ALB
Dev: 2 zones · 2 nodes · 1 shared NAT   |   Prod: 3 zones · 3 nodes · NAT per zone
Dev: 2 zones · 2 nodes · 1 shared NAT | Prod: 3 zones · 3 nodes · NAT per zone
DNS lookup
DNS lookup
Alias → ALB
Alias → ALB
HTTPS 443
HTTPS 443
Certificate
Certificate
HTTP 80
HTTP 80
PromQL
PromQL
Cluster metrics
Cluster metrics
PVC
PVC
PVC
PVC
Outbound only
Outbound only
Via internet gateway
Via internet gateway
Kubernetes API / node registration
Kubernetes API / node registration
1
1
2
2
3
3
4
4
5
5
6
6
7
7
How to read this page
How to read this page
1
1
Resolve Grafana
The existing hudson.tf zone gets Grafana alias records and ACM validation records. Terraform does not create the hosted zone.
Resolve GrafanaThe existing hudson.tf zone gets Grafana alias records and ACM validationrecords. Terraform does not create the hosted zone.
2
2
Terminate public TLS
Internet clients reach the ALB on HTTPS. HTTP redirects to HTTPS; ACM supplies the certificate.
Terminate public TLSInternet clients reach the ALB on HTTPS. HTTP redirects to HTTPS; ACMsupplies the certificate.
3
3
Forward only Grafana
The controller creates the ALB from Ingress. IP targets forward HTTP to Grafana pods in private subnets.
Forward only GrafanaThe controller creates the ALB from Ingress. IP targets forward HTTP toGrafana pods in private subnets.
4
4
Query internal metrics
Grafana queries Prometheus. Prometheus scrapes cluster metrics and has no public ingress.
Query internal metricsGrafana queries Prometheus. Prometheus scrapes cluster metrics and has nopublic ingress.
5
5
Persist monitoring data
Grafana and Prometheus each have a zonal, encrypted gp3 volume. Monitoring is not replicated.
Persist monitoring dataGrafana and Prometheus each have a zonal, encrypted gp3 volume.Monitoring is not replicated.
6
6
Reach images and APIs
Private nodes use NAT and the VPC internet gateway. Dev shares one NAT; prod has one per zone.
Reach images and APIsPrivate nodes use NAT and the VPC internet gateway. Dev shares one NAT;prod has one per zone.
7
7
Run managed Kubernetes
EKS manages the control plane; one managed node group spans private subnets. API access uses configured public CIDRs and a private endpoint.
Run managed KubernetesEKS manages the control plane; one managed node group spans privatesubnets. API access uses configured public CIDRs and a private endpoint.
Additional details
Control-plane logs go to CloudWatch. CNI, EBS CSI and the ALB controller use separate IRSA roles.

One logical deployment is shown. Dev and prod have separate states and VPCs in the same account.

Page 2
Deployment roles, Terraform root order and S3 state paths.
Additional detailsControl-plane logs go to CloudWatch. CNI, EBS CSI and the ALB controller useseparate IRSA roles.One logical deployment is shown. Dev and prod have separate states and VPCs in thesame account.Page 2Deployment roles, Terraform root order and S3 state paths.
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/cleanup.md b/docs/cleanup.md index a9fd532..654efbc 100644 --- a/docs/cleanup.md +++ b/docs/cleanup.md @@ -1,5 +1,7 @@ # Cleanup and audit +[Repository overview](../README.md) · [Deployment](deployment.md) · [Costs](costs.md) + This procedure removes an environment and checks AWS for resources that still exist. It also covers retiring the shared backend and CI identities when the whole reference is finished. The existing `hudson.tf` hosted zone and domain diff --git a/docs/costs.md b/docs/costs.md new file mode 100644 index 0000000..ec184a3 --- /dev/null +++ b/docs/costs.md @@ -0,0 +1,67 @@ +# AWS cost estimates + +[Repository overview](../README.md) · [Deployment](deployment.md) · [Cleanup](cleanup.md) + +Estimate checked on October 14, 2026, for Linux on-demand instances in the sample +region, 730 hours per month, and the desired node counts. No free-tier credits, +Savings Plans, taxes, or support plan are included. Monitoring runs on the listed +nodes; there are no extra EC2 monitoring servers. + +| Component | Dev / month | Prod / month | Basis | +| --- | ---: | ---: | --- | +| EKS control plane | $73.00 | $73.00 | One cluster at $0.10/hour | +| Worker nodes | $60.74 | $210.24 | Two at $0.0416/hour; three at $0.096/hour | +| Node root disks | $4.80 | $7.20 | 30 GiB gp3 per node at $0.08/GiB-month | +| Monitoring disks | $1.20 | $2.80 | 15 / 35 GiB total gp3 | +| NAT gateways | $32.85 | $98.55 | One / three at $0.045/hour | +| NAT public IPv4 | $3.65 | $10.95 | One / three at $0.005/IP-hour | +| Grafana ALB | $16.43 | $16.43 | One at $0.0225/hour | +| ALB public IPv4 | $7.30 | $10.95 | At least one address per enabled zone: two / three | +| ALB capacity allowance | $5.84 | $5.84 | Budget assumption: average one LCU at $0.008/LCU-hour | +| ACM certificate | $0 | $0 | Non-exportable public certificate used by ALB | +| Route 53 alias record | $0 additional | $0 additional | Existing zone; alias queries to ELB are free | +| **Estimated subtotal** | **$205.80** | **$435.96** | Before usage costs below | + +The existing `hudson.tf` hosted zone costs $0.50/month once, not per environment; +it is already being paid for. Domain registration and renewal are separate. +State storage and S3 requests are usually well under $1/month for this project. +CloudWatch control-plane log ingestion and storage depend on traffic; allow a +few dollars initially and measure it. A reasonable light-use budget is roughly +**$210–225/month for dev** and **$440–465/month for prod**, or **$650–690/month +for both**. These are budget estimates, not limits or a bill guarantee. + +For a short verification run, use these subtotal estimates: + +| Runtime | Dev | Prod | Both | +| --- | ---: | ---: | ---: | +| 3 hours | $0.85 | $1.79 | $2.64 | +| 6 hours | $1.69 | $3.58 | $5.27 | +| 24 hours | $6.77 | $14.33 | $21.10 | + +Allow **$2–3 for a short dev test** or **$8–10 for a day of dev**, including some +startup downloads and logs. Those allowances assume light traffic and successful +cleanup; they are not caps. Timing starts when AWS resources are provisioned, +including time spent waiting for a deployment or teardown. + +Public Grafana adds about **$29.57/month in dev** or **$33.22/month in prod**, +including the one-LCU allowance and minimum public IPs. Capacity and IP counts +can grow under load. Without the LCU allowance, the fixed ingress floor is about +$23.73 / $27.38 per month. A 24-hour dev run is about **$6.77** at the subtotal +rate; prod is about **$14.33**, before variable charges. There is no separate +one-time EKS provisioning fee. Deployment, idle time, and teardown all count. + +Add $0.045 for each GB processed by NAT, plus applicable internet and cross-zone +data transfer. Image pulls during first startup use NAT too. Dev's shared NAT +makes cross-zone charges more likely. Burstable CPU credit use can add EC2 +charges. Extra nodes cost their instance rate and root disk storage. Retained +PVCs, snapshots, or ALBs left after a failed teardown continue to cost money. +If a cluster enters extended Kubernetes support, EKS's price rises to +$0.60/hour, adding $365/month; this configuration opts into STANDARD support. + +Sources: [EKS](https://aws.amazon.com/eks/pricing/), +[EC2 Linux regional rate data](https://b0.p.awsstatic.com/pricing/2.0/meteredUnitMaps/ec2/USD/current/ec2-ondemand-without-sec-sel/US%20East%20(N.%20Virginia)/Linux/index.json), +[EBS](https://aws.amazon.com/ebs/pricing/), +[NAT and public IPv4](https://aws.amazon.com/vpc/pricing/), +[ALB](https://aws.amazon.com/elasticloadbalancing/pricing/), +[ACM](https://aws.amazon.com/certificate-manager/pricing/), and +[Route 53](https://aws.amazon.com/route53/pricing/). diff --git a/docs/delivery.svg b/docs/delivery.svg new file mode 100644 index 0000000..cb6c762 --- /dev/null +++ b/docs/delivery.svg @@ -0,0 +1,4 @@ + + + +
Deployment and state
Deployment and state
GitHub OIDC · selected Terraform workspace · platform before monitoring
GitHub OIDC · selected Terraform workspace · platform before monitoring
GitHub Actions
GitHub Actions
AWS account · bootstrap owns the OIDC provider, roles and state bucket
AWS account · bootstrap owns the OIDC provider, roles and state bucket
Federation
Federation
GitHub OIDC
sts.amazonaws.com
GitHub OIDCsts.amazonaws.com
Plan IAM role
dev-plan / prod-plan
Read-only AWS + state
Plan IAM roledev-plan / prod-planRead-only AWS + state
Apply IAM role
dev / prod
Manual run on main
Apply IAM roledev / prodManual run on main
Platform root
environments/
Network, EKS and IRSA
Platform rootenvironments/Network, EKS and IRSA
Monitoring root
environments/observability/
Helm, ingress, ACM and DNS
Monitoring rootenvironments/observability/Helm, ingress, ACM and DNS
Remote state
Remote state
Amazon S3
Versioning + native lockfiles
Amazon S3Versioning + native lockfiles
Platform workspace state
environments/dev/platform.tfstate
environments/prod/platform.tfstate
Platform workspace stateenvironments/dev/platform.tfstateenvironments/prod/platform.tfstate
Monitoring workspace state
environments/dev/observability.tfstate
environments/prod/observability.tfstate
Monitoring workspace stateenvironments/dev/observability.tfstateenvironments/prod/observability.tfstate
Infrastructure
Infrastructure
EKS + VPC
AWS API resources
EKS + VPCAWS API resources
Releases
Releases
Helm / Kubernetes
Monitoring on existing EKS
Helm / KubernetesMonitoring on existing EKS
Settings   terraform.tfvars maps select dev/prod by terraform.workspace. No -var-file.
Settings terraform.tfvars maps select dev/prod by terraform.workspace. No -var-file.
Bootstrap state is separate: bootstrap/terraform.tfstate.
Environment roles cannot read it. No DynamoDB table.
Bootstrap state is separate: bootstrap/terraform.tfstate.Environment roles cannot read it. No DynamoDB table.
OIDC token
OIDC token
Exact environment subject
Exact environment subject
Plan only · no lock writes
Plan only · no lock writes
Plan + apply
Plan + apply
Then, once EKS is ACTIVE
Then, once EKS is ACTIVE
Read/write via selected role
Read/write via selected role
Separate state
Separate state
Objects in one bucket
Objects in one bucket
AWS provider
AWS provider
Helm / Kubernetes providers
Helm / Kubernetes providers
1
1
2
2
3
3
4
4
5
5
How to read this page
How to read this page
1
1
Run checks first
Every PR and main push validates and runs offline tests. Fork PRs receive no AWS session. Merging does not apply.
Run checks firstEvery PR and main push validates and runs offline tests. Fork PRs receive noAWS session. Merging does not apply.
2
2
Exchange an OIDC token
Approved same-repository PR plans use dev-plan/prod-plan. Manual applies use dev/prod, with environment protection.
Exchange an OIDC tokenApproved same-repository PR plans use dev-plan/prod-plan. Manual appliesuse dev/prod, with environment protection.
3
3
Deploy in dependency order
The selected workspace plans the platform before monitoring. A manual apply creates EKS first. A missing cluster defers the read-only monitoring plan.
Deploy in dependency orderThe selected workspace plans the platform before monitoring. A manual applycreates EKS first. A missing cluster defers the read-only monitoring plan.
4
4
Keep states separate
Each workspace has separate platform and monitoring objects in a shared S3 bucket. Applies use native .tflock objects; read-only plans are unlocked.
Keep states separateEach workspace has separate platform and monitoring objects in a shared S3bucket. Applies use native .tflock objects; read-only plans are unlocked.
5
5
Connect after EKS exists
The monitoring root discovers EKS through AWS and authenticates with aws eks get-token. It does not read the platform state.
Connect after EKS existsThe monitoring root discovers EKS through AWS and authenticates with awseks get-token. It does not read the platform state.
Plan feedback
One PR comment per environment is updated with counts, status, commit and workflow link. Raw plan values are not published.

Cleanup reverses the order
Destroy monitoring while controllers can remove the ALB and volumes, then destroy the platform and run the AWS audit.

Scope
IAM apply roles are privileged. Workspaces separate state; they do not isolate AWS accounts.
Plan feedbackOne PR comment per environment is updated with counts, status, commit and workflowlink. Raw plan values are not published.Cleanup reverses the orderDestroy monitoring while controllers can remove the ALB and volumes, then destroythe platform and run the AWS audit.ScopeIAM apply roles are privileged. Workspaces separate state; they do not isolate AWSaccounts.
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..10ecfaf --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,192 @@ +# Deployment + +[Repository overview](../README.md) · [GitHub Actions](github-actions.md) · [Cleanup](cleanup.md) + +## Before you run it + +You need Terraform 1.16.5, AWS CLI v2, kubectl compatible with the configured +Kubernetes version, and an AWS account with room for the nodes, NAT gateways, +and load balancers. Bash, jq, and Python 3 are used by the scripts; the Python code uses only the +standard library. Helm is only +needed if you want to inspect charts yourself; Terraform installs the releases. + +Use an AWS profile or SSO session with permission to bootstrap IAM and S3 and +to provision the platform. Configure a persistent IAM administrator role in +`administrator_arns` before applying; use its IAM ARN, not an STS session ARN. +The cluster does not grant implicit administrator access to its creator. + +The public `hudson.tf` hosted zone must already exist in the same account, with +its nameservers delegated correctly. Set its exact `hosted_zone_id` in bootstrap +and in both environment entries in `environments/observability/terraform.tfvars`. +Terraform selects that ID and checks the zone name and public-zone setting +before adding Grafana alias records and ACM validation records. It does not create or replace +the zone or register the domain. Use unused Grafana hostnames; existing records +need to be imported or moved deliberately before this configuration owns them. + +Review both sets of tfvars. The shipped values use one AWS region; if you change +it, update the AZs, backend region, bootstrap inputs, and GitHub variables as +well. Verify that the configured Kubernetes version is available and in +[standard EKS support](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). +The control plane uses the STANDARD upgrade policy, so AWS can upgrade it at +the end of standard support; schedule explicit upgrades before that happens. + +## Run dev first + +Start each shell block below at the repository root. The commands enter the +relevant Terraform folder and return to the root when finished. They provision +billable AWS resources. Read the [cost breakdown](costs.md) before leaving a +cluster running. + +### Bootstrap state and CI roles + +Authenticate your AWS profile first using your configured login method +(`aws login` or `aws sso login`), then select it for this shell: + +```bash +export AWS_PROFILE=YOUR_PROFILE +aws sts get-caller-identity +cp bootstrap/terraform.tfvars.example bootstrap/terraform.tfvars +``` + +Read the repository value GitHub actually uses in its OIDC subject: + +```bash +gh api repos/OWNER/REPOSITORY/actions/oidc/customization/sub \ + --jq '.sub_claim_prefix | sub("^repo:"; "")' +``` + +Set `github_repository` in the local tfvars file to that exact result. New +repositories use `OWNER@OWNER-ID/REPOSITORY@REPOSITORY-ID`; older repositories +may use `OWNER/REPOSITORY`. Do not infer the value from the repository URL. +The module adds `repo:` and the exact environment to build the trust condition. +An incorrect value causes `Not authorized to perform sts:AssumeRoleWithWebIdentity`. +After correcting it, plan and apply bootstrap again to update the role trusts, +then retry the workflow. This does not require reapplying the EKS platform. +See [GitHub's subject formats](https://docs.github.com/en/actions/reference/security/oidc#immutable-subject-claims). + +Choose a globally unique state bucket and confirm the cluster and role names +match the environment inputs. Confirm that `hosted_zone_id` identifies the zone whose +name servers are delegated for your domain; duplicate hosted-zone names are +allowed in AWS. If this account already has GitHub's OIDC provider, set +`github_oidc_provider_arn` to its ARN. There is only one provider per issuer per +account; this configuration can reuse it. + +```bash +cd bootstrap +terraform init +terraform plan -out=bootstrap.tfplan +terraform apply bootstrap.tfplan +terraform output +cd .. +``` + +Bootstrap starts with local state because the bucket does not exist yet. +Move that state into the bucket once it has been created: + +```bash +cp bootstrap/backend.tf.example bootstrap/backend.tf +cp bootstrap/backend.hcl.example bootstrap/backend.hcl +# Edit bootstrap/backend.hcl with the bucket output and region. +cd bootstrap +terraform init -migrate-state -backend-config=backend.hcl +cd .. +``` + +Confirm the migration prompt. Secure any local backup state left by the +migration. Bootstrap state lives at `bootstrap/terraform.tfstate`; environment +roles cannot read or modify it. Bucket deletion is blocked with `prevent_destroy`. +S3 versioning, encryption, TLS-only access, and public-access blocking are enabled. + +### Initialize the two roots + +```bash +cp environments/backend.hcl.example environments/backend.hcl +cp environments/observability/backend.hcl.example environments/observability/backend.hcl +# Set the same bucket and region in both files. Keep their distinct keys. +cd environments +terraform init -backend-config=backend.hcl +terraform workspace select -or-create dev +terraform workspace select -or-create prod +terraform workspace select dev +cd observability +terraform init -backend-config=backend.hcl +terraform workspace select -or-create dev +terraform workspace select -or-create prod +terraform workspace select dev +cd ../.. +``` + +Create both workspaces using your bootstrap credentials before enabling PR +plans. A read-only plan role cannot initialize an empty workspace state. The +backend uses native S3 locking with `use_lockfile = true`, without DynamoDB. +The [S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) +explains lock permissions and workspace paths. + +| Workspace | Platform state | Monitoring state | +| --- | --- | --- | +| dev | `environments/dev/platform.tfstate` | `environments/dev/observability.tfstate` | +| prod | `environments/prod/platform.tfstate` | `environments/prod/observability.tfstate` | + +### Plan and apply dev + +In `environments/terraform.tfvars`, set `administrator_arns` inside the +`environments.dev` object to your human administrator's persistent IAM ARN: + +```hcl +administrator_arns = { + operator = "REPLACE_WITH_YOUR_PERSISTENT_IAM_ROLE_ARN" +} +``` + +Then apply using that role's AWS session: + +```bash +cd environments +terraform workspace select dev +terraform plan -out=dev.tfplan +terraform apply dev.tfplan +cd observability +terraform workspace select dev +terraform plan -out=dev.tfplan +terraform apply dev.tfplan +cd ../.. +``` + +The second apply waits for Helm, certificate validation, and ALB reconciliation. +An access-denied error is a reason to check your IAM principal and its EKS access +entry. Do not rerun monitoring with an unrelated administrator identity. + +### Open Grafana and check the cluster + +```bash +cd environments +cluster_region=$(terraform output -raw region) +cluster_name=$(terraform output -raw cluster_name) +aws eks update-kubeconfig --region "$cluster_region" --name "$cluster_name" +kubectl get nodes +kubectl -n monitoring get pods,pvc,ingress +cd observability +terraform output -raw grafana_url +kubectl -n monitoring get secret monitoring-grafana \ + -o jsonpath='{.data.admin-password}' | base64 --decode +cd ../.. +``` + +Log in as `admin` with that generated password. The dev URL is +`https://grafana.dev.hudson.tf`; prod uses `https://grafana.prod.hudson.tf`. +HTTP redirects to HTTPS. Anonymous access and account sign-up are disabled. +Keep the password private and change it after first login. Prometheus provides +node, pod, and Kubernetes object metrics; Grafana ships the chart's dashboards. + +The API endpoint is public with IAM authentication, and private access is also +enabled. The sample API CIDRs allow GitHub-hosted runners with changing egress +addresses. For a fixed-egress runner, restrict `public_access_cidrs` in platform +tfvars. `ingress_cidrs` separately controls who can reach Grafana's ALB. +For a long-lived shared Grafana deployment, add SSO before giving more people access. + +### Run prod + +Select `prod` in both roots and configure `environments.prod.administrator_arns` +in the platform tfvars. Use the same plan and apply commands; the selected +workspace supplies the prod settings. Name the saved plans `prod.tfplan`. +Do not copy Terraform code or change modules to deploy prod. diff --git a/docs/eks-platform-reference.drawio b/docs/eks-platform-reference.drawio new file mode 100644 index 0000000..02fbee4 --- /dev/null +++ b/docs/eks-platform-reference.drawio @@ -0,0 +1,546 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/github-actions.md b/docs/github-actions.md new file mode 100644 index 0000000..d56bc42 --- /dev/null +++ b/docs/github-actions.md @@ -0,0 +1,83 @@ +# GitHub Actions + +[Repository overview](../README.md) · [Deployment](deployment.md) · [Cleanup](cleanup.md) + +![GitHub OIDC roles, Terraform roots and separate S3 workspace states](delivery.svg) + +[Open full size](delivery.svg) · [Editable source for both pages](eks-platform-reference.drawio) + +Create GitHub environments `dev-plan`, `prod-plan`, `dev`, and `prod`. Each needs +these environment variables; they are configuration, not stored AWS secrets: + +| Variable | Value | +| --- | --- | +| `AWS_ROLE_ARN` | Corresponding role from bootstrap's `github_role_arns` output | +| `AWS_ACCOUNT_ID` | Account allowed by the AWS credentials action | +| `AWS_REGION` | Region matching tfvars and the state bucket | +| `TF_STATE_BUCKET` | Bootstrap state bucket output | + +Restrict deployment branches on `dev` and `prod` to `main`. Require reviewers +for `prod`. Restrict who can approve same-repository PRs against `dev-plan` and +`prod-plan`: plan roles can read state and Kubernetes Secrets to refresh Helm +releases. The plan environments must allow PR refs, not just `main`. Protect +`main`, require the validate job, and require review for changes to workflows, +Terraform, and IAM policies. Environment branch restrictions are essential: +the OIDC environment subject does not itself prove which branch requested it. + +Every PR runs formatting, validation, and offline plan tests. Same-repository +PRs also plan both environments using the read-only roles. Fork PRs run the +credential-free checks only. No `pull_request_target` execution is used. + +For an environment without a cluster yet, the platform plan runs and the live +monitoring plan is explicitly deferred. Static validation and offline monitoring +plans still run. Once a cluster exists, both live plans run. PR plans are unlocked +because their roles cannot write lock files; they can be stale. Manual applies +create new saved plans under S3 locks and apply those exact files in the same job. +PR plan artifacts are never reused for deployment or published as artifacts. + +Each PR plan updates one bot comment per environment with platform and monitoring +resource counts: add, change, destroy and replace. The comment includes the +commit, job result and workflow link. Failed plans and deferred monitoring are +marked explicitly. Comments contain counts only; raw plan values and JSON are +not posted. Retries update the existing comment for that environment. + +A merge or push to `main` runs validation only. It does not request AWS +credentials or run a live plan or apply. This lets the demo be merged and +maintained without starting billable infrastructure. +Manual runs select one environment and either `plan` or `apply`; they default +to a dev plan. Manual applies require `main`. Deployment jobs serialize per environment and +are never canceled mid-apply. The prod environment approval sits before AWS +credentials are requested. This is not a sequential dev-to-prod promotion gate. + +After the workflow is merged into `main`, use Actions > Terraform > Run workflow +to select the environment and operation, or run: + +```bash +gh workflow run terraform.yml --ref main -f environment=dev -f operation=plan +gh workflow run terraform.yml --ref main -f environment=prod -f operation=apply +``` + +Manual plans use the corresponding `dev-plan` or `prod-plan` environment and +read-only role. Manual applies use `dev` or `prod`, including its approval rules. +To retry a failed PR job before merging, use GitHub's Re-run failed jobs button +or `gh run rerun RUN_ID --failed`. To retry just one environment job, find its +`databaseId` with `gh run view RUN_ID --json jobs --jq '.jobs[] | {name, databaseId}'`, +then run `gh run rerun RUN_ID --job JOB_ID` using that ID. +Plans do not take a state lock, so an active local apply does not block them; +wait until the apply finishes for a reliable result. Applies wait up to five +minutes for the S3 lock. Retry after the other apply finishes if that wait expires. + +CI refuses to apply plans that delete or replace the EKS cluster. Those changes need +monitoring-first cleanup and an AWS audit while its controllers still run; +follow [the cleanup procedure](cleanup.md) before changing the cluster's +identity or other settings that require replacement. Initial creation and +in-place updates run normally. + +The apply roles can manage EC2 and EKS resources in the configured region, +cluster-prefixed IAM roles, and records in the chosen hosted zone. Direct IAM grants target cluster-prefixed roles, and state-object deletion is +not granted; lock files can be deleted. This is not a permissions boundary: +creating roles and attaching policies can allow privilege escalation beyond +those direct grants, including access to bootstrap identities. Treat apply +workflows and their approvers as fully trusted within this AWS account. These +are privileged infrastructure roles, not least-privilege application roles. Workspaces separate state, not accounts or failure domains. At larger +scale, use separate accounts and backends and narrow IAM permissions further. diff --git a/docs/verification.md b/docs/verification.md index 953389a..fc5b4f9 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -88,3 +88,24 @@ The explicit zone ID lookup succeeded, and its four name servers matched the domain's public delegation. No DNS records or other AWS resources were changed by that check. Bootstrap and monitoring use the same selector; the monitoring test also rejects a selected zone belonging to another domain. + +## Architecture diagrams + +The editable source is [eks-platform-reference.drawio](eks-platform-reference.drawio). +It has two pages: AWS topology, and deployment/state flow. The README embeds +[architecture.svg](architecture.svg); the pipeline guide embeds +[delivery.svg](delivery.svg). Both SVGs also contain editable diagram XML. + +When infrastructure or workflow behavior changes, update the source in draw.io +and export both pages from the repository root with draw.io Desktop: + +```bash +drawio -x -f svg -e -u -p 1 --theme auto --embed-svg-fonts false -b 20 \ + -o docs/architecture.svg docs/eks-platform-reference.drawio +drawio -x -f svg -e -u -p 2 --theme auto --embed-svg-fonts false -b 20 \ + -o docs/delivery.svg docs/eks-platform-reference.drawio +``` + +Review both pages at full size and in light and dark themes. Commit the source +and its exports together. The drawings show logical subnet tiers and deployment +order; they do not assert that monitoring is replicated across availability zones.