Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
416 changes: 69 additions & 347 deletions README.md

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions docs/architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions docs/cleanup.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Cleanup and audit

[Repository overview](../README.md) · [Deployment](deployment.md) · [Costs](costs.md)

This procedure removes an environment and checks AWS for resources that still
exist. It also covers retiring the shared backend and CI identities when the
whole reference is finished. The existing `hudson.tf` hosted zone and domain
Expand Down
67 changes: 67 additions & 0 deletions docs/costs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# AWS cost estimates

[Repository overview](../README.md) · [Deployment](deployment.md) · [Cleanup](cleanup.md)

Estimate checked on October 14, 2026, for Linux on-demand instances in the sample
region, 730 hours per month, and the desired node counts. No free-tier credits,
Savings Plans, taxes, or support plan are included. Monitoring runs on the listed
nodes; there are no extra EC2 monitoring servers.

| Component | Dev / month | Prod / month | Basis |
| --- | ---: | ---: | --- |
| EKS control plane | $73.00 | $73.00 | One cluster at $0.10/hour |
| Worker nodes | $60.74 | $210.24 | Two at $0.0416/hour; three at $0.096/hour |
| Node root disks | $4.80 | $7.20 | 30 GiB gp3 per node at $0.08/GiB-month |
| Monitoring disks | $1.20 | $2.80 | 15 / 35 GiB total gp3 |
| NAT gateways | $32.85 | $98.55 | One / three at $0.045/hour |
| NAT public IPv4 | $3.65 | $10.95 | One / three at $0.005/IP-hour |
| Grafana ALB | $16.43 | $16.43 | One at $0.0225/hour |
| ALB public IPv4 | $7.30 | $10.95 | At least one address per enabled zone: two / three |
| ALB capacity allowance | $5.84 | $5.84 | Budget assumption: average one LCU at $0.008/LCU-hour |
| ACM certificate | $0 | $0 | Non-exportable public certificate used by ALB |
| Route 53 alias record | $0 additional | $0 additional | Existing zone; alias queries to ELB are free |
| **Estimated subtotal** | **$205.80** | **$435.96** | Before usage costs below |

The existing `hudson.tf` hosted zone costs $0.50/month once, not per environment;
it is already being paid for. Domain registration and renewal are separate.
State storage and S3 requests are usually well under $1/month for this project.
CloudWatch control-plane log ingestion and storage depend on traffic; allow a
few dollars initially and measure it. A reasonable light-use budget is roughly
**$210–225/month for dev** and **$440–465/month for prod**, or **$650–690/month
for both**. These are budget estimates, not limits or a bill guarantee.

For a short verification run, use these subtotal estimates:

| Runtime | Dev | Prod | Both |
| --- | ---: | ---: | ---: |
| 3 hours | $0.85 | $1.79 | $2.64 |
| 6 hours | $1.69 | $3.58 | $5.27 |
| 24 hours | $6.77 | $14.33 | $21.10 |

Allow **$2–3 for a short dev test** or **$8–10 for a day of dev**, including some
startup downloads and logs. Those allowances assume light traffic and successful
cleanup; they are not caps. Timing starts when AWS resources are provisioned,
including time spent waiting for a deployment or teardown.

Public Grafana adds about **$29.57/month in dev** or **$33.22/month in prod**,
including the one-LCU allowance and minimum public IPs. Capacity and IP counts
can grow under load. Without the LCU allowance, the fixed ingress floor is about
$23.73 / $27.38 per month. A 24-hour dev run is about **$6.77** at the subtotal
rate; prod is about **$14.33**, before variable charges. There is no separate
one-time EKS provisioning fee. Deployment, idle time, and teardown all count.

Add $0.045 for each GB processed by NAT, plus applicable internet and cross-zone
data transfer. Image pulls during first startup use NAT too. Dev's shared NAT
makes cross-zone charges more likely. Burstable CPU credit use can add EC2
charges. Extra nodes cost their instance rate and root disk storage. Retained
PVCs, snapshots, or ALBs left after a failed teardown continue to cost money.
If a cluster enters extended Kubernetes support, EKS's price rises to
$0.60/hour, adding $365/month; this configuration opts into STANDARD support.

Sources: [EKS](https://aws.amazon.com/eks/pricing/),
[EC2 Linux regional rate data](https://b0.p.awsstatic.com/pricing/2.0/meteredUnitMaps/ec2/USD/current/ec2-ondemand-without-sec-sel/US%20East%20(N.%20Virginia)/Linux/index.json),
[EBS](https://aws.amazon.com/ebs/pricing/),
[NAT and public IPv4](https://aws.amazon.com/vpc/pricing/),
[ALB](https://aws.amazon.com/elasticloadbalancing/pricing/),
[ACM](https://aws.amazon.com/certificate-manager/pricing/), and
[Route 53](https://aws.amazon.com/route53/pricing/).
4 changes: 4 additions & 0 deletions docs/delivery.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
192 changes: 192 additions & 0 deletions docs/deployment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
# Deployment

[Repository overview](../README.md) · [GitHub Actions](github-actions.md) · [Cleanup](cleanup.md)

## Before you run it

You need Terraform 1.16.5, AWS CLI v2, kubectl compatible with the configured
Kubernetes version, and an AWS account with room for the nodes, NAT gateways,
and load balancers. Bash, jq, and Python 3 are used by the scripts; the Python code uses only the
standard library. Helm is only
needed if you want to inspect charts yourself; Terraform installs the releases.

Use an AWS profile or SSO session with permission to bootstrap IAM and S3 and
to provision the platform. Configure a persistent IAM administrator role in
`administrator_arns` before applying; use its IAM ARN, not an STS session ARN.
The cluster does not grant implicit administrator access to its creator.

The public `hudson.tf` hosted zone must already exist in the same account, with
its nameservers delegated correctly. Set its exact `hosted_zone_id` in bootstrap
and in both environment entries in `environments/observability/terraform.tfvars`.
Terraform selects that ID and checks the zone name and public-zone setting
before adding Grafana alias records and ACM validation records. It does not create or replace
the zone or register the domain. Use unused Grafana hostnames; existing records
need to be imported or moved deliberately before this configuration owns them.

Review both sets of tfvars. The shipped values use one AWS region; if you change
it, update the AZs, backend region, bootstrap inputs, and GitHub variables as
well. Verify that the configured Kubernetes version is available and in
[standard EKS support](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).
The control plane uses the STANDARD upgrade policy, so AWS can upgrade it at
the end of standard support; schedule explicit upgrades before that happens.

## Run dev first

Start each shell block below at the repository root. The commands enter the
relevant Terraform folder and return to the root when finished. They provision
billable AWS resources. Read the [cost breakdown](costs.md) before leaving a
cluster running.

### Bootstrap state and CI roles

Authenticate your AWS profile first using your configured login method
(`aws login` or `aws sso login`), then select it for this shell:

```bash
export AWS_PROFILE=YOUR_PROFILE
aws sts get-caller-identity
cp bootstrap/terraform.tfvars.example bootstrap/terraform.tfvars
```

Read the repository value GitHub actually uses in its OIDC subject:

```bash
gh api repos/OWNER/REPOSITORY/actions/oidc/customization/sub \
--jq '.sub_claim_prefix | sub("^repo:"; "")'
```

Set `github_repository` in the local tfvars file to that exact result. New
repositories use `OWNER@OWNER-ID/REPOSITORY@REPOSITORY-ID`; older repositories
may use `OWNER/REPOSITORY`. Do not infer the value from the repository URL.
The module adds `repo:` and the exact environment to build the trust condition.
An incorrect value causes `Not authorized to perform sts:AssumeRoleWithWebIdentity`.
After correcting it, plan and apply bootstrap again to update the role trusts,
then retry the workflow. This does not require reapplying the EKS platform.
See [GitHub's subject formats](https://docs.github.com/en/actions/reference/security/oidc#immutable-subject-claims).

Choose a globally unique state bucket and confirm the cluster and role names
match the environment inputs. Confirm that `hosted_zone_id` identifies the zone whose
name servers are delegated for your domain; duplicate hosted-zone names are
allowed in AWS. If this account already has GitHub's OIDC provider, set
`github_oidc_provider_arn` to its ARN. There is only one provider per issuer per
account; this configuration can reuse it.

```bash
cd bootstrap
terraform init
terraform plan -out=bootstrap.tfplan
terraform apply bootstrap.tfplan
terraform output
cd ..
```

Bootstrap starts with local state because the bucket does not exist yet.
Move that state into the bucket once it has been created:

```bash
cp bootstrap/backend.tf.example bootstrap/backend.tf
cp bootstrap/backend.hcl.example bootstrap/backend.hcl
# Edit bootstrap/backend.hcl with the bucket output and region.
cd bootstrap
terraform init -migrate-state -backend-config=backend.hcl
cd ..
```

Confirm the migration prompt. Secure any local backup state left by the
migration. Bootstrap state lives at `bootstrap/terraform.tfstate`; environment
roles cannot read or modify it. Bucket deletion is blocked with `prevent_destroy`.
S3 versioning, encryption, TLS-only access, and public-access blocking are enabled.

### Initialize the two roots

```bash
cp environments/backend.hcl.example environments/backend.hcl
cp environments/observability/backend.hcl.example environments/observability/backend.hcl
# Set the same bucket and region in both files. Keep their distinct keys.
cd environments
terraform init -backend-config=backend.hcl
terraform workspace select -or-create dev
terraform workspace select -or-create prod
terraform workspace select dev
cd observability
terraform init -backend-config=backend.hcl
terraform workspace select -or-create dev
terraform workspace select -or-create prod
terraform workspace select dev
cd ../..
```

Create both workspaces using your bootstrap credentials before enabling PR
plans. A read-only plan role cannot initialize an empty workspace state. The
backend uses native S3 locking with `use_lockfile = true`, without DynamoDB.
The [S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3)
explains lock permissions and workspace paths.

| Workspace | Platform state | Monitoring state |
| --- | --- | --- |
| dev | `environments/dev/platform.tfstate` | `environments/dev/observability.tfstate` |
| prod | `environments/prod/platform.tfstate` | `environments/prod/observability.tfstate` |

### Plan and apply dev

In `environments/terraform.tfvars`, set `administrator_arns` inside the
`environments.dev` object to your human administrator's persistent IAM ARN:

```hcl
administrator_arns = {
operator = "REPLACE_WITH_YOUR_PERSISTENT_IAM_ROLE_ARN"
}
```

Then apply using that role's AWS session:

```bash
cd environments
terraform workspace select dev
terraform plan -out=dev.tfplan
terraform apply dev.tfplan
cd observability
terraform workspace select dev
terraform plan -out=dev.tfplan
terraform apply dev.tfplan
cd ../..
```

The second apply waits for Helm, certificate validation, and ALB reconciliation.
An access-denied error is a reason to check your IAM principal and its EKS access
entry. Do not rerun monitoring with an unrelated administrator identity.

### Open Grafana and check the cluster

```bash
cd environments
cluster_region=$(terraform output -raw region)
cluster_name=$(terraform output -raw cluster_name)
aws eks update-kubeconfig --region "$cluster_region" --name "$cluster_name"
kubectl get nodes
kubectl -n monitoring get pods,pvc,ingress
cd observability
terraform output -raw grafana_url
kubectl -n monitoring get secret monitoring-grafana \
-o jsonpath='{.data.admin-password}' | base64 --decode
cd ../..
```

Log in as `admin` with that generated password. The dev URL is
`https://grafana.dev.hudson.tf`; prod uses `https://grafana.prod.hudson.tf`.
HTTP redirects to HTTPS. Anonymous access and account sign-up are disabled.
Keep the password private and change it after first login. Prometheus provides
node, pod, and Kubernetes object metrics; Grafana ships the chart's dashboards.

The API endpoint is public with IAM authentication, and private access is also
enabled. The sample API CIDRs allow GitHub-hosted runners with changing egress
addresses. For a fixed-egress runner, restrict `public_access_cidrs` in platform
tfvars. `ingress_cidrs` separately controls who can reach Grafana's ALB.
For a long-lived shared Grafana deployment, add SSO before giving more people access.

### Run prod

Select `prod` in both roots and configure `environments.prod.administrator_arns`
in the platform tfvars. Use the same plan and apply commands; the selected
workspace supplies the prod settings. Name the saved plans `prod.tfplan`.
Do not copy Terraform code or change modules to deploy prod.
Loading
Loading