diff --git a/src/content/docs/aws/cost.mdx b/src/content/docs/aws/cost.mdx index ad6b3b6..2a5a443 100644 --- a/src/content/docs/aws/cost.mdx +++ b/src/content/docs/aws/cost.mdx @@ -1,10 +1,16 @@ --- title: Cost -description: "What an installation costs on AWS: the small part that runs all the time, the runners that run only while workspaces do, and the settings that change it." +description: "What an installation costs on AWS: the part that runs all the time, the runners while workspaces run, the bytes a stopped one keeps, and how to read the bill." reads: - "spin-terraform-aws:variables.tf@v20261027.02" - "spin-terraform-aws:modules/controlplane/variables.tf@v20261027.02" - "spin-terraform-aws:modules/runners/variables.tf@v20261027.02" + - "spin-terraform-aws:modules/controlplane/bucket.tf@v20261027.02" + - "spin-terraform-aws:modules/controlplane/budget.tf@v20261027.02" + - "spin-terraform-aws:bootstrap/cost.tf@v20261027.02" + - "spin:internal/controlplane/domain/autoscale/autoscale.go@v20261114.01" + - "spin:internal/controlplane/domain/finops/rate.go@v20261114.01" + - "spin:internal/controlplane/domain/bill/job.go@v20261114.01" --- import Scenarios from '../../../components/Scenarios.astro'; @@ -12,66 +18,163 @@ import Value from '../../../components/Value.astro'; import { generated, published } from '../../../lib/pricing'; An installation has two kinds of cost: a small one that runs all the time, and runners, which -run only while workspaces do. Prices depend on your region and change; use the -[AWS Pricing Calculator](https://calculator.aws/) with the sizes below. +run while workspaces do and for a while after (`runner_idle_minutes`). Once its runner is given back, +a stopped workspace costs what it keeps in the bucket. +Prices depend on the region and change; every price here says where and when it was read, and +the [AWS Pricing Calculator](https://calculator.aws/) has yours. ## Three months, priced -The module's default machines, on demand, in the regions Spin OS has images in, at the prices -the AWS pricing page listed on {published} (the record was written on {generated}, and the -daily sync rewrites it): +The module's default machines, on demand, in the regions Spin OS has images in. The machines are +at the prices the AWS pricing page listed on {published} (the record was written on {generated}, +and the daily sync rewrites it); disks, addresses, keys and zones at AWS's price lists of +2026-11-19: -Runners are spot by default, and a spot runner costs a fraction of the on-demand hour shown: -the calculator has your region's spot price, and `aws ec2 describe-spot-price-history` today's. -Not in the table: what the volumes bucket holds, at S3 Standard's price per GB a month, and -what leaves the region, at EC2's price per GB - both depend on what your workspaces do, and the -calculator has both. - -## All the time - -| Resource | Default size | -|---|---| -| Control plane | One machine, [`instance_type`](/reference/terraform/controlplane/#instance_type), which also runs the database | -| Proxy | One machine, [`proxy_instance_type`](/reference/terraform/controlplane/#proxy_instance_type), and an elastic IP | -| Route 53 | Two hosted zones, $0.50 a month each | -| S3 | What the volumes, the database's archive, logs and metric backups hold | -| CloudWatch | The flow log and the machines' boot logs, | - -There is no NAT gateway, no load balancer and no interface endpoint. +These are list-price estimates. Runners are spot by default, so a runner usually costs less than +the on-demand hour these use ([Spot](#spot)). Not in the table: what the buckets hold, the +logs, and what leaves the region, at EC2's price per GB; they depend on what your workspaces do. + +## With nobody working + +| What | Default | List price, us-east-1 and us-west-2 (2026-11-19) | +|---|---|---| +| Control plane | One machine, [`instance_type`](/reference/terraform/controlplane/#instance_type), which also runs the database | In the table above | +| Proxy | One machine, [`proxy_instance_type`](/reference/terraform/controlplane/#proxy_instance_type) | In the table above | +| Their disks | gp3: the control plane's [`root_volume_gb`](/reference/terraform/controlplane/#root_volume_gb) and the proxy's 20 GB | $0.08 per GB a month | +| Public IPv4 addresses | Two: the control plane's, and the proxy's elastic IP | $0.005 an hour each | +| KMS keys | Three: the installation's encryption, identity and volume keys | $1 a month per key version | +| Route 53 | Two hosted zones | $0.50 a month each | +| S3 | What the volumes, the database's archive, logs, documents and certificates hold | S3 Standard, $0.023 per GB a month | +| CloudWatch | The flow log and the machines' boot logs, | Cents a month at a few hosts | + +There is no NAT gateway, no load balancer and no interface endpoint: every machine has its own +public address, and the bucket is reached through a gateway endpoint, which is free. + +At list price that is the table's first row, before the buckets and the logs. The one +measurement so far is higher: on a day the module's own install test left an installation +standing in us-east-1, the region's bill was $2.34 (2026-11-01), about $70 over a month. That day's +bill may include the test itself, and it was not broken down: one data point, not a figure for +your installation. Your own is on the [Costs page](#reading-the-bill), +as the standby. ## While workspaces run -Runners are large spot machines with local NVMe ([`instance_types`](/reference/terraform/runners/#instance_types)). They are the largest cost, and they -cost only while they are up: - -- The control plane starts the first runner when a workspace needs one. -- It adds another only when a workspace has waited longer than a runner takes to start. -- It removes them all when nothing has run for [`runner_idle_minutes`](/reference/terraform/root/#runner_idle_minutes), and at - once during `quiet_hours` when nothing is running. - -A runner holds many workspaces, so one runner is often enough for a team. `max_runners` caps -how many can be up. - -A workspace nobody uses is suspended, and so is one that runs past the installation's limit, so -a forgotten workspace does not keep a runner up all night. See -[Workspaces](/concepts/workspaces/). +Runners are the largest cost, and they cost only while they are up. By default a runner is a +spot machine with local NVMe, of 32 to 48 vCPUs: a c8id, m8id or r8id 8xlarge or 12xlarge +([`runner_instance_types`](/reference/terraform/root/#runner_instance_types), +[`runner_vcpus`](/reference/terraform/root/#runner_vcpus)). Each also has a public IPv4 address +and a gp3 root disk. + +[The control plane sizes the runners' group](): + +- It starts the first runner when a workspace waits for one, and adds another only when a + workspace has waited longer than a runner takes to start, up to `max_runners`. +- When no workspace has run, started or waited for + [`runner_idle_minutes`](/reference/terraform/root/#runner_idle_minutes), it empties the group, + and at once during `quiet_hours`. +- While workspaces run on other runners, it gives back a runner that has held nothing only it + has for as long, one at a time. + +[Runners and hosts](/concepts/runners/#scaling-from-zero) has the details. A workspace nobody uses +is suspended, and so is one that runs past the installation's limit, so a forgotten workspace +does not keep a runner up all night ([Workspaces](/concepts/workspaces/)). + +How many workspaces fit on one runner is not measured yet. A workspace's share of a runner is +what it held, not what it used: the control plane divides the runners' cost by the cores and +memory each workspace held while it ran ([The bill](/admin/operations/#the-bill)). Runner +capacity nobody held is shown apart, as idle. + +### Spot + +AWS prices spot by zone and hour. Over the 30 days to 2026-11-19, the default runner types were +at these prices an hour, against the on-demand list: + +| Type | On demand | Spot, us-east-1: median (low to high) | Spot, us-west-2: median (low to high) | +|---|---|---|---| +| c8id.8xlarge | $1.77 | $0.53 ($0.34 to $1.06) | $0.61 ($0.47 to $0.89) | +| c8id.12xlarge | $2.66 | $0.78 ($0.52 to $1.11) | $0.79 ($0.67 to $1.20) | +| m8id.8xlarge | $2.09 | $0.72 ($0.47 to $1.32) | $0.75 ($0.62 to $1.05) | +| r8id.8xlarge | $2.66 | $1.00 ($0.67 to $2.13) | $1.02 ($0.77 to $1.43) | + +The group asks for all six types the defaults match (the four above, and m8id and r8id 12xlarge), with the price-capacity-optimized strategy, in every +zone the installation has a subnet in (up to +[`availability_zones`](/reference/terraform/controlplane/#availability_zones)), and with Capacity +Rebalancing on; a runner it takes away is held for +[`drain_seconds`](/reference/terraform/runners/#drain_seconds) while it empties. On 2026-11-19, +AWS's spot placement score for those six types together was 9 of 10 for one runner in both +regions, and in 5 of the 9 zones asked one zone at a time. For more runners it held in us-west-2 +(9 at 10 and at 20 runners) and fell in us-east-1 (5 at 10, 2 at 20). AWS gives a low score to +any request that names fewer than three types, so ask for the set. Scores change from day to day, +and they say how likely AWS is to start spot capacity, not how long it keeps it. + +```bash +aws ec2 describe-spot-price-history --region us-east-1 --product-descriptions Linux/UNIX \ + --instance-types c8id.8xlarge c8id.12xlarge --start-time 2026-10-20T00:00:00Z +aws ec2 get-spot-placement-scores --region us-east-1 --region-names us-east-1 --target-capacity 1 \ + --instance-types c8id.8xlarge c8id.12xlarge m8id.8xlarge m8id.12xlarge r8id.8xlarge r8id.12xlarge +``` + +When AWS has no spot capacity, the group keeps asking for it. A host class can be set to fall +back instead: the control plane starts the group's new runners on demand for 30 minutes, then asks for spot again; +runners already started on demand stay until they are given back. It is off for the default +class and on only where a class turns it on (Admin → Runners), and each switch is in the audit +log. + +[A runner AWS reclaims suspends its workspaces to the bucket]() +within the notice AWS gives, and they resume on another runner +([When a runner goes away](/concepts/runners/#when-a-runner-goes-away)). Whether a full runner's +memory fits in that notice is not measured yet. + +## A stopped workspace + +Once its runner is gone, a stopped or suspended workspace costs only what it keeps in the +volumes bucket, at S3 Standard's price: $0.023 per GB a month for the first 50 TB, in us-east-1 +and us-west-2 (2026-11-18). A suspended one also keeps its memory checkpoint there, at most its +whole RAM. Clones share their image's layers, so a hundred workspaces from one image store the +image once. Its owner sees what keeping it stopped costs a month, on its page. + +The bucket keeps every object under Object Lock for +[`object_lock_days`](/reference/terraform/controlplane/#object_lock_days) and removes it the day +after. So a deleted workspace is paid for twice over: in the trash, a week by default, with its +disk; then, once it is erased, its bytes, checkpoints included, for about 31 days more by +default. + +## Reading the bill + +The control plane reads the installation's own bill: [the month's costs from Cost Explorer, its +forecast, and its budget if it has one, every six hours](). +A day's cost settles a day or two after it ends. `spin admin costs` shows it, starting with the +standby: the installation's daily cost with nobody working, from the last billed day. + +```bash +spin admin costs +spin admin costs by workspace +``` + +The dashboard's Costs page (Admin) shows the same, and +[each workspace's owner sees its cost and its rate](). +[Jobs, the bill and the store](/admin/operations/#the-bill) explains how the month is shared out. + +The bill is read by the installation's tags, and AWS shows a tag in the bill only once it is a +cost allocation tag. That is the account's to switch on, once, and AWS refuses to before it has +seen the tag on a resource: after the installation has run for a day, apply bootstrap with +`cost_allocation_tags = true`, in one bootstrap per account. Until then the Costs page is empty. + +`monthly_budget_usd` makes an AWS Budget over everything the installation tags, and mails +`budget_emails` at 80% and 100% of what it cost and at 100% of the forecast. Nothing is stopped +when it is passed. ## Settings that change the cost | Setting | Effect | |---|---| -| `runner_idle_minutes` | How long runners stay up after the last workspace stops | +| `runner_idle_minutes` | How long runners stay up once nothing runs, and how long a spare one is kept | | `quiet_hours`, `time_zone` | A window, such as nights, when runners go as soon as nothing runs | | `max_runners` | The most runners at once | | `runner_spot = false` | On-demand runners: more expensive, never reclaimed | -| `runner_instance_types` | Smaller or larger runners | +| `runner_instance_types`, `runner_vcpus` | Smaller or larger runners | | `instance_type`, `proxy_instance_type` | The size of the control plane and the proxy | | `flow_logs`, `log_retention_days` | What is kept in CloudWatch. Variables of the control plane's module, which the root module leaves at their defaults. | - -## Storage - -The volumes bucket keeps every object under Object Lock for [`object_lock_days`](/reference/terraform/controlplane/#object_lock_days), -so a deleted workspace's bytes are paid for until the retention runs out. Clones share their -image's layers, so a hundred workspaces from one image store the image once. +| `monthly_budget_usd` | A budget and its alerts; it stops nothing | diff --git a/src/content/docs/aws/network.mdx b/src/content/docs/aws/network.mdx index ee4996a..3ecc814 100644 --- a/src/content/docs/aws/network.mdx +++ b/src/content/docs/aws/network.mdx @@ -54,8 +54,9 @@ Port 80 stays open to everyone, for Let's Encrypt. ## Why there is no NAT gateway Every machine has a public address in a public subnet, and reaches S3 through the free gateway -endpoint. A NAT gateway costs more per month than the machines that run all the time, plus a -charge for every gigabyte, before any workspace runs. A public address is not an open door: the +endpoint. A NAT gateway is billed by the hour in each zone that has one, plus a charge for every +gigabyte, before any workspace runs: one alone costs most of what the two machines that run all +the time do, and the subnets span several zones. A public address is not an open door: the security groups decide what reaches each machine, and for all but the proxy that is nothing from outside the VPC. diff --git a/src/content/docs/aws/overview.mdx b/src/content/docs/aws/overview.mdx index 1f4b6dd..5d51e30 100644 --- a/src/content/docs/aws/overview.mdx +++ b/src/content/docs/aws/overview.mdx @@ -62,8 +62,8 @@ These are made for you, and each one has a reason: | Every machine is replaced to update, never patched | Its root filesystem is read-only and verified; a machine runs exactly what the release signed. | | The proxy is the only machine the internet reaches | The control plane, the runners and the database take no connection from outside the VPC. | | No SSH on any machine | A shell on the control plane or the proxy is AWS Session Manager, decided by IAM and logged by AWS. A runner takes no Session Manager shell; an administrator's root shell on one goes through the control plane, the runner emptied first and the session recorded ([Operations](/admin/operations/#a-root-shell-on-a-runner)). | -| No NAT gateway | It would cost more than the machines that run all the time. | -| Runners are spot and start from zero | They are the largest cost, and they only cost while a workspace runs. | +| No NAT gateway | One a zone would cost more than the machines that run all the time. | +| Runners are spot and start from zero | They are the largest cost, and they are up only while workspaces run and for `runner_idle_minutes` after. | | Runners join by their IAM role | There is no join token to create, store, rotate or leak. | | No role can grant itself more | A permissions boundary on every role denies IAM writes, parameter writes, and lifting the bucket's lock. | | Telemetry stays in the installation | Logs, traces and metrics are kept on the control plane's machine, over the logs bucket. | diff --git a/src/content/docs/overview/evaluate.md b/src/content/docs/overview/evaluate.md index 6c31af5..9e30765 100644 --- a/src/content/docs/overview/evaluate.md +++ b/src/content/docs/overview/evaluate.md @@ -20,8 +20,8 @@ the processes, back. A workspace's network is closed until a policy opens it. ## What it costs A small part runs all the time: the control plane and the proxy, two machines, and two Route 53 -zones. Runners, the large spot machines workspaces run on, run only while workspaces do, and -are the largest cost. [Cost](/aws/cost/) lists each part with the setting that changes it, and +zones. Runners, the large spot machines workspaces run on, run while workspaces do and stop a while +after, and are the largest cost. [Cost](/aws/cost/) lists each part with the setting that changes it, and what to put into the AWS Pricing Calculator. ## Try it on one Linux machine diff --git a/src/lib/pricing.ts b/src/lib/pricing.ts index 4aabf46..c58c514 100644 --- a/src/lib/pricing.ts +++ b/src/lib/pricing.ts @@ -9,10 +9,18 @@ import pricing from '../data/pricing.json'; const hours = 730; /** Working hours in a month: eight a day, twenty-two days. */ const workingHours = 8 * 22; -// What the installation's smaller parts are billed at, from AWS's pricing pages as of the -// record's date: the public IPv4 address the proxy's elastic IP holds (an hour), and a Route 53 -// hosted zone (a month, and the installation has two). +// What the installation's smaller parts are billed at, from AWS's price lists read on 2026-11-19, +// the same in every region the record prices: a public IPv4 address (an hour; the control plane +// has one and the proxy's elastic IP is the other), gp3 storage (a GB a month; the control plane's +// disk is modules/controlplane's root_volume_gb, 40, and the proxy's is 20 in proxy.tf), a KMS key +// (a key version a month; the installation makes three), and a Route 53 hosted zone (a month, and +// the installation has two). const ipv4Hourly = 0.005; +const addresses = 2; +const gp3Monthly = 0.08; +const diskGB = 40 + 20; +const kmsKeyMonthly = 1; +const kmsKeys = 3; const zoneMonthly = 0.5; const zones = 2; @@ -40,12 +48,13 @@ export function scenarios(): Scenario[] { // The cheapest type a runner may be: what the group's on-demand price is at the least. const runner = pricing.runners[0]; const perRegion = (f: (region: string) => number) => Object.fromEntries(regions.map((r) => [r, f(r)])); - const idle = (r: string) => (price(r, controlPlane) + price(r, proxy) + ipv4Hourly) * hours + zones * zoneMonthly; + const idle = (r: string) => + (price(r, controlPlane) + price(r, proxy) + addresses * ipv4Hourly) * hours + diskGB * gp3Monthly + kmsKeys * kmsKeyMonthly + zones * zoneMonthly; const n = maxRunners; return [ { name: 'Nothing running', - what: `The control plane (${controlPlane}) and the proxy (${proxy}) all month, the proxy's public IPv4 address, and two Route 53 zones. No runner is up.`, + what: `The control plane (${controlPlane}) and the proxy (${proxy}) all month, their ${diskGB} GB of gp3, their two public IPv4 addresses, three KMS keys and two Route 53 zones. No runner is up, and the buckets and logs are left out.`, monthly: perRegion(idle), }, { diff --git a/tools/refgen/figures.txt b/tools/refgen/figures.txt index 79ba449..7cf7993 100644 --- a/tools/refgen/figures.txt +++ b/tools/refgen/figures.txt @@ -22,8 +22,16 @@ admin/recovery.md | five minutes | spin:internal/pgserver/server.go#ArchiveTimeo admin/users.md | seven days | spin:internal/controlplane/settings/registry.go has "Key: SessionDurationHours, Type: spinv1.SettingType_SETTING_TYPE_INT, Default: "168"" admin/users.md | a year | spin:internal/controlplane/server/authserver/apikeys.go#maxAPIKeyLifetime = 8784h aws/cost.mdx | Three months | not a figure: a heading naming the period the cost table prices -aws/cost.mdx | a month | not a figure: the unit of S3's price per GB a month -aws/cost.mdx | a month | not a figure: the unit of Route 53's price per hosted zone +aws/cost.mdx | a month | not a figure: the unit of a price (S3's per GB, gp3's per GB, a KMS key, a Route 53 zone) and of the month the estimates add up to +aws/cost.mdx | an hour | not a figure: the unit of a price (a public IPv4 address, a spot or on-demand instance) +aws/cost.mdx | 20 GB | spin-terraform-aws:modules/controlplane/proxy.tf has "volume_size = 20" +aws/cost.mdx | a day | spin-terraform-aws:bootstrap/cost.tf has "an installation has run for a day" +aws/cost.mdx | 30 days | not held: the window of AWS's spot price history the table was read over, on 2026-11-19 +aws/cost.mdx | 30 minutes | spin:internal/controlplane/domain/autoscale/autoscale.go#OnDemandFor = 30m +aws/cost.mdx | 50 TB | not held: an AWS fact, the first tier of S3 Standard's price +aws/cost.mdx | a week | spin:internal/controlplane/settings/registry.go has "DefaultWorkspaceTrashDays = 7" +aws/cost.mdx | 31 days | spin-terraform-aws:modules/controlplane/bucket.tf has "noncurrent_days = var.object_lock_days + 1" +aws/cost.mdx | six hours | spin:internal/controlplane/domain/bill/job.go#Interval = 6h aws/network.mdx | ten-second | spin-terraform-aws:modules/controlplane/dns.tf has "internal_record_ttl = 10" aws/setup.md | a few minutes | not a figure: an estimate of how long Let's Encrypt issuance takes aws/setup.md | ten minutes | not a figure: an estimate ("About") of the first bring-up time