env-scheduler

Hub-and-spoke scheduler that puts lower environments (EKS nodegroups, RDS/Aurora, ECS services) to sleep outside business hours across multiple AWS accounts, with a static dashboard for waking an environment on weekends. Nothing here touches Kubernetes objects or ArgoCD state — nodegroups go to zero, pods sit Pending, and everything reschedules on wake.

Layout

bin/env-scheduler.ts        CDK app — reads a JSON config, deploys hub and/or spoke
lib/config.ts               Config schema, loader, validation
lib/hub-stack.ts            Hub: registry, Lambda, schedules, internal ALB board
lib/spoke.ts                EnvSchedulerSpoke construct (role + self-registration)
lib/spoke-stack.ts          Standalone stack wrapper around the construct
lib/index.ts                Library exports for importing into another CDK app
lambda/scheduler/handler.py All scheduling logic
lambda/scheduler/index.html The power board
config/                     Your config (gitignored); *.example.json is committed
examples/                   Spoke config, GitHub workflow, existing-app snippet

Configuration

Driven by one JSON file — no editing bin/. Resolution order: -c config=<path>, then $ENV_SCHEDULER_CONFIG, then config/env-scheduler.json.

Two file shapes, distinguished by whether stages is present:

• Hub config (this repo, config/hub.example.json) — deployed once into shared-services.
• Spoke config (a *-gen3-deployment repo, examples/spoke-config.example.json) — uses the same project / naming / environments / stages shape as infra/proxy/proxy.json and friends, so it drops in beside them.

The only thing not in config is SCHEDULER_API_TOKEN, the board's shared header token — pass it as an env var when deploying the hub.

Validation runs at synth, before anything reaches AWS. It rejects: any stage targeting prod, duplicate or malformed envIds, unknown envKeys, malformed CIDRs, a stage with nothing to scale, and hub.vpcCidr overlapping hub.vpnCidrs (the board VPC's own space vs the VPN client range).

Deploying the hub (once, shared-services account)

cp config/hub.example.json config/env-scheduler.json   # set registration.orgId
export SCHEDULER_API_TOKEN=$(openssl rand -hex 24)
npx cdk deploy EnvSchedulerHub

Note the RegistrationRoleArn and AlbDnsName outputs. The registry table has a fixed name (env-scheduler-registry) so spokes in other accounts can address it without a lookup.

Onboarding a project (from its *-gen3-deployment repo)

Two files in the project's repo, matching the pattern the other components use:

1. infra/env-scheduler/env-scheduler.json — copy examples/spoke-config.example.json, set the accounts, and list one stage per lower environment.
2. .github/workflows/deploy-env-scheduler.yml — a thin caller into this repo's reusable workflow, exactly like deploy-proxy.yml calls gen3-proxy:

jobs:
  deploy-test:
    if: ${{ inputs.target == 'test' }}
    uses: AustralianBioCommons/env-scheduler/.github/workflows/deploy-spoke-reusable.yml@main
    with:
      config_path: infra/env-scheduler/env-scheduler.json
      github_environment: test
      role_to_assume: arn:aws:iam::<acct>:role/github-actions-<project>-test

Running it creates env-scheduler-target-role in that account and registers the stage's environments in the hub registry. accountId comes from the deploying stack, so it can never be typed wrong; setting scheduler.enabled: false, removing the stage, or destroying the stack deregisters it. Runtime state the scheduler owns — lastAction, forcedUpUntil, albSavedActions — is never touched by a deploy.

github_environment selects the stage, so --all is safe: only stages whose envKey matches are synthesized, and only into that one account.

Self-registration needs hub.registration set on the hub. With orgId, any account in the AWS Organization can register, so a new project needs no hub change.

Still manual per environment: tag the nodegroups so wake knows what to restore.

scheduler:normal-min = 1
scheduler:normal-desired = 3

Production is structurally excluded

Three independent guards: the caller workflow offers no prod option and the reusable workflow hard-fails on prod/production/prd; the config validator rejects any stage whose envKey or environment name is production; and the hub Lambda refuses to act on any registry item whose tier isn't nonprod.

Schedule

Defaults: up weekdays 08:00, down weekdays 19:00, Australia/Melbourne (EventBridge Scheduler handles DST). Override via upCron / downCron / timezone props on HubStack.

Weekend wake: the dashboard (or POST /envs/{envId}/wake {"hours": 8}) scales up and creates a one-time EventBridge schedule that scales back down and deletes itself (ActionAfterCompletion: DELETE). "Extend" updates the same schedule; "Sleep now" removes it and scales down immediately. The nightly 19:00 sweep skips envs inside a forced-up window.

Dashboard access (VPN-only, split-tunnel safe)

The board is served by an internal ALB in a VPC your VPN reaches, with the hub Lambda as its only target (serving both the page and the API — no CloudFront, S3, WAF, or API Gateway). This is the only restriction that works with a non-captive/split-tunnel VPN: a public endpoint behind a WAF IP allowlist would see staff traffic egressing from home ISPs, not the VPN, and CloudFront's anycast ranges can't practically be routed into the tunnel.

• By default the stack creates its own minimal VPC (10.77.0.0/24, isolated subnets, no NAT/IGW — zero cost beyond the ALB). Pass vpcId / privateSubnetIds / vpcAzs instead to use an existing VPC.

  Connecting the created VPC to the VPN (one-time, after first deploy):

  1. Request a VPC peering connection from the VPN-landing VPC (wherever your VPN terminates) to the BoardVpcId output; accept it in the hub account.
  2. Redeploy with peeringConnectionId: 'pcx-...' — the stack adds this VPC's return routes to every vpnCidrs entry.
  3. In the VPN-landing VPC's route tables, route BoardVpcCidr (10.77.0.0/24) to the peering connection.
  4. Split-tunnel step: add 10.77.0.0/24 to the VPN's pushed/split-include routes so clients actually send it down the tunnel (AWS Client VPN: add a route + authorization rule; corporate VPN: add to the include list). A Transit Gateway attachment works identically if you already run TGW.

• vpnCidrs: security-group ingress (VPN client range + office CIDRs) — defense in depth on top of the ALB being unroutable from the internet.

• certificateArn + boardDomainName: HTTPS with a friendly name. A public DNS-validated ACM cert for a name that only resolves in your private hosted zone is the standard pattern; point the private-zone record at the AlbDnsName output. Without a cert the listener falls back to HTTP :80.

• The x-scheduler-token header check remains as defense in depth; the Auth0 upgrade path still applies if per-person identity matters.

• The sleep page's staff link uses this internal URL — correctly labelled "VPN required" for anyone reading it from outside.

Operational notes

• RDS 7-day auto-restart: AWS restarts stopped RDS after 7 days; the nightly sweep simply stops it again, so long idle periods stay cheap.
• Aurora Serverless v2: if min ACU > 0, stopping the cluster is what saves money — scaling to zero ACU alone isn't available on older engine versions, so the stop/start path is the right one for seqera-aws-rds-style clusters too, if you ever register them.
• Karpenter: if an env uses Karpenter instead of managed-nodegroup autoscaling, add a NodePool limits.cpu: 0 toggle to the Lambda (small extension — the spoke role would need eks:AccessKubernetesApi + an access entry, or do it via a kubectl CodeBuild step). Managed nodegroups need nothing extra.
• Wake latency: nodes join in ~2–4 min; Aurora start is the slow one (~5–10 min). The Lambda starts RDS first for that reason. The dashboard polls every 60 s so people can watch it come up.
• Sweep runtime: envs are processed sequentially; at ~5–10 s each the 5-minute timeout covers dozens of envs. Parallelise with a fan-out to SQS if the fleet ever grows past that.
• Cost: hub components (Lambda, EventBridge, DynamoDB on-demand, CloudFront) round to well under $1/month. Each env sleeps ~71% of the week (13 h nightly + weekends).

Sleeping-env landing page

The ALB survives scale-down, so at sleep time the Lambda rewrites each env's HTTPS listener — every forwarding rule plus the default action — to a response served from the env's own public domain: "This environment shuts down outside business hours… it wakes automatically next weekday morning." No dependency on the VPN-only power board, so it works for external researchers too; a "Staff: wake it from the power board (VPN required)" link is included for the team, and it deep-links with ?env=<envId> so the board highlights that env.

• Status stays a real 503, so scripts, monitors, and crawlers fail honestly while humans get the readable body.
• ALBs are discovered via the load-balancer-controller tag elbv2.k8s.aws/cluster=<cluster>; set albListenerArns on the registry item to override. Only HTTPS listeners are touched (the :80 redirect stays).
• On wake, originals are restored last, so the sleep page stays up while pods are still scheduling. If the ALB was recreated in between, restore is skipped and the controller's startup reconcile rebuilds the rules — it owns them anyway.
• Requires the AlbSleepRedirect statement in the spoke role (included).
• Note: after wake there's still a short window where rules are restored but pods aren't Ready — visitors may see a plain 503 then. If that matters, delay the ALB restore by a few minutes with a second one-time schedule.

Migrating from the pre-config version

The registry table moved from an auto-generated name to the fixed env-scheduler-registry, so the first hub deploy creates a new empty table. That's fine — the registry is now derived state. Deploy each project's spoke with its config and every environment reappears, with correct account ids. Afterwards, delete the old auto-named EnvSchedulerHub-Registry* tables (including any orphans left behind by earlier cdk destroy runs).

retainRegistry defaults to false. With a fixed table name, RETAIN means a later redeploy collides with the retained orphan — which is exactly how the duplicate tables appeared. Leave it false while iterating; set it true once the setup is stable if you want the extra safety net.