feat: add Sunray runtime for VM-identity secret injection

[juanmarin-co]

Problem Statement

Operators need to supply Secret Manager values to arbitrary HTTP and background workloads deployed by Sunstone, without modifying workload images, requiring application-specific secret integration, or adding a wrapper to the workload entrypoint.

Fetching values in the operator process requires operator-side access and exposes payloads outside the target VM. Standalone Docker has no Swarm-style secret store and its API does not consume environment files. Putting values into Docker's environment configuration exposes and persists them in container metadata. Mounted files require application support.

The desired behavior is transparent environment injection inside the VM, authorized by the attached VM service account, while preserving Sunstone's existing sequential deployment and container lifecycle guarantees.

Solution

Introduce Sunray, a small opt-in, runc-compatible OCI runtime executable. Sunstone supplies only secret references in Docker configuration and selects the registered sunray runtime for workloads using secrets. Sunray obtains credentials from the VM metadata server, accesses Secret Manager, verifies payload integrity, and injects values into the OCI process environment before invoking the real runtime.

Sunstone remains the operator CLI; Sunbeam remains the HTTP proxy; Sunray is a short-lived VM-side runtime executable, not a server or central control plane. The workload's entrypoint, arguments, user, and signal behavior remain unchanged.

Secret-backed environment variables use Cloud Run-inspired valueFrom.secretKeyRef configuration. Existing literal environment mappings remain accepted. Secret values must not enter Sunstone's operator process, Docker's stored environment configuration, logs, or durable runtime-owned storage.

Support both initial container creation and subsequent exec processes, including Docker health checks where the verified Docker/containerd path permits this. Exec processes must use the same resolved secret versions as the main container process. Restarting the container starts a new secret snapshot and may resolve aliases to newer versions.

User Stories

1. As an operator, I want to reference Secret Manager values in a workload definition, so that I do not put payloads in source control.
2. As an operator, I want the VM's attached service account to authorize fetching, so that my CLI identity does not need secret-access permissions.
3. As an operator, I want payloads to stay inside the target VM, so that deployment does not export them to my workstation or CI runner.
4. As an operator, I want arbitrary existing images to receive secrets as environment variables, so that I do not rebuild images for Sunstone.
5. As an operator, I want original entrypoints and commands preserved, so that secret support does not change how workloads run.
6. As an operator, I want scratch and non-root images supported, so that secret injection does not require a shell or root inside the workload.
7. As an operator, I want literal and secret-backed environment variables in one configuration, so that there is one coherent environment model.
8. As an existing user, I want literal environment mappings to keep working, so that adopting Sunray does not break released configurations.
9. As an operator, I want empty literal values to remain valid, so that omission and an explicitly empty value are distinguishable.
10. As an operator, I want malformed, duplicate, or ambiguous environment entries rejected before deployment, so that configuration mistakes do not partially change my VMs.
11. As an operator, I want short secret names resolved against the workload's GCP project, so that same-project references are concise.
12. As an operator, I want fully qualified secret names supported, so that I can reference secrets in another project.
13. As an operator, I want numeric versions supported, so that deployments can use reproducible secret versions.
14. As an operator, I want aliases such as latest supported with documented snapshot behavior, so that rotation can take effect on a later container start.
15. As an operator, I want each unique reference fetched once per process preparation, so that repeated references do not waste requests or resolve inconsistently.
16. As an operator, I want independent accesses bounded and concurrent, so that fetching several secrets does not unnecessarily serialize startup or overload Secret Manager.
17. As an operator, I want checksum verification, so that corrupted payloads never reach a workload.
18. As an operator, I want unrepresentable environment values rejected, so that NUL bytes or invalid encodings are not silently changed.
19. As an operator, I want missing, denied, or disabled secret versions to prevent the affected process from starting, so that workloads never silently run with incomplete credentials.
20. As an operator, I want secret-fetch deadlines, so that unavailable metadata or Secret Manager cannot hang startup indefinitely.
21. As an operator, I want a useful reference and failure reason in Docker client errors, so that I can diagnose runtime failures through Sunstone.
22. As an operator, I want errors to exclude tokens and payloads, so that operational logs are safe to share.
23. As an operator, I want secrets absent from Docker's stored environment configuration, so that docker inspect does not disclose injected values.
24. As an operator, I want payloads kept out of durable runtime-owned files, so that the runtime does not turn secret fetching into plaintext disk storage.
25. As an operator, I want the original OCI bundle restored after success or failure, so that transient injection does not permanently modify its configuration.
26. As an operator, I want interrupted injection handled explicitly, so that a subsequent runtime invocation cannot unknowingly reuse a stale payload.
27. As an operator, I want container restart to fetch a fresh secret snapshot, so that restarting can apply rotation without modifying image configuration.
28. As an operator, I want reboot recovery, so that Docker restart policies still work after COS reconstructs its stateless configuration.
29. As an operator, I want docker exec to receive the container's secret-backed environment, so that diagnostic processes have the expected credentials.
30. As an operator, I want exec and health-check processes to use the main process's resolved versions, so that rotating an alias does not give processes in one container different credentials.
31. As an operator, I want a failed exec secret fetch to fail that exec only, so that an already-running workload is not stopped.
32. As an operator, I want failed HTTP replacement startup to preserve the previous route and container, so that secret problems do not interrupt existing traffic.
33. As an operator, I want failed background replacement to follow existing rollback eligibility, so that only a previous container stopped by that deployment can be restored.
34. As an operator, I want sequential multi-VM rollout to stop at the first failure and retain partial results, so that secret support does not change the rollout policy.
35. As an operator, I want status and removal to avoid secret access, so that observation and cleanup work even when secret access has been revoked.
36. As an operator, I want ordinary workloads and Sunbeam to keep using the default runtime, so that Sunray is opt-in rather than a host-wide behavioral change.
37. As an operator, I want a clear failure when Sunray is not installed or registered, so that there is no silent fallback to an uninjected container.
38. As an infrastructure owner, I want secret-level IAM and a documented shared-VM trust boundary, so that I can choose appropriate VM identities and workload placement.
39. As an infrastructure owner, I want reproducible COS provisioning and a safe runtime upgrade procedure, so that the executable and registration survive reboot without replacing unrelated Docker settings.

Implementation Decisions

Runtime responsibility and module design

• Build a static Linux executable named Sunray with a registered Docker runtime name of sunray.
• Concentrate meaningful runtime behavior in one deep module, with a small root interface and hidden implementation. The executable is the composition root. Keep external Secret Manager/metadata and runtime-process integrations behind real seams only where needed; explicitly provide collaborators rather than optional fallbacks.
• Use the OCI runtime specification as the data contract. The Go specs-go types may assist validation, but modifications must preserve unknown/vendor fields rather than discarding them through a whole-document typed round trip.
• Sunray acts as a runc-compatible executable selected through Docker; containerd's existing runtime shim invokes it. This is not a Docker volume driver, replacement containerd daemon, or workload entrypoint wrapper.
• Configure a trusted absolute path for the real runtime. Do not carry the POC's environment-variable runtime override into production.
• Preserve arguments, working directory, stdio, explicitly inherited file descriptors, exit status, and appropriate signal propagation. Prevent delegation to Sunray itself.
• Delegate runtime commands that do not require injection without secret fetches. Cover runtime feature discovery and normal start, stop/kill, state, and delete behavior.
• Creation must fail closed if references cannot be validated or fetched. Never silently bypass injection for a secret-enabled supported operation.

Configuration and reference contract

• The YAML adapter remains strict. Canonical environment input is a list with name and exactly one of literal value or valueFrom.secretKeyRef.

• A secret reference separates name (short secret ID or fully qualified secret resource) from key (numeric version or supported alias). A full version resource in name is invalid; the version belongs in key.

• Normalize short names against the workload's GCP project. Require an explicit nonempty version; do not implicitly select latest.

• Accept existing literal string mappings as adapter compatibility input. Reject unknown fields, duplicate names, both value sources, neither source, non-string values, invalid resources, and invalid environment names. An explicitly empty literal is valid.

• Keep literal values and secret references distinguishable in the internal workload model. Payloads never become part of that model in the operator process.

• The prototype established the decision-rich runtime contract below. This is OCI annotation data, not the workload YAML schema:

  runtime: sunray
  annotation: dev.sunstone.secret-env
  annotation value: JSON object mapping environment names to fully qualified Secret Manager version resource names
  

• Pass this annotation through Docker's existing annotation support. Preserve unrelated annotations. Select Sunray only for secret-enabled workloads, and include runtime selection and normalized references in configuration comparison/fingerprinting.

• Secret-backed entries override image-default entries of the same name; the resulting OCI environment must not contain duplicate entries for those names. Literal-versus-secret duplicates in user configuration are rejected.

• Configuration comparison uses references, not fetched payloads. Changing a numeric reference is a configuration change. Rotation behind an unchanged alias alone does not cause a repeated deploy to replace a current container; explicit restart fetches a new snapshot.

Authentication, access, and snapshot behavior

• Authenticate solely through the attached VM service account via the metadata server, not operator credentials or arbitrary ADC credential files. Reuse one short-lived access token across the accesses in one preparation.
• Enable Secret Manager through user-owned infrastructure. Grant roles/secretmanager.secretAccessor at the individual-secret level where practical. Cross-project access requires IAM on the referenced secret.
• Access each unique version reference once within a preparation; bounded concurrent accesses are permitted. Verify CRC32C for every successful response. Do not start any affected process until all required values are ready.
• Use an explicit bounded overall deadline and bounded transient retries. Do not retry permanent denied/missing/checksum/validation failures as transient failures.
• Preserve whitespace, quotes, equals signs, empty values, and newlines. Reject payloads that cannot be represented losslessly in OCI environment strings, including NUL and invalid UTF-8.
• On initial creation, resolve aliases to the returned numeric version names and retain only those non-secret names as a per-container runtime snapshot. Store no payload cache. Clean this metadata on container deletion and failed creation.
• Exec preparation obtains references from that snapshot, not by assuming Docker copies annotations into its separate exec process specification. Verify the actual exec invocation contract on the supported COS stack before implementing parsing.
• Exec and Docker health-check processes use the numeric snapshot for the running main process. They must not independently resolve latest. A revoked or disabled pinned version fails the new exec process; the existing main process continues.
• The initial scope makes referenced names authoritative for runtime-prepared processes; per-exec overrides of secret-backed variables are not supported. Preserve other explicitly supplied exec variables.
• Container restart/recreation refreshes the snapshot. Background stop/start restart and HTTP replacement restart retain their existing distinct lifecycle strategies.
• Numeric versions are recommended for consistent sequential multi-VM deployments. Aliases are independently resolved on each VM start; do not claim a fleet-wide atomic secret snapshot.

Bundle mutation and failure reporting

• Inject only in an ephemeral runtime preparation area. Do not write payloads into Docker container metadata, persistent secret files, or durable runtime-owned state.
• Preserve the original OCI process fields and restore the exact original bundle after the delegated creation or exec operation returns, including failures. Protect temporary content with restrictive permissions and serialize preparation for the same bundle/process where necessary.
• The POC's restore-on-return design is insufficient under SIGKILL or process crash. Add fault-injection tests and explicit interruption/recovery handling. Minimize the plaintext window, document its remaining limits, and prevent silent reuse of stale modified specifications. Do not claim that userspace cleanup can guarantee erasure after SIGKILL.
• Check runtime-owned state after injection; do not assume restoring the bundle erases every copy created by lower layers. Refuse or clearly constrain configurations that would violate the no-durable-payload requirement.
• When preparation fails before real runc is called, write an error-level entry to the supplied runc JSON log using its level, msg, and time fields, and return nonzero. The prototype verified that stderr alone loses the cause in containerd's error propagation.
• Errors may identify the environment name/reference and a safe status/category, but must not include payloads, metadata tokens, authorization headers, arbitrary response bodies, or rewritten process specifications.
• Treat Docker's classification as a Docker runtime failure. The verified stack returns HTTP 500 and a client internal error even when the nested Secret Manager failure was 403 or 404. Do not map Docker IsNotFound to secret absence or promise structured Secret Manager errors across this interface.
• A nonexistent secret without access can produce 403; do not tell users that 403 proves the secret exists.

Sunstone and infrastructure integration

• Extend the existing deployment module and Docker adapter to carry normalized references, select Sunray, and compare runtime/reference configuration. Do not add a second rollout orchestrator.
• Status and removal neither resolve nor print values. Displaying references may be acceptable; payloads are not.
• HTTP candidate secret failure occurs during Docker start, before probing or route switching. Capture safe diagnostics, remove the candidate, preserve the previous route/container, and stop rollout with ordered partial results.
• Background replacement never overlaps versions. On candidate failure, remove it and restore only a previous container that Sunstone found running and stopped during this deployment. If refetching for rollback now fails, report that failure without claiming recovery.
• Never restart a previously crashed/stopped container as a rollback target. Explicit operator restart remains a separate action.
• COS provisioning installs the static executable in a verified executable, persistent location and restores/retains runtime registration across reboot. General writable directories can be mounted noexec; do not assume all persistent directories are executable.
• Merge registration into existing Docker daemon configuration; preserve the default runtime and unrelated settings. Pin and verify executable artifacts. Runtime installation/upgrades may require maintenance and must not be silently performed as a workaround during deployment.
• Keep infrastructure ownership with users. Packaging/provisioning instructions should establish the supported COS/Docker/containerd/runc combination. Do not update the main README for this work.

Testing Decisions

• The user confirmed three seams: Docker Go client through the Sunray executable as the primary seam; the existing deployment module interface; and the existing strict YAML adapter interface. Avoid adding private-helper test surfaces merely to mirror implementation structure.
• Good tests assert externally observable process environments, startup/exec success or failure, diagnostic propagation, unchanged workload behavior, routing continuity, and durable configuration/reference state. They should survive internal parser or adapter refactoring.
• Prefer hermetic metadata/Secret Manager HTTP servers and a controlled delegated-runtime process for repeatable executable tests, plus opt-in real Docker/runc and COS tests for behavior that substitutes cannot establish. Use generated Mockery mocks where mocks at existing external seams are needed.
• Prior art includes deployment module tests for ordered rollout, candidate cleanup, rollback eligibility, and stopping on failure; routing tests exercised through HTTP/ConnectRPC; strict adapter tests; and the throwaway runtime/Docker Go client POCs.
• Verify creation with pinned versions and aliases; multiple entries; deduplication; literal preservation; replacement of image-default variables; empty/multiline values; and rejection of malformed input, NUL, invalid UTF-8, and checksum mismatch.
• Verify missing/denied/disabled versions, API disabled, metadata unavailable, request timeout, and cancellation. Assert no process runs with a partial environment and errors contain no test payload/token.
• Verify the actual Docker split: container metadata creation can succeed before injection; runtime preparation happens at container start. Failed preparation leaves no running candidate and is surfaced at the expected Docker client operation.
• Verify error logging through the runc JSON log, client response/error classification, container state, and cleanup—not only stderr output.
• Verify Docker exec and health-check injection, including how references are recovered, identical pinned version selection across processes, unrelated explicit exec variables, and failure isolation from the main process.
• Verify features, normal lifecycle operations, original entrypoints/arguments/user, scratch images, non-root workloads, signals, inherited descriptors, and exit-code propagation.
• Verify exact bundle restoration after success and delegated failure; unknown OCI/vendor field preservation; concurrency on a shared preparation area; and crash/SIGKILL at each mutation/delegation/cleanup stage.
• Inspect Docker configuration, containerd/runc state, logs, process environment, temporary files, and persistent VM files for known test markers. Explain that absence from inspected files is evidence for that tested stack, not proof of protection from host root or forensic erasure.
• Verify rotated alias refresh on container restart and restart-policy recovery after COS reboot, including restored runtime registration and Sunbeam route restoration.
• Verify a real HTTP deployment with secret-start failure preserves the previous route and container, removes the candidate, and produces no failed requests in a continuous traffic monitor. The earlier live attempt was blocked by SSH authentication and did not establish this guarantee.
• Verify sequential background and HTTP multi-VM failures retain completed results, do not contact later targets, and do not automatically roll back earlier successful targets.

Out of Scope

• Swarm services, Kubernetes, an NRI plugin, a Docker volume plugin, or replacing containerd's runtime-v2 shim.
• Application-specific secret fetching or workload entrypoint wrappers.
• File-mounted secrets, dotenv files, persistent plaintext secret caches, or secret-store provisioning by Sunstone.
• Cross-process SSH key caching, generic SSH reliability work, or concurrent modifying VM rollouts.
• Transparent hot rotation of the environment of an already-running process.
• Fleet-wide atomic alias resolution or secret rotation. Prefer numeric versions when that consistency matters.
• Separate IAM identities per container on a shared VM. VM credentials remain a shared trust boundary unless infrastructure independently blocks metadata access or isolates workloads.
• Automatically enforcing metadata-server firewall isolation, granting project-wide Secret Manager access, or owning IAM/network infrastructure.
• Protection against host root, Docker administrators, or application access to its own environment; guaranteed payload erasure after uncatchable termination.
• Checkpoint/restore and direct runc run secret injection in the initial supported integration. Explicitly document limitations and reject annotated unsupported creation paths rather than silently start without injection.
• Committing, pushing, or publishing a release without explicit user authorization.

Further Notes

The throwaway POC established the following on COS with Docker 27.5.1/API 1.47, containerd 2.0.10, and runc 1.2.9:

• Docker forwards custom annotations into its OCI bundle and selects a registered runc-compatible executable.
• The executable can fetch Secret Manager payloads using the attached VM service account, verify CRC32C, and inject an environment value without changing the image's entrypoint.
• Docker's stored environment configuration lacks injected payloads. The application receives them; ordinary Docker exec does not in the create-only POC.
• Containerd retrieves preparation diagnostics from runc's JSON log. After adding that protocol, missing authorized versions and denied access are visible in Docker client error text, but both are classified as internal Docker errors.
• A numeric missing version of an accessible secret produced 404. A denied secret and a nonexistent secret without access both produced 403.
• Rotating latest and restarting fetched the new value. The runtime container and fetching recovered after VM reboot; cloud-init/PAM/Sunbeam needed time to finish boot.
• Known payload markers were absent from scanned runc/containerd/Docker container files after restoration and present in the application's environment. This does not establish crash safety or protection from privileged inspection.
• A real HTTP failure-safety attempt was interrupted at SSH authentication before Docker and remains unverified. Its 300-request monitor had zero failures, but that is not evidence of secret-failure rollback behavior.

The POC is not production code: it has create-only injection, limited argument parsing, incomplete interruption handling, and a test runtime override. Use its verified contracts as evidence rather than promoting it wholesale. Production exec support and crash behavior require explicit tests before claiming completion.

[juanmarin-co]

Implemented on the current branch in local commit 8a5562b (not pushed). Added Sunray create/exec injection, VM metadata credentials, CRC32C checking, bounded concurrent access/retries, pinned version snapshots, tmpfs-only preparation and recovery journals, strict compatible YAML, runtime/reference fingerprinting, static amd64/arm64 builds, release artifact packaging, and docs/sunray.md.

Validation: full go test -race ./... and go vet ./... passed; static Linux builds passed. Live COS Docker Go client tests passed for non-root creation, exec, Docker health checks, missing-version diagnostics, and absent payloads in Docker environment metadata. Live alias rotation, restart, reboot, and exec version pinning passed. Scanned Docker/containerd/runc/Sunray files contained no test payload marker after restoration. Temporary containers, secrets, binaries, and registration were cleaned up; original Docker configuration was restored and Podinfo returned HTTP 200.

Two-axis self-review (parallel review agents unavailable): Standards review reduced helper duplication and separated create orchestration. Spec review found and fixed recovery-association loss after restoration failure and the journal size limit for large valid bundles; regression tests passed.

Leaving this issue open: live normal-Sunstone HTTP secret-start failure verification was blocked by existing bounded SSH authentication rejection, before Docker. Previous container and persisted routing state were unchanged, but this does not verify the intended failure path. Module-level HTTP failure policy is covered. Uncatchable interruption during delegated creation, orphan-file cleanup, failed restoration, and pre-fetch cancellation are tested; fault injection across every intermediate filesystem mutation and broader scratch/interactive-runtime compatibility are not yet exhaustively verified.