Distributed / Multi-node runners

[phinze]

In order for people to depend on Miren running critical services, they need to know that it can handle nodes crashing or being all together deleted. Not just for survivability, running workloads on multiple nodes adds capacity!

For this first take, we're looking to allow multiple "runner" nodes to be added to a single node running the control plane. Ergo the system will be able to survive the death of these "runner" nodes, but not the node running the control plane.

The Miren philosophy is also that these runners need not be confined to a single region. This will allow people to run their control plane node in a different region than their runners, both to improve the reliability of the control plane and give a lot of flexibility to how the runners can be set up!

Some diagrams to help illustrate the pieces involved:

Current (single node):

  graph TB
      subgraph "Single Node"
          I[Ingress] --> C[Coordinator]
          C --> R[Runner]
          R --> S1[Sandbox]
          R --> S2[Sandbox]
      end

Loading

After (multi-node runners):

  graph TB
      subgraph "Control Plane Node"
          I[Ingress] --> C[Coordinator]
      end

      subgraph "Runner Node 1 (Region A)"
          R1[Runner] --> S1[Sandbox]
          R1 --> S2[Sandbox]
      end

      subgraph "Runner Node 2 (Region B)"
          R2[Runner] --> S3[Sandbox]
          R2 --> S4[Sandbox]
      end

      subgraph "Runner Node 3 (Region C)"
          R3[Runner] --> S5[Sandbox]
          R3 --> S6[Sandbox]
      end

      C --> R1
      C --> R2
      C --> R3

Loading

This feature relates directly to #2 as well, as when the runners are in many regions, you need a global way to direct traffic to the runners.