Multi-node Control Plane / Coordinator

[phinze]

The control plane itself becomes a point of failure as the criticality of the cluster grows. We've architected the single node control plane to be ready for resilience but we explicitly haven't finished the work.

When the work is finished, a cluster will be able to have a control plane capable of losing nodes and continuing to run. There should be some minor performance improvements that we're able to deliver as well, but that's not generally the focus.

Before (single coordinator):

Assumes #1 is complete.

graph TB
    C[Coordinator] --> R1[Runner]
    C --> R2[Runner]
    C --> R3[Runner]

Loading

After (multi-node control plane):

graph TB
    subgraph "Control Plane"
        C1[Coordinator]
        C2[Coordinator]
        C3[Coordinator]

        C1 --- C2
        C2 --- C3
        C1 --- C3
    end

    C1 --> R1[Runner]
    C2 --> R2[Runner]
    C3 --> R3[Runner]

Loading