Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/design/2026-09-30-guest-reconnect-host.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,7 @@ PR146/147/148. No RAM persistence is introduced.
Guest identity and preamble implementation are described in
[the guest-side slice](2026-09-30-guest-reconnect-session.md). Host listener
integration and capability enablement remain separate gates.

The [host stream transport](2026-10-01-guest-reconnect-transport.md) supplies
the bounded proof callback and moves legacy listener admission before worker
creation. Live listener integration remains gated on the complete handoff.
79 changes: 79 additions & 0 deletions docs/design/2026-10-01-guest-reconnect-transport.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Host guest-proof transport and listener admission

This dependent slice connects the [runner authorization port](2026-09-30-guest-reconnect-host.md)
to the [guest proof frames](2026-09-30-guest-reconnect-session.md). It is a transport
prerequisite, not enablement of live reconnection.

## Admission before work

The existing microVM listener now claims its one boot connection inline before
starting a serving goroutine. Later peers are closed inline, without a worker or
per-peer log entry. A guest that holds the first configuration write open cannot
cause an unbounded number of rejected-peer goroutines or log entries. Teardown
still owns and closes the claimed connection; a failed first write does not
release the single-use claim. Boot configuration and legacy behavior are otherwise
unchanged. The listener still refuses every later connection without sending data.

## One admitted stream, one proof

`driver.AuthorizeGuestConnection` supplies the real bounded stream exchange to
`GuestReconnectHost.AuthorizeGuestReconnect`. The driver supplies the session
assignment; guest bytes cannot select it. The helper validates the challenge and
expected session before writing, sends only that challenge, reads one strict
4096-byte outer frame, requires the proof kind and matching attempt ID, and uses
the shared strict proof decoder before returning its signature to the host.
The control plane, not this transport, verifies the signature and authority.

The entire call has a five-second ceiling. The caller deadline and the host's
proof context can shorten it. Cancellation closes blocked peer I/O; failures
close the connection and return zero acceptance with only a fixed refusal code.
Unknown host/provider text becomes `unavailable`. Duplicate proof callbacks,
acceptance without a completed proof, malformed acceptance and late success are
refused. The call remains synchronous; a host violating its cancellation contract
retains its admitted slot rather than creating detached work.

Success returns committed acceptance to the caller and leaves the same connection
open, including any buffered bytes. It does not send the accepted token, a boot
configuration or a refusal payload, and does not attach or replace a relay. This
separates the proof transport from the guarded installation that still must be
implemented. Rebuilding a reader around the raw socket would lose buffered bytes;
the caller must retain the same `relay.Conn`.

## Integration limits

No shipping listener invokes this helper yet and no boot opt-in/capability is
advertised. Relaxing `guestChannel.served` without the complete handoff would
replay configuration or let an unproven peer evict a healthy guest, so that guard
remains intact.

The next integration needs an instance-bound handoff that retains original runner
connection, local handle/boot, placement and accepted epoch through current
configuration delivery and relay installation. It must fence the old relay before
new input/RPC, bound peer admission before launching work, validate recovered
VMM/jail/socket ownership, and resolve cold enrollment/generation ordering.
Guest configuration redemption/readiness must complete before attachments can
interleave with the guest preamble; authorization success alone is insufficient.
Current launch configuration must come through the authenticated control-plane
path, not the cached initial boot token/configuration. Boot-time artifact invariants
and existing-child environment limits from the guest design still apply.

An all-in-one listener change using cached `g.boot` was rejected because it would
violate fresh configuration and single-use token rules. Spawning a goroutine for
all peers and then checking the claim was rejected because admission no longer
bounds allocated work. Keeping stream encoding in callers was rejected because
it would duplicate strict framing, cancellation and attempt correlation rules.

## Verification

Tests pin admission before the next accept, unchanged first-boot behavior,
challenge/proof round trips, invalid scope/attempt/kind/schema, oversized data
without newline, duplicate callbacks, zero authority on refusal, fixed errors,
caller and host cancellation, deadline, and buffered-reader continuity.

The built runner probe now runs the production host port and this transport over
real agent WebSocket and guest TCP streams. Its synthetic guest signs the
challenge; the control-plane fixture verifies that exact signature. Wrong scope,
revoked authority, wrong attempt and oversized proof are refused. It asserts that
no acceptance/configuration bytes reach the guest from this helper. This is
executable transport coverage, not a shipping listener, AF_VSOCK/KVM, enrolled
guest recovery, relay takeover or real coding-agent qualification result.
29 changes: 10 additions & 19 deletions internal/driver/microvm_vsock.go
Original file line number Diff line number Diff line change
Expand Up @@ -368,15 +368,10 @@ func (m *Microvm) openGuestChannel(sessionID, udsPath, listenPath string, cfg ru
return g, nil
}

// acceptGuests keeps accepting for the life of the instance.
//
// It keeps accepting even though exactly one connection is ever SERVED,
// because the alternative is worse in both directions: an accept loop that
// stopped would leave later dials queued in the kernel with nobody to refuse
// them, and a loop that served inline would be wedged for good by the first
// guest that connected and did not read (the boot config can exceed a
// megabyte — see bootConfigWriteTimeout). So each connection gets a goroutine
// of its own, and all but the first are refused in it.
// acceptGuests keeps accepting for the life of the instance. Admission happens
// inline before starting a worker: only the one claimed boot connection may
// allocate a serving goroutine. Refused peers receive no bytes and no per-peer
// log line. A blocked boot write cannot prevent rejection of later peers.
func (m *Microvm) acceptGuests(sessionID string, g *guestChannel) {
for {
c, err := g.listener.Accept()
Expand All @@ -386,11 +381,16 @@ func (m *Microvm) acceptGuests(sessionID string, g *guestChannel) {
}
return
}
if !g.claim(c) {
_ = c.Close()
continue
}
go m.serveGuest(sessionID, g, c)
}
}

// serveGuest hands ONE guest connection its configuration and then hands the
// serveGuest handles the ONE connection already claimed by acceptGuests.
// It sends that guest its configuration and then hands the
// connection to the runner.
//
// The boot configuration is the FIRST frame on the conn, written here before
Expand All @@ -407,15 +407,6 @@ func (m *Microvm) acceptGuests(sessionID string, g *guestChannel) {
// holding it. There is nothing to attach it to, and the claim stays spent:
// this boot has had its one connection.
func (m *Microvm) serveGuest(sessionID string, g *guestChannel, c net.Conn) {
if !g.claim(c) {
// Not an error the operator can act on and not a rarity worth a
// line per occurrence — but it IS somebody in the guest dialling a
// socket that is not theirs, so it is said once per attempt and
// names nothing about the session but its id.
log.Printf("microvm: session %s: refusing a second connection on a control channel that serves one guest per boot", sessionID)
_ = c.Close()
return
}
conn := relay.NetConn(c)
ctx, cancel := context.WithTimeout(context.Background(), bootConfigWriteTimeout)
defer cancel()
Expand Down
112 changes: 112 additions & 0 deletions internal/driver/reconnect_transport.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
package driver

import (
"context"
"encoding/json"
"errors"
"sync/atomic"
"time"

"github.com/tokencanopy/rainier/internal/relay"
"github.com/tokencanopy/rainier/protocol/runner"
)

var errGuestStream = errors.New("unavailable")

// AuthorizeGuestConnection exchanges one bounded challenge/proof on an already
// admitted guest stream using the runner's original-connection authorization
// port. session is the driver's assignment, never data read from the guest.
// Failure closes the stream and returns zero authority with a fixed code.
//
// Success does NOT write accepted/configuration, replace a relay, or transfer
// ownership of c. The caller must fence the original instance/placement/epoch,
// acquire fresh configuration and install the relay before publishing readiness.
// Keep the same Conn for that handoff: it may buffer bytes after the proof.
// There is deliberately no shipping listener caller until those gates exist.
//
// The caller must bound peer admission before invoking this synchronously. Like
// GuestReconnectHost, this helper does not detach a host that ignores context.
func AuthorizeGuestConnection(ctx context.Context, host GuestReconnectHost, session string, c relay.Conn) (runner.GuestReconnectAcceptResponse, error) {
var zero runner.GuestReconnectAcceptResponse
if c == nil {
return zero, errGuestStream
}
success := false
defer func() {
if !success {
_ = c.Close()
}
}()
if host == nil || !guestStreamSession(session) || ctx.Err() != nil {
return zero, errGuestStream
}
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
stop := context.AfterFunc(ctx, func() { _ = c.Close() })
defer stop()
var invoked, proved, invalid atomic.Bool
accepted, err := host.AuthorizeGuestReconnect(ctx, session, func(proofCtx context.Context, challenge runner.GuestReconnectChallenge) (string, error) {
if !invoked.CompareAndSwap(false, true) {
invalid.Store(true)
return "", errGuestStream
}
body, _ := json.Marshal(challenge)
if _, err := runner.DecodeGuestReconnectChallenge(body); err != nil || challenge.SessionID != session {
invalid.Store(true)
return "", errGuestStream
}
// Both the outer transport budget and the host's connection-bound proof
// context must remain live. Neither may extend the other.
pctx, pcancel := context.WithCancel(ctx)
defer pcancel()
pstop := context.AfterFunc(proofCtx, pcancel)
defer pstop()
if proofCtx.Err() != nil {
return "", errGuestStream
}
if relay.WriteGuestReconnectFrame(pctx, c, relay.KindGuestReconnectChallenge, body) != nil {
return "", errGuestStream
}
event, err := relay.ReadGuestReconnectFrame(pctx, c)
if err != nil || event.Kind != relay.KindGuestReconnectProof {
return "", errGuestStream
}
proof, err := runner.DecodeGuestReconnectAcceptRequest(event.Payload)
if err != nil || proof.AttemptID != challenge.AttemptID || pctx.Err() != nil || proofCtx.Err() != nil {
return "", errGuestStream
}
proved.Store(true)
return proof.Signature, nil
})
if err != nil || !proved.Load() || invalid.Load() || ctx.Err() != nil {
return zero, guestStreamError(err)
}
body, _ := json.Marshal(accepted)
accepted, err = runner.DecodeGuestReconnectAcceptResponse(body)
if err != nil || ctx.Err() != nil {
return zero, errGuestStream
}
success = true
return accepted, nil
}

func guestStreamError(err error) error {
if err != nil {
switch err.Error() {
case "invalid", "expired", "fenced":
return errors.New(err.Error())
}
}
return errGuestStream
}
func guestStreamSession(id string) bool {
if len(id) == 0 || len(id) > 256 {
return false
}
for i := range id {
if id[i] < 0x21 || id[i] > 0x7e {
return false
}
}
return true
}
Loading
Loading