BuildKit session is killed by the daemon after 10 seconds: `withSession` does not serve `grpc.health.v1.Health`

[davidfurey]

Summary

Every session lib/session.js opens is closed by the daemon 10 seconds later. Builds that need the session after that point fail with:

failed to resolve source metadata for docker.io/library/alpine:3.19:
no active session for 8a1f…: context deadline exceeded

BuildKit healthchecks every session it accepts. dockerode's gRPC server does not implement grpc.health.v1.Health, so every check returns UNIMPLEMENTED, which the monitor counts as a failure. Two consecutive failures and it closes the connection.

Fast builds succeed because nothing needs the session after the first couple of seconds, but this can cause slow builds to fail.

Where it comes from

In moby/buildkit:session/grpc.go every accepted session gets a monitorHealth goroutine:

var defaultHealthCheckConfig = healthCheckConfig{
        interval:              5 * time.Second,
        defaultTimeout:        15 * time.Second,
        failureThreshold:      2,
        successResetThreshold: 1,
}

_, err := healthClient.Check(checkCtx, &grpc_health_v1.HealthCheckRequest{})
...
if err != nil {
        consecutiveFailures++
        if consecutiveFailures >= cfg.failureThreshold {
                closeConn(errors.Wrap(err, "session healthcheck failed fatally"))
                return
        }
}

UNIMPLEMENTED is an err. There is no special case for it. So a server that does not implement Health fails the check at t+5s and again at t+10s, and closeConn fires.

closeConn cancels the session context, which makes Manager.handleConn delete the entry from sm.sessions. Any later Manager.Get(id) then blocks until its deadline and returns no active session for <id>.

Why it usually goes unnoticed

Currently the session is only used to answer Auth.Credentials while the daemon resolves base-image metadata. That normally happens in the first second or two, well before t+10s, so the session being killed is harmless.

It becomes a problem when anything delays the daemon past 10 seconds before or during that resolution. We hit it via testcontainers, which POSTs the whole build context as a tar: with a ~375 MB context the daemon is still ingesting the tar at t+10s, and load metadata then finds no session. With the same Dockerfile and the same daemon but a smaller context it succeeds.

Filtered context	Result
374.9 MB / 33,084 files	FAILED
78.0 MB / 547 files	OK

The successful runs all finish their metadata resolution inside the 10s window.

Anything else that pushes past 10s should reproduce it too - a slow registry, a cold pull, a loaded machine, --platform emulation - so this is not really about context size.

Suggested fix

Register a Health service in withSession alongside Auth:

server.addService(healthService, {
  Check(_call, callback) {
    callback(null, { status: "SERVING" });
  },
  Watch(call) {
    call.write({ status: "SERVING" });
  },
});

@grpc/grpc-js-health-check provides an implementation, or it is a dozen lines and a small .proto in lib/proto/ to do in-tree and avoid the dependency. BuildKit only calls Check.

I'm happy to send a PR for this, would you prefer an inline implementation or the extra @grpc/grpc-js-health-check dependency?

[davidfurey]

I believe this would be resolved by #847