clusterd: derive CLUSTERD_PROCESS from pod hostname for distroless by jasonhernandez · Pull Request #36872 · MaterializeInc/materialize

jasonhernandez · 2026-06-02T17:45:57Z

Summary

Distroless clusterd has no shell entrypoint to set CLUSTERD_PROCESS (the process ordinal), which entrypoint.sh derived from the pod hostname before the distroless migration (#36099) removes it. clusterd now derives it itself when running under Kubernetes and the value is unset: the ordinal is the last --delimited segment of the StatefulSet pod name, matching how orchestrator-kubernetes parses the process id back out of pod names.

No signal handling is added. tini (added to the distroless image in #36099) runs as PID 1 and forwards signals, so clusterd is a child process and SIGTERM takes its default action rather than being ignored by PID 1.

Part of the distroless migration: #36099 (image), #36101 (UID/GID gating), #36876 (removes the unused CTP FQDN check, stacks on this). Supersedes the closed #36100.

Merge order

Land before or with #36099. Without it, distroless clusterd has no process ordinal and won't start. Safe to land first: on the current Ubuntu images entrypoint.sh still sets CLUSTERD_PROCESS, and the in-process derivation only runs when it is unset.

Test plan

cargo check -p mz-clusterd
Confirm a distroless clusterd pod in k8s picks up its ordinal and joins the cluster

antiguru · 2026-06-02T19:18:28Z

+                ..Default::default()
+            },
+            EnvVar {
+                name: "CLUSTERD_GRPC_HOST".to_string(),


What does the CLUSTERD_GRPC_HOST env symbol actually do? We only use GRPC for persist pubsub now, and I'm not sure it uses this.

Thanks - digging into it - I don't think it is used. I reworked this draft PR to just add the SIGTERM handler (which should unblock switching clusterd to distroless). #36876 removes some more cruft.

Distroless containers run the binary directly as PID 1 (no tini/shell), so two things from the removed entrypoint.sh are handled here: 1. PID 1 ignores signals with a SIG_DFL disposition, so SIGTERM from Kubernetes is silently dropped. Add an explicit termination-signal handler in clusterd (environmentd already has one). The StatefulSet ordinal (CLUSTERD_PROCESS) is derived in-process from the hostname. 2. entrypoint.sh set CLUSTERD_GRPC_HOST via `hostname --fqdn`. That value fed the CTP handshake's optional `server_fqdn` check: clusterd advertised its FQDN and the controller compared it against the address it dialed (transport.rs::handshake). The check only fired when the value was set, is unrelated to gRPC despite the name, and guards only against reaching a misrouted replica. Rather than re-plumb it for distroless, remove it: drop `--grpc-host`/`CLUSTERD_GRPC_HOST` and the `server_fqdn` field from the CTP `Hello`, along with the now-unused `host_from_address` helper and the `test_handshake_fqdn_mismatch` test. test_metrics byte-count bounds loosened since the handshake shrank. This is the "rip it out" alternative to the FQDN-handling PRs (#36100 in-process resolve, #36872 orchestrator-injected). Pick one. Part of SEC-236 distroless migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Stacks on #36872 (SIGTERM handler). Removes the optional CTP `server_fqdn` handshake check: clusterd advertised its FQDN (via CLUSTERD_GRPC_HOST, set by the now-removed entrypoint.sh) and the controller compared it to the address it dialed. The check only fired when the value was set, is unrelated to gRPC despite the name, and guards only against reaching a misrouted replica. Drops `--grpc-host`/`CLUSTERD_GRPC_HOST`, the `server_fqdn` field from the CTP `Hello`, the `host_from_address` helper, and the `test_handshake_fqdn_mismatch` test. CTP version-gates the handshake, so dropping the field is fine across a release boundary. Part of SEC-236 distroless migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

antiguru

Looks fine. Do we need the SIGTERM handler in other binaries, too? Then I'd recommend extracting common logic into mz-ore.

jasonhernandez · 2026-07-01T18:42:13Z

🔗 Distroless migration — coordination

Part of a set that ships together: #36099 (distroless image), #36101 (UID/GID gating), #36876 (removes the unused CTP FQDN check, stacks on this).

Merge order: land before or with #36099. Without the in-process CLUSTERD_PROCESS derivation here, distroless clusterd has no process ordinal and won't start. Safe to land first: current Ubuntu images still set CLUSTERD_PROCESS via entrypoint.sh, and the in-process path only runs when it is unset.

No signal handling is included: tini (added to the distroless image in #36099) runs as PID 1 and forwards signals, so clusterd is a child process and SIGTERM takes its default action. This supersedes the earlier SIGTERM-handler framing, which assumed clusterd would be PID 1.

Stacks on #36872 (SIGTERM handler). Removes the optional CTP `server_fqdn` handshake check: clusterd advertised its FQDN (via CLUSTERD_GRPC_HOST, set by the now-removed entrypoint.sh) and the controller compared it to the address it dialed. The check only fired when the value was set, is unrelated to gRPC despite the name, and guards only against reaching a misrouted replica. Drops `--grpc-host`/`CLUSTERD_GRPC_HOST`, the `server_fqdn` field from the CTP `Hello`, the `host_from_address` helper, and the `test_handshake_fqdn_mismatch` test. CTP version-gates the handshake, so dropping the field is fine across a release boundary. Part of SEC-236 distroless migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Distroless clusterd has no shell entrypoint to set CLUSTERD_PROCESS (the process ordinal) from the pod hostname. Derive it in clusterd itself when running under Kubernetes and the value is unset, via `process_ordinal_from_hostname`, which takes the trailing `-`-delimited segment of the StatefulSet pod name and returns it only when it parses as a process index. This mirrors how orchestrator-kubernetes recovers the process id from pod names. No in-process signal handler is needed: tini (added to the distroless image in #36099) runs as PID 1 and forwards signals, so clusterd is a child process and SIGTERM takes its default action rather than being ignored by PID 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Stacks on #36872 (SIGTERM handler). Removes the optional CTP `server_fqdn` handshake check: clusterd advertised its FQDN (via CLUSTERD_GRPC_HOST, set by the now-removed entrypoint.sh) and the controller compared it to the address it dialed. The check only fired when the value was set, is unrelated to gRPC despite the name, and guards only against reaching a misrouted replica. Drops `--grpc-host`/`CLUSTERD_GRPC_HOST`, the `server_fqdn` field from the CTP `Hello`, the `host_from_address` helper, and the `test_handshake_fqdn_mismatch` test. CTP version-gates the handshake, so dropping the field is fine across a release boundary. Part of SEC-236 distroless migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jasonhernandez mentioned this pull request Jun 2, 2026

clusterd: SIGTERM handler + in-process gRPC FQDN (Variant A) #36100

Closed

3 tasks

antiguru reviewed Jun 2, 2026

View reviewed changes

jasonhernandez mentioned this pull request Jun 2, 2026

clusterd: remove unused CTP server-FQDN validation #36876

Draft

3 tasks

jasonhernandez force-pushed the jason/distroless-orchestrator-fqdn branch from 0907409 to 2efe3f1 Compare June 2, 2026 20:33

jasonhernandez changed the title ~~clusterd: SIGTERM handler + orchestrator-injected gRPC FQDN (Variant B)~~ clusterd: SIGTERM handler for distroless containers Jun 2, 2026

jasonhernandez marked this pull request as ready for review June 4, 2026 21:40

jasonhernandez requested a review from a team as a code owner June 4, 2026 21:40

antiguru approved these changes Jun 5, 2026

View reviewed changes

Comment thread src/clusterd/src/lib.rs Outdated

def- requested changes Jun 9, 2026

View reviewed changes

Comment thread src/clusterd/src/lib.rs Outdated

jasonhernandez force-pushed the jason/distroless-orchestrator-fqdn branch from 2efe3f1 to ec0fd3c Compare July 1, 2026 18:40

jasonhernandez mentioned this pull request Jul 1, 2026

containers: switch environmentd and clusterd to distroless base image #36099

Draft

5 tasks

jasonhernandez changed the title ~~clusterd: SIGTERM handler for distroless containers~~ clusterd: derive CLUSTERD_PROCESS from pod hostname for distroless Jul 1, 2026

jasonhernandez mentioned this pull request Jul 1, 2026

orchestratord: version-gate pod security context UID/GID for distroless #36101

Draft

3 tasks

jasonhernandez force-pushed the jason/distroless-orchestrator-fqdn branch from ec0fd3c to 0931afb Compare July 1, 2026 22:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

clusterd: derive CLUSTERD_PROCESS from pod hostname for distroless#36872

clusterd: derive CLUSTERD_PROCESS from pod hostname for distroless#36872
jasonhernandez wants to merge 1 commit into
mainfrom
jason/distroless-orchestrator-fqdn

jasonhernandez commented Jun 2, 2026 •

edited

Loading

Uh oh!

antiguru Jun 2, 2026

Uh oh!

jasonhernandez Jun 2, 2026

Uh oh!

antiguru left a comment

Uh oh!

Uh oh!

Uh oh!

jasonhernandez commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jasonhernandez commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Merge order

Test plan

Uh oh!

antiguru Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

jasonhernandez Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

antiguru left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jasonhernandez commented Jul 1, 2026

🔗 Distroless migration — coordination

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jasonhernandez commented Jun 2, 2026 •

edited

Loading