Worker Node

Worker Node

🚧 This wiki describes DevNet 2.0, soon to be launched. It follows the contracts and dincli on the develop branch; the currently running DevNet may still use older contracts and commands. Pre-launch discussion: #102.

The worker node (din-worker) is DIN's sandbox for untrusted code. Every training, scoring, and aggregation job on the network runs the model owner's Python service code — code the validator did not write and must treat as hostile. Rather than execute it in the trusted dincli process, dincli spawns a short-lived worker container per job, runs the job inside it, and destroys it when the job finishes.

This layer exists whether dincli runs directly on the host or inside a DIN Node container — in the containerized setup, workers launch as siblings of din-node on the host Docker daemon.

The sandbox

Each worker container runs with:

• No network — --network none. The model owner's code cannot phone home, exfiltrate data, or reach the chain.
• No secrets — no wallet, no config, no Docker socket. The trusted control plane stays in dincli/din-node.
• Resource limits — --cpus 2 --memory 4g, so a malicious or buggy job can't starve the host.
• Read-only inputs — the model directory, the job file and installed packages are mounted read-only; only role-specific output directories are writable.
• Operator's UID/GID — output files on the host stay owned by the operator, not root.
• Minimal image — python:3.12-slim; the model owner's dependencies come from their pinned requirements.txt (see below).

The worker is stateless: everything it produces goes to the bind-mounted state directory, nothing of value lives in the container itself. Removing an exited worker is always safe — the next job just spawns a fresh one.

Lifecycle

Workers are created on demand and self-remove (--rm) the moment their job exits:

dincli (din-node)                        host Docker daemon
   │  job starts: docker run --rm             │
   │  --network none, cpu/mem limits,         ▼
   │  read-only mounts              din-worker-<role>-model-<id>-gi-<n>-batch-<b>…
   │                                          │ runs model owner's service fn
   │  ◄── results in state dir ───────────────┘ container removed on exit

In steady state docker ps --filter "name=din-" shows only din-node; a visible din-worker-* container means a job is in flight. If the daemon restarts or the host hard-reboots mid-job, an orphaned worker can be left behind — exited workers can be removed freely, but never stop a running worker unless you've confirmed no job should be in flight (killing it can leave half-written output).

Known gap: the dependency-install step

Before a job first runs, the model owner's pinned requirements.txt is installed in a separate container that does have network access — it must, to download packages — and pip can execute build hooks from those packages. So model-owner dependencies are network-isolated during the job, but not during install. This is documented deliberately so the threat model stays accurate; a locked-down/offline install path is future work. Installed packages are cached (cache/dincli-worker/, keyed by a hash of requirements.txt) so this cost is paid once per dependency set, and the cache is always safe to delete.

Future work

• Daemon-level isolation (P4): when the DIN Daemon (dind) lands, it adds its own key/state isolation on top of this container sandbox for code paths running under the daemon's process context.
• Install-step hardening: offline or locked-down dependency installation.

Further reading

• DIN Node — the trusted control plane that spawns workers (and the Docker-socket tradeoff)
• Operator runbook — worker monitoring and cleanup commands
• Manifest & Services — the model-owner code that runs inside workers