feat(dify-agent): add an opt-in Rust local runtime canary

[pineapple880066]

Important

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue and you have been assigned to it
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Note

This PR is intentionally opened as a draft for architecture and scope
feedback. It is not ready for final review. Issue #39976 contains the full
benchmark methodology and results; the issue is currently unassigned.

Summary

Fixes #39976

This draft adds a protocol-compatible Rust implementation of
dify-agent-runtime behind an explicit, sticky canary in dify-agent. The
existing Go runtime remains the default in the base Compose deployment and
continues to own every legacy, unprefixed resource.

The goal is not to rewrite Dify in Rust. It is to evaluate a narrow replacement
for local shell-runtime process management, where matched-container and full
Agent v2 tests show a measurable latency and memory difference. The additional
Rust toolchain and dependency surface are the main tradeoffs.

What changed

• Added Rust implementations of shellctl, shellctl-runner,
  shellctl-runner-exit, and shellctl-sanitize-pty.
• Preserved the current HTTP/auth contract, tmux supervision, SQLite state,
  output offsets, interactive input, termination/deletion, and Linux Landlock
  behavior.
• Added a locked multi-stage Rust image using the same Python 3.12 production
  base and user-facing Node, pnpm, uv, tmux, CLI, and shell tooling as Go.
• Added an opt-in local-runtime router. The base deployment remains Go-only;
  the Rust service and routing are enabled by a separate Compose overlay.
• New unpinned Bindings use deterministic BLAKE2b canary selection. Rust-owned
  opaque refs carry a rust+ prefix; existing and new Go refs retain the legacy
  unprefixed format.
• Added a bounded Rust /healthz preflight. A failed preflight assigns a new,
  still-unowned Binding to Go before any Rust mutation begins.
• Kept routing sticky for the complete Binding, Workspace, and Home Snapshot
  lifecycle. Mixed-runtime refs are rejected before mutation.
• Deliberately avoided cross-runtime replay after a create, command, snapshot,
  or destroy operation begins because those operations are not generally
  idempotent.
• Added dual-runtime acceptance coverage, real rollout/fallback integration
  tests, Rust CI, Docker build entries, and a paired benchmark harness.
• Backported shared Go-side process-management improvements: combined tmux
  session/pipe startup, cheap completion-artifact polling with lower-frequency
  tmux liveness probes, and reusable sanitizer buffers.

New configuration:

DIFY_AGENT_LOCAL_SANDBOX_RUST_ENDPOINT=
DIFY_AGENT_LOCAL_SANDBOX_RUST_AUTH_TOKEN=
DIFY_AGENT_LOCAL_SANDBOX_RUST_CANARY_PERCENT=0
DIFY_AGENT_LOCAL_SANDBOX_PREFLIGHT_TIMEOUT_SECONDS=1

Rollout and rollback behavior

1. The normal docker-compose.yaml remains Go-only.
2. Enabling docker-compose.rust-runtime.yaml starts an independent Rust
   sandbox. Go and Rust do not share SQLite, tmux, Home, Workspace, or Snapshot
   state.
3. Admission can be staged at 1%, 5%, 25%, or another value before
   increasing to 100%.
4. Setting the percentage to 0% stops new Rust allocations immediately.
   Existing rust+ resources remain pinned to Rust and can drain safely.
5. The Rust endpoint/service must remain available until those resources have
   drained. Removing it earlier would intentionally fail instead of replaying
   stateful operations in Go.
6. After drain, removing the overlay returns to the original direct Go backend
   path. Legacy and canary-era Go refs require no migration.

Automatic fallback is intentionally limited to the pre-mutation health check
for a new unowned Binding.

Full Agent v2 Linux shadow A/B

To test more than the shellctl microbenchmark, I ran the complete Agent v2
/execution-bindings -> /runs -> events/status ->
/workspace/files/read -> binding-destroy path on a self-hosted Linux Dify
host. Temporary exact-image shadow containers had no host ports, received no
production traffic, and did not restart or replace production containers.

The test used a deterministic in-process model to remove provider latency, two
warm-up pairs, and 20 alternating measured Go/Rust pairs. Every sample used a
fresh Binding and Workspace and executed five sequential shell actions,
including 32 KiB file/stdout paths. Every tool result, exit code, terminal
state, marker, and file payload was validated. Confidence intervals use a
10,000-resample paired bootstrap.

Full Agent v2 metric	Go median	Rust median	Go/Rust	Rust reduction (95% CI)
Binding create	187.236 ms	41.883 ms	4.470x	77.6% [70.0%, 79.0%]
Agent run	882.419 ms	290.734 ms	3.035x	67.1% [66.8%, 68.5%]
32 KiB file read	424.267 ms	150.012 ms	2.828x	64.6% [63.4%, 65.4%]
Binding destroy	134.766 ms	36.042 ms	3.739x	73.3% [72.6%, 73.6%]
Full session	1,999.212 ms	670.795 ms	2.980x	66.4% [65.8%, 67.4%]

Rust was faster in all 20 paired full-session samples. Full-session p95 was
2,126.475 ms for Go and 691.185 ms for Rust.

This production-host experiment used Dify source
8e74d7d715c88f1bd0a681cb275a8f7811522c97, while this PR is rebased onto
current main at 48451f5adf1171c673e37f8a3871313eea940826. The PR tip has
separately passed the clean build and compatibility tests listed below, but I
do not present the older deployment timing as an exact-tip benchmark.

The matched Linux-container microbenchmark reported in #39976 independently
measured approximately 4.6-5.9x lower short-job latency and 48.4% lower idle
long-lived-process RSS. These measurements cover local runtime overhead; they
do not represent external LLM/provider, database, vector-search, or general UI
latency.

Validation on the current PR tip

• Rebased onto upstream main at
  48451f5adf1171c673e37f8a3871313eea940826.
• Go runtime: go test -race -count=1 ./... passed; golangci-lint reported
  zero issues.
• Rust runtime in an official rust:1.95-bookworm Linux container: clean
  locked dependency resolution, formatting, Clippy with -D warnings, release
  build, and all 17 direct unit tests passed. The added cases cover state
  transitions, concurrent runner-exit CAS, restart reconciliation, incomplete
  exit markers, UTF-8 windows, atomic writes, sanitization, and helper contracts.
• dify-agent: Ruff passed; scoped BasedPyright reported 0 errors/warnings;
  the rollout/backend suite passed with 73 passed; the complete test suite
  passed with 777 passed, 30 skipped and one existing deprecation warning.
  New coverage includes canary bounds, sticky ownership, malformed refs,
  snapshot routing, no post-mutation replay, and health/close failure ordering.
• Benchmark harness: all 13 unit tests passed, including terminal correctness,
  memory parsing, paired aggregation, deterministic bootstrap, fresh health
  probes, and dynamic Docker port parsing.
• The shared acceptance suite ran against both freshly built Go and Rust
  images. All 35 test groups passed, including exact auth/validation errors,
  env/cwd, non-zero exits, zero/default limits, large UTF-8 output reassembly,
  wait/offset/tail/status/list, terminal conflicts, idempotency, eight-way
  concurrency per runtime, dedicated Rust container restart recovery, and
  Landlock allow/deny/bypass behavior. Go and Rust no-isolation variants also
  passed.
• Real dify-agent rollout integration passed: 100% Rust canary and snapshot
  restore, Go/Rust state isolation, and unavailable-Rust preflight fallback to
  real Go (3 passed, 1 deselected; the deselected case requires E2B).
• The combined base+Rust Compose configuration renders successfully; edited
  workflow YAML parses locally; git diff --check passes.
• All temporary integration containers and state files were removed after the
  run.
• Hosted GitHub Actions on current tip 9bf8ac12de0193fab3e53cb2f5aedb0bb1ec441b: all 23 executed Main CI jobs completed successfully (7 conditional skips), including Python/style, API unit/integration, Web unit/E2E, database migration, VDB, and all four sandbox-runtime jobs. The selected Rust/Go sandbox, API, and Web Docker builds also passed, as did Semantic Pull Request and autofix.ci.

Review scope

This draft intentionally presents the complete safe rollout slice so the
ownership and fallback semantics can be reviewed end to end. If that is too
large, I can split it into:

1. Go process-management improvements plus the paired benchmark/acceptance
   harness; and
2. the Rust implementation, image, sticky router, Compose overlay, and Rust CI.

Screenshots

Not applicable; this changes the local sandbox runtime, routing, deployment,
and CI behavior only.

Checklist

• This change requires a documentation update, included: Dify Document
  • Included in langgenius/dify-docs#942: English, Chinese, and Japanese rollout/rollback guidance, configuration reference, sticky ownership, and preflight fallback behavior.
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods
  • Backend make lint and make type-check passed locally. The template's literal frontend working directory is stale: the staged config now lives in the repository-root vite.config.ts, so the root-equivalent pnpm exec vp staged was used. Because this PR has no frontend changes and no pending staged files, I also ran stronger controlled-path checks: Vite+ formatting over 7,964 files, Oxlint, and ESLint all completed successfully. API and frontend remain unchanged.

From Codex

[crazywoola]

Thank you for taking the time to contribute to Dify. This pull request currently has merge conflicts with the latest main, so we are closing it to keep the pull request queue current. If the change is still relevant, please rebase onto the latest main, resolve the conflicts, and open an updated pull request or let us know if you would like this one reopened. We appreciate your contribution and understanding.