feat: Enterprise Home Snapshot — runtime endpoints and dify-agent adapter - #40996
Merged
Conversation
An empty Home now streams an ordinary 16-byte archive with no entries instead of a 204. Callers no longer need a branch for it: the snapshot is stored, resolved, and restored through the same path as any other, and a consumer cannot mistake an absent snapshot for an empty one.
Workspace content is not logically part of a Home Snapshot, per the agent backend's HomeSnapshotBackend protocol. Enforcing that through the default value of SHELLCTL_HOME_SNAPSHOT_EXCLUDES let any deployment silently opt out of it by setting the variable to something else. SaveHome now always skips the top-level workspace directory. The env var becomes a purely additive list of further excludes and defaults to empty.
SnapshotTimeout was hardcoded to 600s, so tuning it for slow networks, large Home directories, or loaded nodes meant rebuilding the image. SHELLCTL_SNAPSHOT_TIMEOUT now overrides it with a Go duration string. An unparseable or non-positive value fails startup rather than falling back to the default. A non-positive deadline is the dangerous case: it expires before the first write, and since the deadline is what releases the single-operation gate when a peer stalls, every save and restore would return 409 from then on. Both snapshot env vars are parsed and checked in DefaultConfig, which returns an error, so an unusable value yields no config at all and the caller has a single gate. That absorbs Config.Validate, a one-line pass-through to snapshot.ValidateExcludes, and ValidateExcludes itself: exported for this one consumer and called by nothing in its own package, its rule is about the shape of an env value, not about snapshotting. A malformed exclude is already inert in excluded(), which only matches top-level names.
Removed comments explaining the parseHomeSnapshotExcludes function.
This deadline is the runtime defending its own liveness, not a budget for the operation. It exists because the save/restore gate is held for the whole call: if the caller stops reading, the write blocks, the gate is never released, and every later save and restore on that sandbox answers 409 snapshot_busy until the deadline fires. At 600s that wedge lasted nearly ten minutes. 45s puts it just above the 30s its caller now allows, which is the ordering that matters. Above, and the caller aborts first and reports its own timeout while this gate frees shortly after. Equal or below, and the runtime aborts first, the caller reads a stream that ends without an ok trailer, and a slow snapshot is reported as a corrupt archive. SHELLCTL_SNAPSHOT_TIMEOUT still overrides it; only the shipped default moves.
AgentWorkspaceService._client() built its backend client with the factory default of 30s, one layer below the sandbox-gateway's own 30s snapshot budget and below the dify-agent adapter's 35s create-with-ref leg. That inversion let the api client time out before the gateway's structured snapshot_not_found error could win the race, and could orphan a sandbox that was in fact created and restored. Move HOME_SNAPSHOT_CLIENT_TIMEOUT_SECONDS into clients/agent_backend/factory.py as a shared constant and pass it from both AgentHomeSnapshotService._client() and AgentWorkspaceService._client(), so capture and restore carry the same headroom instead of duplicating the literal.
The Gateway's snapshot delete lists the ref's object-storage namespace and deletes what it finds, returning 200 even when nothing is there — it never answers 404 for a missing snapshot. absent_status=404 on EnterpriseHomeSnapshotBackend.delete therefore bought no idempotency; it only masked a route-level 404, which is exactly the symptom of the two snapshot/sandbox path encodings getting swapped. Remove it so a 404 raises _GatewayStatusError and surfaces as BindingDestroyError like any other failure. EnterpriseExecutionBindingBackend._delete_sandbox keeps its own absent_status=404, which is genuine there. Also: rename a test that asserted no delete fires on a failed create-with-ref but was named as if it asserted the opposite, and drop the dead gateway_timeout field from EnterpriseHomeSnapshotBackend (both its methods use snapshot_timeout; only EnterpriseExecutionBindingBackend reads gateway_timeout).
The 45s budget for snapshot-bearing Agent backend calls was a module constant, so an operator whose object store is slower than ours had to edit code to raise it. Move it to a config Field, AGENT_BACKEND_HOME_SNAPSHOT_TIMEOUT_SECONDS, and wire it through the compose env. The default is unchanged at 45.0 — this changes who can set the value, not the value. It still has to stay above the Agent backend's own DIFY_AGENT_ENTERPRISE_SANDBOX_SNAPSHOT_TIMEOUT so the lower layer's structured error wins the race instead of an opaque client timeout, and that constraint now lives in the Field description where an operator reading the config will actually see it. The Workspace client test asserted against the same constant it was built from, which held for any value including a wrong one. It now patches a distinct timeout and asserts that value arrives, so it covers the wiring rather than restating it.
Test-only branch. feat/shellctl-snapshot-endpoints carries the runtime's /v1/snapshot/save and /v1/snapshot/restore handlers; this branch carries the dify-agent adapter and API client that drive them. Neither half is deployable alone, so an end-to-end sandbox run needs both in one image set. Both PRs stay on their own branches; this one exists to build from.
Contributor
Pyrefly Diffbase → PR--- /tmp/pyrefly_base.txt 2026-08-21 10:20:24.838530053 +0000
+++ /tmp/pyrefly_pr.txt 2026-08-21 10:20:10.079365580 +0000
@@ -1938,6 +1938,8 @@
--> tests/unit_tests/configs/test_dify_config.py:38:32
ERROR Unpacked keyword argument `object` is not assignable to parameter `AGENT_BACKEND_STREAM_MAX_RECONNECTS` with type `Decimal | bool | bytes | float | int | str` in function `_IsolatedDifyConfig.__init__` [bad-argument-type]
--> tests/unit_tests/configs/test_dify_config.py:38:32
+ERROR Unpacked keyword argument `object` is not assignable to parameter `AGENT_BACKEND_HOME_SNAPSHOT_TIMEOUT_SECONDS` with type `Decimal | bool | bytes | float | int | str` in function `_IsolatedDifyConfig.__init__` [bad-argument-type]
+ --> tests/unit_tests/configs/test_dify_config.py:38:32
ERROR Unpacked keyword argument `object` is not assignable to parameter `AGENT_BACKEND_BINDING_FILE_DOWNLOAD_TIMEOUT_SECONDS` with type `Decimal | bool | bytes | float | int | str` in function `_IsolatedDifyConfig.__init__` [bad-argument-type]
--> tests/unit_tests/configs/test_dify_config.py:38:32
ERROR Unpacked keyword argument `object` is not assignable to parameter `AGENT_SHELL_ENABLED` with type `Decimal | bool | float | int | str` in function `_IsolatedDifyConfig.__init__` [bad-argument-type]
|
Contributor
Pyrefly Type Coverage
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #40996 +/- ##
==========================================
- Coverage 86.88% 86.86% -0.02%
==========================================
Files 5207 5221 +14
Lines 292214 292973 +759
Branches 58173 58203 +30
==========================================
+ Hits 253879 254496 +617
- Misses 33324 33438 +114
- Partials 5011 5039 +28
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
shellctl keeps its job records and SQLite database under $HOME/.local/share/shellctl, so a save captured them and a restore overwrote the live server's own state while that server was serving the restore — the connection died mid-request and binding creation failed. Turn the unconditional skip into a list of top-level Home entries, workspace plus .local, that caller-supplied excludes add to but cannot subtract from.
…re syntax The configurable excludes were read from SHELLCTL_HOME_SNAPSHOT_EXCLUDES, baking them into the sandbox pod spec at provision time. Move them to the body of POST /v1/snapshot/save so the consumer owns them, and drop the env and its config field. Excludes are now gitignore patterns matched at any depth rather than exact top-level names: globs, ** across segments, / anchoring, trailing-/ for directories, and ! within the caller's own list. Values are taken as given; validation belongs to the consumer. The runtime's own defaults stay hardcoded and stay out of the matcher, so a ! pattern can cancel another caller pattern but can never re-include the Workspace or the runtime state dir. Matching is go-git's gitignore engine, vendored as two verbatim stdlib-only files under Apache-2.0 with provenance recorded. Upstream's dir.go is left out: it reads .gitignore off a billy filesystem, which is the only reason that package needs go-billy, and patterns arrive over HTTP here. No new module dependencies.
GareArc
marked this pull request as ready for review
August 20, 2026 09:47
GareArc
requested review from
QuantumGhost,
crazywoola and
laipz8200
as code owners
August 20, 2026 09:47
dosubot
Bot
added
the
size:XL
Both sides added a client timeout knob in the same spot: main's AGENT_BACKEND_BINDING_FILE_DOWNLOAD_TIMEOUT_SECONDS and this branch's AGENT_BACKEND_HOME_SNAPSHOT_TIMEOUT_SECONDS. They cover different calls, so create_agent_backend_client forwards both. Two changes merged cleanly and broke anyway: - DefaultConfig() returns (*Config, error) on this branch, while main's new service_test.go called the single-value form. - main moved RuntimeLayout.workspace_dir to /workspace and now clears the directory's contents with find -exec rm -rf instead of removing the directory itself, so the snapshot test's assertion no longer matched.
…pture fails
create_for_build_apply caught only DifyAgentNotFoundError, so every other
backend failure escaped the service uncaught and the Apply endpoint answered
{"message":"Internal Server Error","code":"unknown","status":500}. The reason
was intact the whole way up — the gateway's 413 snapshot_size_exceeded reaches
the API client as a DifyAgentHTTPError detail dict — it just never became an
API error.
Catch DifyAgentHTTPError and DifyAgentClientError and raise
AgentHomeSnapshotCreateFailedError with the backend's own message, keeping the
404 -> AgentBuildSandboxNotFoundError mapping ahead of them.
The detail extraction already existed in the Agent App sandbox controller, so
it moves to clients.agent_backend.errors.backend_error_detail and both call it.
Removed unnecessary description about timeout behavior.
BaseHTTPException.data is typed dict[str, Any] | None, so pyrefly rejects direct subscripting.
…nd error The client raises DifyAgentValidationError, a DifyAgentHTTPError subclass, when a 2xx response fails DTO validation. Both call sites treated that as a backend-reported failure: the service surfaced a Pydantic error list as the user-facing message, and the sandbox controller answered with the response's own 2xx status, so a failed read returned HTTP 200 with an error body.
Excluding the whole .local directory dropped $HOME/.local/bin and $HOME/.local/share from every Home Snapshot -- the exact locations the shell layer tells agents to install tooling into. Narrow the default exclude to .local/share/shellctl and match defaults on the full relative path rather than a top-level name.
EACCES and EPERM fell through to ErrMalformed, so a permission failure on the target filesystem answered 400 archive_malformed and told the caller their archive was corrupt when it was fine.
The save endpoint documented a SHELLCTL_HOME_SNAPSHOT_EXCLUDES env var that exists nowhere in the tree; excludes arrive in the request body in gitignore syntax. Also drop the stated timeout default, which had already drifted from the code.
Since #41023 the Workspace lives at /workspace, a sibling of /home/dify rather than a child, so the Home walk can never reach it and the default exclude only ever matched a user's own ~/workspace -- silently dropping it from every snapshot. The runtime state dir is now the sole default.
GareArc
force-pushed
the
feat/esq1-220-integration
branch
from
August 21, 2026 06:39
b9a7885 to
52e1a4e
Compare
The post-create layout script ran with no cwd, so the runtime fell back to $HOME and Landlock granted Home only — leaving the root-level Workspace outside the writable set once #41023 moved it out of Home. Pass the layout so the cwd resolves to the Workspace, as local.py already does. The script's failures went to stderr, which stdio mode captures to a file the control-command path never reads, so every failure surfaced as binding_create_failed with an empty message. Merge stderr into stdout.
The Gateway already rejects an oversized capture with 413 snapshot_size_exceeded, but every Gateway status collapsed into HomeSnapshotCreateError, which the backend reported as 502 and dify-api passed through as 502. Edge proxies replace the body of an origin 502 with their own error page, so the message the Gateway wrote never reached the user who could act on it. Give the size limit its own backend error and carry 413 to the console, leaving genuine infrastructure failures on 502.
BeautyyuYanli
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ESQ1-220.
Summary
Adds Home Snapshot support: capture a sandbox's home directory and restore it into a new one.
dify-agent-runtime) —POST /v1/snapshot/saveand/v1/snapshot/restore, streaming tar+zstd. Save signals success in theX-Snapshot-Statustrailer since the status line is committed before the body. Restore extracts underos.Root, strips setuid, never preserves ownership.dify-agent,api) —EnterpriseHomeSnapshotBackendandhome_snapshot_refon binding creation, driving the sandbox-gateway. Restore is a parameter of binding creation, not its own call, matching the e2b and local backends.The two halves merged from separate branches and touch disjoint trees. #40975 covers the runtime half on its own if that is easier to review.
Checklist
make lint && make type-check(backend) andvp staged(frontend) to appease the lint gods