feat(agent): expose ripgrep regex grep options on the grep tool#1309
Open
srtab wants to merge 31 commits into
Open
feat(agent): expose ripgrep regex grep options on the grep tool#1309srtab wants to merge 31 commits into
srtab wants to merge 31 commits into
Conversation
…mmit/MR Move the one true workspace into the daiv-sandbox container under /workspace (repo at /workspace/repo, seeded skills at /workspace/skills, per-run scratchpad at /workspace/tmp) and make the sandbox authoritative for both file and git state. - SandboxFileBackend: a deepagents BackendProtocol that proxies file ops to the sandbox fs/* endpoints over DAIVSandboxClient — one RPC per op, no local mirror, root-configurable and bound at session start. Retires SandboxSyncer and the local patch-apply path. - Dual-mode GitManager (for_sandbox / for_local factories): runs git either via run_commands in /workspace/repo (sandbox-authoritative runs) or as a subprocess against a GitPython clone (sandbox-disabled / repoless runs), behind one async _git runner with exit-code discipline so a failing git command raises instead of folding fatal text into its output. Push failures are classified into GitPushPermissionError / GitPushNetworkError. open_git_manager() builds the right mode from the run's session_id. - Agent-owned publishing: new commit_changes / create_merge_request tools let the agent author its own commit and MR; commit, push and MR-API failures surface as tool-visible error Commands so the agent can recover rather than crashing the run. GitMiddleware nudges the agent to publish (bounded by MAX_GIT_NUDGES) and falls back to a direct daiv publish as a safeguard. GIT_SYSTEM_PROMPT gains an agent_owns_commit variant; raw git mutation in bash stays hard-blocked by policy. - gitlab/gh tool output spills verbatim to a /workspace/tmp scratch file when it exceeds the inline cap. SlashCommandMiddleware split out of the skills middleware. - fs_* client methods and Fs* wire schemas mirrored byte-identically to daiv-sandbox (enforced by the schema-drift consistency test); regenerated schemas.dump.json. Requires the companion daiv-sandbox fs/* endpoints to be deployed.
Agent-level middleware hooks receive a langgraph Runtime, which has no .config attribute (unlike the ToolRuntime used in git_platform.py). Reading runtime.config would AttributeError at runtime on every chat turn. Read the conversation thread_id from the run config contextvar via get_config() instead, falling back to None (reuse disabled) outside a runnable context.
PR-review follow-ups: - Wrap all Redis cache reads/writes (_cache_get/_cache_set/_cache_delete) so a cache outage degrades to a cold create+seed instead of crashing the agent run (the calls sat inside abefore_agent's except BaseException, turning a pure optimization into a hard Redis dependency). - Log when get_config() has no runnable context so silently-disabled reuse is diagnosable. - Fix session_exists docstring (404=>False, any other success=>True; 204 is the sandbox's current answer, not a client-enforced contract) and add cross-service contract pointers; reframe the TTL/reaper coupling as best-effort/self-healing. - Add resilience tests: session_exists propagates non-404; _reuse_warm_session swallows httpx.HTTPError without dropping the mapping; cache read/write outages degrade gracefully; aafter_agent swallows a 404 from close_session.
Register SlashCommandMiddleware only when slash_commands.enabled, matching the conditional-registration pattern used by the sandbox/web/deferred middleware, instead of always registering it and short-circuiting inside abefore_agent. When disabled the middleware is now never instantiated or run rather than no-op'ing every turn.
Move the thread->session cache mapping out of SandboxMiddleware into a new SandboxSessionStore (core/sandbox/session_store.py). This separates the persistence mechanism (key scheme, TTL, best-effort cache I/O) from the reuse orchestration (validate liveness, remember/forget) that stays in the middleware. The store knows nothing of the sandbox client, so the backing store can be swapped without touching the middleware. Behavior is unchanged. Warm-session cache tests move to the store's own unit suite; the middleware tests now drive a fake store and assert orchestration only.
The branch added commit_changes/create_merge_request tools plus a nudge loop so the agent published its own work. Revert to the prior behavior: GitMiddleware commits, pushes, and opens/updates the MR via GitChangePublisher when the agent's turn ends. Keep the sandbox/fs infrastructure intact — the publisher still threads session_id through open_git_manager so it runs git in sandbox or local mode. Remove the git_publish tools, the aafter_model nudge, MAX_GIT_NUDGES, and the agent_owns_commit prompt variant; GIT_SYSTEM_PROMPT returns to the "committing is automatic" contract.
The revert dropped _is_unpublished and let aafter_agent call the publisher unconditionally. The publisher only short-circuits on "dirty OR diff-vs-base", so on an idle follow-up turn of a warm-session chat (clean tree, but an earlier turn's commit is already pushed and ahead of base) it would re-run — burning a diff-to-metadata model call and doing a no-op push — where both main and the pre-revert branch skipped. Restore _is_unpublished as the gate: it adds the has_unpushed check that distinguishes "already live" from "needs pushing", so publishing now fires under the same circumstances as before. It is sandbox-aware publishing logic, not agent-owned, so it belongs after the revert.
The gitlab/gh platform tools previously took an agent-chosen output_file path and wrote oversized output to a sandbox scratch file via a short-lived client, with bespoke inline truncation and auto-eviction. Replace that with an output_to_file boolean that writes the full result through the agent's deepagents filesystem backend, into the same large_tool_results dir the FilesystemMiddleware auto-evicts to. The tools are now closures built inside GitPlatformMiddleware capturing the bound backend; the sandbox backend is wrapped in a DAIVCompositeBackend to carry an artifacts_root under /workspace, and _bind_backend unwraps it to bind the live session. Write failures (raised, backend-rejected, or a missing tool_call_id) surface as agent-visible error strings. Drop the now-orphaned SCRATCH_PATH constant and the legacy truncation helpers.
…on (#1287) * feat(sandbox): own the sandbox transport per-run in set_runtime_ctx * refactor(sandbox): inject run client by construction; state-based session reuse * refactor(git): move GitManager into automation layer, drop lazy import * refactor(git): remove unused GitManager.commit_and_push_changes/checkout * perf(git): batch publish reads into status_snapshot (<=2 round-trips) * refactor(git): fold publish decision into publisher; inject run client into git path * perf(git): run MR create/update and context-file suggestion off the event loop * refactor(git): extract _effective_mr_iid helper for MR resolution * refactor(git): harden publish path and sandbox client teardown Address review feedback on the sandbox-transport-injection branch: - Guard the run-scoped sandbox client teardown in set_runtime_ctx so a transport-close error can't mask the in-flight exception, and always reset the contextvar. - Fold protected_branch_fallback_source into the frozen PublishOutcome, dropping the mutable publisher side-channel read by both callers. - Remove the dead GitManager query methods (is_dirty / get_diff / has_unpushed / remote_branches) superseded by status_snapshot. - Add status_snapshot error-branch tests; re-target the empty-results and no-index hard-error tests onto the surviving helpers. - Fix the "read once" client comments (BaseManager is a second reader), annotate GitMiddleware.sandbox_client, and flag the possibly orphaned container in _session_exists.
…on handle (#1292) * feat(sandbox): add run_commands to SandboxFileBackend * refactor(git): thread bound sandbox backend through git/publish path * refactor(sandbox): run bash through the bound backend; thread it to subagents * test(sandbox): guard that the backend never advertises execution * feat(sandbox): classify bash failures as transient or permanent The bash tool degraded every transport/HTTP error to the same generic "sandbox call failed" string, so the agent could not tell a momentary blip (worth one retry) from a non-recoverable rejection (stop using the tool). Introduce a BashFailure enum that maps httpx errors to TRANSIENT (no response, or a retryable status: 408/425/429/5xx) vs PERMANENT (auth, session-gone, bad-request, not-implemented), and return distinct agent-facing guidance for each. The transient message is byte-stable so the system prompt's "two identical error strings => stop" backstop still fires when a retry fails the same way. Non-httpx failures (malformed 200 body, unbound-backend RuntimeError) are left to propagate as loud wire/programming bugs.
Tool loading called MultiServerMCPClient.get_tools(), which gathers all servers with no timeout, so one server whose handshake hangs blocked the entire toolset build and froze every chat and run. Load each server independently via asyncio.wait_for with a per-server timeout (MCP_TOOL_LOAD_TIMEOUT, default 30s); a server that times out or errors is skipped instead of blocking the others. CancelledError still propagates so outer cancellation is preserved.
…tives Filesystem tools (ls/read_file/grep/glob/write_file/edit_file) now branch on a structured FsError/FsErrorCode from the sandbox instead of free-form strings: a missing path reads as "does not exist" (distinct from an empty dir / no match), reading a directory or writing over an existing file routes the agent to the right tool, and deletes are idempotent. Requires the matching daiv-sandbox release with the structured fs/* error responses. Path directives and subagent prompts are now derived from the run's working directory (REPO_PATH in a sandbox, clone basename on disk) via a single _resolve_working_directory helper, so the main prompt and subagents address the same repo root. SandboxFileBackend._abs normalises the virtual root and prefix-dropped repo slips onto the workspace/repo root.
…fence (#1293) * feat(agent-fs): add TMP_PATH and disk-mode workspace fence permissions * feat(agent-fs): add build_disk_workspace_backend for the unified /workspace namespace * feat(agent): unify graph workspace root to /workspace with disk-only fence * feat(agent-skills): upload global skills under /workspace/skills * feat(agent-subagents): fence subagents to /workspace subtrees in disk mode * refactor(agent): drop unused GLOBAL_SKILLS_* constants after /workspace unification Remove GLOBAL_SKILLS_PATH and GLOBAL_SKILLS_ROUTE from constants.py now that the skills middleware addresses skills via SKILLS_PATH (/workspace/skills) directly. Update skills/services.py and associated tests to classify skill invocations against SKILLS_PATH instead of the removed /skills virtual route. * fix(agent-fs): allow read-back of offloaded-artifact dirs under /workspace fence The disk-mode fence denied everything under /workspace except the three real subtrees, which also blocked reading the offloaded large_tool_results/ conversation_history files. deepagents eviction and git_platform's output_to_file write those through the backend directly (bypassing the fence) and hand the agent the path to read back — so the read-back dead-ended on the /workspace/** deny. Add a read-only carve-out for the artifact dirs ahead of the deny (write stays denied; the agent never writes there itself), mirror it in the explore permissions, and pin the prefixes with a drift-guard test against deepagents' computed values so a framework rename fails loudly.
When the model batches tool calls, LangGraph runs them concurrently, but the sandbox serves one op per session (a Redis lock with a short wait) and returns 409 "Session is busy" to the loser. The file backend's raise_for_status() propagated that 409, aborting the whole run. The agent-facing file tools (ls/read/grep/glob/write/edit + delete) now catch httpx transport/HTTP faults and return a soft, agent-actionable result (transient -> retry once; permanent -> tools unavailable) like the bash tool already does, logging WARNING for transient and ERROR for permanent so genuine faults still reach the logs/Sentry. The bash tool's classifier now also treats 409 as transient (was permanent). The shared transient/permanent classifier lives in core/sandbox/client.py to avoid an import cycle. Pairs with a daiv-sandbox change raising the per-session lock wait so batched ops queue instead of failing fast.
* feat(codebase): add ephemeral GitLab clone token provisioning * fix(codebase): degrade clone-token provisioning on transport errors * feat(codebase): clone GitLab repos with ephemeral project tokens * feat(agent): log git push auth failures for diagnosability * docs: document GitLab ephemeral clone token behavior * fix(codebase): Harden clone-token provisioning failure handling Refine how clone-token provisioning degrades when GitLab cannot mint a project access token: - Negative-cache transient failures (network, 429, 5xx) for 5 minutes instead of an hour so a single blip doesn't park clones on the PAT. - Name the real culprit when GitLab rejects the configured PAT (401) rather than claiming a benign fallback, and guard against a created token arriving without a secret. - Raise ImproperlyConfigured when neither an ephemeral token nor a PAT is available instead of building an oauth2:None@ clone URL, and log per clone which credential was embedded. - Expand the git push auth-failure message with expired-clone-token and branch-protection guidance; log push network failures too. * chore(docker): Build sandbox locally and pin compose network name Build the sandbox service from the sibling daiv-sandbox checkout with a source mount so local development runs against the unreleased sandbox the clone-token work depends on. Pin the default compose network to daiv_default so sandbox-launched containers can attach to it by a stable name, and pin the gitlab host via DAIV_SANDBOX_EXTRA_HOSTS since gVisor can't use Docker's embedded DNS resolver.
Tool calls made inside subagents were indistinguishable from main-agent ones in the logs. ToolCallLoggingMiddleware now prefixes each line with the agent name read from the run config's lc_agent_name metadata, set by create_agent(name=...) and deepagents' SubAgentMiddleware, falling back to <unknown-agent> when it cannot be resolved. Two contract tests pin the undocumented upstream key location and the task tool's metadata isolation, so a langchain/deepagents upgrade that breaks name resolution fails CI instead of silently degrading logs.
With the sandbox as the authoritative workspace, the local-clone diff the SWE-bench harness read at run end was always empty. GitMiddleware now optionally captures the working-tree diff (vs HEAD, untracked included) at turn end via GitManager.get_diff and exposes it as model_patch in the output state; evals/swebench.py reads it from ainvoke's result and fails loudly if the wiring drifts. Also: - Refresh OpenRouter model list (GLM 5.1, MiniMax M3, Kimi K2.6) and default the eval harness to MiniMax M3. - Bump swerebench dataset split to 2026_03 and sample size to 10. - Coerce site_settings.agent_thinking_level into ThinkingLevel in RepositoryConfig so raw-string site settings stop triggering pydantic serialization warnings on every config load.
SWE-bench eval runs logged a spurious '404 Project Not Found' traceback on every instance: GitMiddleware's pre-run MR lookup ran on the commit-pinned (detached HEAD) checkout and asked the settings-default platform about a repo living elsewhere. - Short-circuit the lookup on detached HEAD: no branch, no MR. - Create the repo client with the run's git_platform instead of the settings default, which could 404 or match a same-named repo's MR. - SWERepoClient: get_merge_request_by_branches returns None (SWE runs have no MR concept); repo_host is now optional at construction so the kwargs-less factory path works, but get_repository fails fast without it so a wrong-host clone can't silently corrupt an eval. - Add requests.RequestException to the soft-fail tuples in GitMiddleware and chat.repo_state: the platform SDKs are requests-based, so network blips surfaced as uncaught transport errors and crashed the run.
Runs that hit recursion_limit raise GraphRecursionError mid-flight, skipping after_agent hooks (patch capture, sandbox teardown) and discarding otherwise-finished work. StepBudgetMiddleware injects a step-budget reminder into the in-flight model request when the run nears the limit, escalating to a finalize-now notice close to the hard stop. Implemented via wrap_model_call so it adds no graph node and the reminder is never persisted to conversation state.
Replace the after_model no-op tool-call injection with a wrap_model_call retry. The hook variant added a graph node to every model/tools cycle, raising the superstep cost per turn from 2 to 3 and silently cutting the effective tool-call budget under recursion_limit by a third. Retrying within the node costs zero extra supersteps and keeps the synthetic no-op round-trip out of the persisted history; after MAX_EMPTY_RESPONSE_RETRIES the empty response is returned as-is so the loop ends instead of spinning.
The captured model_patch is a plain working-tree diff vs HEAD, so any dirt present before the agent acts (e.g. seeding artifacts) poisons every patch and can break eval grading. GitMiddleware now checks the workspace at run start on capture_patch runs and exposes the dirty file list as pre_run_dirty_files in state and on the LangSmith run tree; the SWE-bench harness prints a warning next to the run. Adds GitManager.get_changed_files (one batched round-trip, exact names straight from git) and a dedicated SandboxGitProtocolError so degrade paths stop swallowing bare RuntimeError programming bugs.
SWE-bench style evals check out a historical base commit of a full clone; the upstream fix for the issue under evaluation is often already merged, and agents were observed reading it via git log --all on remote refs. Detach HEAD at the base commit and drop branches, remotes, reflogs and non-ancestor tags after checkout. Tags pointing at ancestors are kept so VCS-versioning tools (setuptools-scm, hatch-vcs) still work for editable installs.
Models routinely send 'foo|bar'-style patterns to the literal grep backends, get a silent zero-match back, and conclude the symbol does not exist. When an aggregate grep finds nothing, none of the backends errored, and the pattern contains unambiguous regex fragments, return an explicit literal-semantics reminder instead of a bare no-matches. The hint lives on DAIVCompositeBackend.agrep only: a per-sub-backend error would abort composite aggregation and suppress real matches from sibling backends.
Add execution-constraint and how-to-work guidance to the eval prompt (verify reporter analysis, provision test deps in one step, prefer minimal fixes, never revert the offending feature, one final check then stop) and a matching one-batch dependency-install line to the agent system prompt. Trim the langsmith-fetch trace limit to 7.
Mirror the sandbox `FsGrepRequest` (`case_insensitive`/`multiline`/ `head_limit`, `invalid_pattern`) and forward them through `SandboxFileBackend.agrep` and the re-implemented composite routing. Add `DAIVFilesystemMiddleware`, which overrides the grep tool with an extended schema (regex always on, no literal mode, escaping for literals), and rebind `deepagents.graph.FilesystemMiddleware` so `create_deep_agent` builds it for the main agent (the exclude+re-add path is unavailable — upstream lists it as required scaffolding). Remove the now-obsolete literal-grep no-match hint. `head_limit` is enforced across aggregated backends, not just per backend.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Daiv-side half of converging the agent's
greptool to a ripgrep-backed, Claude-Code-style regex search (pairs with srtab/daiv-sandbox#323).FsGrepRequest(case_insensitive/multiline/head_limit) and the newinvalid_patternerror code; regeneratesschemas.dump.jsonand keeps the consistency test green.SandboxFileBackend.agrepand a re-implementedDAIVCompositeBackend.agreprouting.head_limitis now enforced across aggregated backends, not just per backend.DAIVFilesystemMiddleware, overriding the grep tool with an extended schema (regex always on, no literal mode; escape metacharacters for literals). The main agent gets it by rebindingdeepagents.graph.FilesystemMiddleware— the usual exclude+re-add path is unavailable because upstream listsFilesystemMiddlewareas required scaffolding.invalid_patternerror instead).Base
Targets
feat/sandbox-filebackend-scratchpadto stack on the in-flight WIP (keeps the diff to just the ripgrep work).Tests
Full unit suite passes (3013). New coverage: composite multi-backend merge + remap + cross-backend
head_limitcap, the grep-tool body (async happy path, read-permission security gate, traversal-path rejection), and an end-to-end test thatcreate_deep_agentactually builds the main agent with the extended grep schema (fails loudly if the rebind ever no-ops).Known follow-up (deliberately out of scope)
Disk-backed (non-sandbox) runs use upstream's literal grep while the tool description says "always regex" — needs a product call (condition the description on mode, give disk mode regex, or restore the hint on the disk path). Sandbox mode (eval/prod) is unaffected.
🤖 Generated with Claude Code