feat(orchestration): config-gated delegate_jobs tool for cross-repo delegation#1378
Open
srtab wants to merge 52 commits into
Open
feat(orchestration): config-gated delegate_jobs tool for cross-repo delegation#1378srtab wants to merge 52 commits into
srtab wants to merge 52 commits into
Conversation
- sandbox_envs migration test now restores migrations to head: reversing sandbox_envs.0006 cascade-reverses agent_sessions.0001 (drops its tables), which left later Session/Run tests failing with 'no such table'. - test_by_owner_admin_sees_all uses a leak-immune subset assertion: async ORM writes commit to the shared in-memory SQLite DB and escape rollback. - Drop the earlier threaded cleanup fixture (wrong approach: a worker thread hit a separate empty in-memory DB and corrupted connection state).
…_envs to Session/Run
…ff Run - Replace MemoryObservation.activity FK (→ activity.Activity) with MemoryObservation.run FK (→ agent_sessions.Run); migration 0002 adds the new column, copies activity_id values whose PKs exist in Run, then drops the old column. - Switch memory.signals receiver from activity_finished to run_finished; add CHAT-trigger guard so chat turns are never mined for observations. - Update extract_observations_task to accept run_id, look up Run, and use run.session_id as the LangGraph thread_id. - notifications.signals: on_run_finished receiver (already scaffolded) now fully wired with CHAT- and webhook-trigger guards, schedule resolution via run.session.scheduled_job, and a top-level try/except so errors never crash the run lifecycle. - Update memory/detail.html template to reference obs.run instead of obs.activity.
Complete the FK swap: memory models, signals, tasks, views and templates now reference sessions.Run instead of activity.Activity. Tests updated to use Run/Session fixtures. Notifications on_run_finished receiver completed with chat-skip guard and run-based helpers.
…un form move - Replace activity/chat URL mounts with permanent redirect patterns (301) via sessions.urls_legacy; LegacyActivityDetailRedirectView resolves Run by pk → session_detail#run-<id> - Move AgentRunCreateView, AgentRunCreateForm, RepoListField, AgentRunFieldsMixin from activity app into sessions app; run-form templates copied to sessions/templates/sessions/ - Switch runs include from activity.urls_runs to sessions.urls_runs (app_name="runs", route name "agent_run_new" preserved) - Sidebar: two items (Activity + Chat) → one "Sessions" item with running-jobs badge; "New chat" CTA → session_new - context_processors: replace "activity"/"chat" SECTION_URL_NAMES with "sessions"; running_jobs_count now queries Run instead of Activity - accounts/views.py dashboard tiles: all segment URLs → session_list - schedules/forms.py: import AgentRunFieldsMixin/RepoListField from sessions.forms (not activity.forms) - Fix all live stragglers: schedules/views.py, schedules/_schedule_row.html, memory/detail.html, notifications/signals.py, dashboard.html - Tests: new test_redirects.py (8 tests); update accounts/sidebar, breadcrumbs, context_processors and schedules tests to use session routes
…t become stubs - Empty activity/models.py and chat/models.py to module docstrings; both apps stay in INSTALLED_APPS with migration history intact (other apps' historical migrations reference them) - Delete dead files from activity app: views, urls, urls_runs, filters, forms, services, signals, management commands, static assets, and most templates; keep _agent_run_fields.html and _copy_markdown_button.html (still included by schedules and sessions templates respectively), and the activity_tags templatetag library (still loaded by notification email templates and sessions/_prompt_disclosure.html) with Activity-model-dependent tags removed - Delete dead files from chat app: views, urls, managers, chat_list.html, chat_detail.html; keep api/, turns.py, repo_state.py, static/, templatetags/, and _composer.html (included by sessions/session_detail.html) - Generate DeleteModel migrations for Activity (activity/0016) and ChatThread (chat/0004) with explicit dependencies on agent_sessions.0002 and memory.0002 so tables are never dropped before data is copied - Remove on_activity_finished receiver and all Activity-based helpers from notifications/signals.py; keep on_run_finished and all _run helpers - Drop legacy chat_thread/activity branches from generate_title_task in automation/titling/tasks.py; narrow Literal to ["session", "run"] - Delete all tests/unit_tests/activity/ and dead chat test files (test_models, test_views, test_composer_agent_picker, test_composer_env_select, test_sandbox_env_link); delete sessions/test_data_migration.py which depended on live Activity/ChatThread model access - Update notifications/test_signals.py to remove Activity-based test classes; keep TestUserBindingSeeder and TestOnRunFinished - Update automation/titling/test_tasks.py to use Session/Run entities instead of Activity; preserve all behavioural coverage - Fix global conftest mock_generate_title_task to remove the now-dead patches on activity.services and chat.models
…hedule forms load it
Follow-up hardening for the Session/Run unification based on review: - SessionLock.try_claim logs stale takeovers and splits the free-slot fast path from the takeover path (one query in the common case); heartbeat reports lost ownership and run_job_task awaits the cancelled heartbeat before releasing the slot. - Backfill wraps bulk_create with a contextual error plus summary logs and corrects the merge-precedence comment (user is first-wins). - Add DB CheckConstraints pinning Session.origin, Run.trigger_type and Run.status to their enums (new migration 0003). - Surface the orphan-task failure mode in error_message via a shared LINK_FAILED_PREFIX constant. - SSE stream guards malformed frames, auto-reconnects, and re-subscribes on timeout; the turns poller caps consecutive failures. - hydration returns a HydratedThread NamedTuple; rename _mark_failed_and_release -> _mark_failed_and_advance. Tests: backfill migration (historical registry), run-form view, run sync/usage parsing, sandbox-env propagation, templatetags, forms, enum-drift and effective_notify_on.
- sync_stuck_runs now fails chat runs (which have no DBTaskResult, so the task-result sync can never reconcile one whose streamer finally never ran) once their session heartbeat goes stale — the same staleness signal SessionLock uses to declare a holder dead — so a genuinely long-running chat turn is never reaped. A direct .update() (no run_finished emit) mirrors finalize_chat_run: chat runs stay out of the notification / memory / dispatch receivers. - notifications batch payload renders the human-readable status via RunStatus(agg_status).label instead of the raw enum value. - webhook callbacks log "Failed to create run ..." (was "activity").
_run_timeline.html included _status_pill.html without the pill's required variant/label args, so every run's timeline pill rendered with no color and no status text. Pass variant=run.status|status_variant and label=run.status, matching the session list page.
Fill 705 previously untranslated or fuzzy msgstr entries across 12 Django apps' Portuguese catalogs (European Portuguese), keeping terminology consistent with the existing glossary and preserving all format placeholders, HTML markup, and product/code names.
Follow-up refinements to the Activity/ChatThread → Sessions unification (callbacks, signals, views, managers, MCP, email templates) together with fixes from a code review of the change: - streaming: detect the agent's RUN_ERROR event (ag_ui surfaces it without raising) so an errored chat turn is finalized FAILED with an error_message instead of a silent SUCCESSFUL that also fed memory extraction - notifications: serialize Notification.context via DjangoJSONEncoder so lazy translation proxies no longer crash the save — the batch-rollup payload was silently failing, so batch notifications were never delivered - models: add DB CheckConstraints for agent_thinking_level (Session, Run) and Run.notify_on, consistent with the existing enum-constraint pattern - run timeline: render human-readable status labels and non-empty API/MCP origin badges instead of raw enum values / empty pills - sidebar: highlight the Sessions section on the namespaced "Start a run" route - remove dead code and stale comments, fix a sandbox docs link Add coverage for the chat-run failure paths, the notification batch rollup + fanout suite (restoring what the rename dropped), the new enum constraints, RunManager.by_owner, the API auth/ownership gates, and context serialization.
Reconcile main's per-user repository authorization (#1365, #1366) with this branch's activity+chat -> sessions unification. Conflict resolution highlights: - Ported the authorization wiring into the sessions app: assert_can_run in sessions/forms.py, aassert_can_run in sessions/services.py, chat/jobs/mcp submission gates, and the RepositoryAccessDenied handling in sessions views. - Split the run/session manager: by_owner stays ownership-only (async-safe, used by thread-continuation lookups in chat/jobs/mcp/sessions APIs) and a new visible_to adds the repo-read visibility clause for the sync viewing surfaces (list/detail/download/stream, dashboard, nav badge), matching main's own ownership-vs-visibility split. The stream view wraps visible_to in sync_to_async since identity resolution does a sync DB read. - Repointed the conftest allow-all authorization fixture to sessions.* targets (activity.services/forms were removed) and adapted the auto-merged authorization tests that still referenced the deleted Activity model to Run/Session (context-processors, schedules dispatch, batch-submit authz). - Kept the retired activity app as a migration-only stub; removed the leftover activity views/templates/tests carried in by main. Verified: make test (3549 passed), ruff check/format, makemigrations --check.
…fill The 0002 backfill copied text columns verbatim from historical Activity/ChatThread rows into Session/Run, whose text columns (repo_id, ref, title, prompt, agent_model, external_username, ...) are NOT NULL. Legacy rows can carry NULL on any of them (the columns pre-date their NOT NULL / default="" tightening on some deployments) — both a NULL repo_id and a NULL ref were hit in production — aborting the whole atomic migration with an IntegrityError. Coalesce NULL -> "" on every non-nullable text column via model introspection (``_blank_null_text``) for the activity Session/Run copy, and inline on the chat get_or_create defaults. Discriminator columns (origin/trigger_type/status) are deliberately excluded — a NULL there is corruption, and "" would violate their enum check constraints. Regression tests drive run_backfill with stubbed Activity/ChatThread iterators yielding NULL-text rows (SQLite enforces NOT NULL on the historical tables, so such rows can't be seeded directly).
…of schema drift activity.0016 / chat.0004 dropped the source tables by first issuing an individual RemoveIndex/RemoveConstraint per object and then DeleteModel. On deployments whose tables drifted from migration history — e.g. the activity_thread_id_nonempty check constraint (added in activity.0009) that was never actually created in production — RemoveConstraint aborts the migration with "constraint does not exist". Since both migrations drop the table wholesale, wrap the operations in SeparateDatabaseAndState: the database side is now a single DeleteModel (DROP TABLE CASCADE on PostgreSQL), which removes every index/constraint at once regardless of which ones are actually present, while the state side keeps the exact original operations so Django's migration bookkeeping and reversibility are unchanged. End state and behavior on a clean database are identical (verified: full suite + makemigrations --check green; backfill tests exercise both directions on a fresh DB).
- _run_timeline.html applied the duration filter to the Run object instead of its duration property, raising a TypeError whenever a session had a finished run. - session_list.html dropped the stretched-link anchor during the sessions unification, leaving rows styled as clickable (hover, chevron) but not actually linking to session_detail.
Apply code-review fixes to the delegate_jobs / orchestration feature: - Guard the delegate_jobs submission so an env-resolution or batch-submit raise returns the tool's JSON error contract instead of crashing the tool node. - Surface a failed resume-continuation enqueue with an error log; the coordinator would otherwise silently never auto-resume. - Pin the delegation-depth ceiling with a session CheckConstraint (move MAX_SPAWN_DEPTH to models) so no caller can bypass the tool-level guard. - Give a failed leg with no captured message an explicit line in the batch rollup instead of a blank block. - Enforce a non-empty per-target prompt and correct the orchestration docs: the delegate_jobs per-target prompt is required, not optional. - Add coverage for the depth boundary, submission failure, the busy-coordinator QUEUED-fallback resume, and the duplicate/limit guards.
…a-1fe974 # Conflicts: # daiv/accounts/context_processors.py # daiv/accounts/templates/accounts/_sidebar.html # daiv/jobs/tasks.py # daiv/sessions/filters.py # daiv/sessions/locale/pt/LC_MESSAGES/django.po # daiv/sessions/locks.py # daiv/sessions/management/commands/sync_stuck_runs.py # daiv/sessions/migrations/0001_initial.py # daiv/sessions/models.py # daiv/sessions/services.py # daiv/sessions/signals.py # daiv/sessions/static/sessions/js/session-filters.js # daiv/sessions/static/sessions/js/session-stream.js # daiv/sessions/templates/sessions/_filter_bar.html # daiv/sessions/templates/sessions/_session_row.html # daiv/sessions/templates/sessions/session_detail.html # daiv/sessions/templates/sessions/session_list.html # daiv/sessions/urls.py # daiv/sessions/urls_legacy.py # daiv/sessions/views.py # daiv/static_src/css/input.css # tests/unit_tests/sessions/test_filters.py # tests/unit_tests/sessions/test_management.py # tests/unit_tests/sessions/test_models.py # tests/unit_tests/sessions/test_redirects.py # tests/unit_tests/sessions/test_templatetags.py # tests/unit_tests/sessions/test_views_detail.py # tests/unit_tests/sessions/test_views_list.py # tests/unit_tests/sessions/test_views_runs.py
Revising the merge-conflict resolution: main's #1374 is the reviewed successor of this branch's own sessions foundation, so the session-UI files should match main exactly and PR #1378 should add only delegate_jobs (the DELEGATED_JOB origin flows into the filter UI automatically via SessionOrigin.choices — no bespoke UI needed). Reverts the two hand-merge grafts that re-introduced branch-only UI main's review had dropped: - run-timeline rail + _run_timeline.html partial (deleted) - the "N runs" pill in _session_row.html - the 3 run-timeline tests spliced into test_views_detail.py _session_row.html, session_detail.html and test_views_detail.py are now identical to origin/main.
…a-1fe974 # Conflicts: # tests/unit_tests/sessions/test_models.py
The merge with main introduced a diamond in the agent_sessions migration graph: both main's 0003_run_message_id and the branch's 0003_enum_check_constraints depended on 0002. The initial merge migration resolved the multiple-leaf error but the diamond broke reverse migrations — django-test-migrations rolling sandbox_envs down to 0005 (which cascades to drop agent_sessions tables) reversed both branches of the merge, attempting DROP COLUMN message_id out of order and failing (sandbox_envs/test_migrations.py::test_off_env_with_policy_loses_policy). Drop the merge migration and renumber the branch chain to sit linearly after main's 0003_run_message_id (0004..0008). main's migration is untouched; the branch migrations were not deployed anywhere.
Every provider was effectively unbounded: langchain passed timeout=None straight through to each SDK client, so a stalled provider could block a run indefinitely, and default retry counts only multiplied the wait. Add site-configurable model_request_timeout_seconds (default 120, minimum 1) and model_max_retries (default 1), applied to every provider and model-backed task at model-construction time. The timeout is shaped per provider integration (httpx.Timeout for OpenAI/OpenRouter, bare float for Anthropic/Google), and reused by the insecure http clients so they no longer fall back to httpx's 5s default. web_fetch now splits model construction from invocation: a build failure (misconfiguration) logs at ERROR and returns raw content flagged as unavailable, while a transient invocation failure degrades quietly to the raw page. Its summariser keeps a tighter 60s budget.
Flip orchestration.enabled to default true, so delegate_jobs is bound on every repository (opt out per repo with `orchestration.enabled: false`). Rewrite the delegate_jobs system prompt to present the tool as a capability for cross-repo work rather than asserting every run is a "coordination repository" — now that it is injected on every run, the old role framing would nudge single-repo runs into spurious delegation. Guard against self-delegation: a target matching the coordinator's own repo + ref is refused and steered to the `task` subagent tool for in-repo parallel work. Generalize the tool schema, system prompt and orchestrate skill away from ticket-specific wording toward the real invariant (leg isolation), and update the docs and skill charter to reflect on-by-default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Orchestration / delegated jobs
Adds a config-gated
delegate_jobsagent tool that lets an agent running on a coordination repository fan tailored sub-jobs out to other repositories in a single step, then resume automatically with a consolidated rollup once every leg finishes.What's included
delegate_jobstool + middleware — bound only when the repo setsorchestration.enabled: truein.daiv.yml. Each target runs as an independent single-repo job (own thread, own MR) with a required per-target prompt.Session.parent_thread_id/spawn_depth(parentage + delegation depth) andRun.continuation_of_batch_id, with a partial unique constraint that elects a single coordinator continuation per batch.run_finishedreceiver enqueues one continuation run on the parent thread when every leg of a delegated batch is terminal (QUEUED-and-released-FIFO when the coordinator session is busy).repos[].promptoverride on the Jobs API (POST /api/jobs) and MCPsubmit_job.orchestrateskill charter + user docs (docs/features/orchestration.md).Review hardening (final commit)
A code-review pass surfaced and fixed:
delegate_jobssubmission so an env-resolution / batch-submit raise returns the tool's JSON error contract instead of crashing the tool node.SessionCheckConstraint(not just the tool-levelif), so no otherasubmit_batch_runscaller can bypass the fuse.delegate_jobsper-target prompt is required, not optional — that fallback semantics belongs to the Jobs API layer).Testing
make teston the affected suites passes (422 tests),ruffclean,makemigrations --checkreports no drift.Stacking
This branch is stacked on #1374 (sessions unification), so it targets
claude/determined-maxwell-3d7694. The diff shows only the delegated-jobs commits; retarget tomainonce #1374 merges.