diff --git a/FORK_CHANGES.md b/FORK_CHANGES.md index 6e75abd0881fb..1910414dfc33f 100644 --- a/FORK_CHANGES.md +++ b/FORK_CHANGES.md @@ -12,37 +12,38 @@ setting is missing here or a setting named here does not exist. ## In T3 Code itself -| Change | Why | Setting | Default | PR | -| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| The server, the CLI and the desktop app look for updates in this fork's releases, not upstream's. | A fork build must never "update" to an upstream build and lose the fork's changes. | none | always | [#1](https://github.com/scuttletoby/t3code/pull/1) | -| A server installed as a boot service can update itself once no thread is working, after an optional "is it quiet?" command agrees. | Updates without someone watching, and without cutting off running work. | `server.automaticUpdates`, `server.automaticUpdatesQuietCommand` | off | [#5](https://github.com/scuttletoby/t3code/pull/5) | -| Every fork setting has a pinned default, and lives in the fork's own file (`forkSettings.ts`). | An upstream sync can never switch a fork feature on, and a new setting touches no upstream file. | none | always | [#171](https://github.com/scuttletoby/t3code/pull/171), [#185](https://github.com/scuttletoby/t3code/pull/185) | -| The server keeps a history of each provider account's usage-limit readings. | Pacing needs to know how fast a limit is being used, not only where it stands. | none | always (read only) | [#24](https://github.com/scuttletoby/t3code/pull/24) | -| The server reads usage limits every 10 minutes while no app is open. | Pacing has current numbers on a machine nobody is looking at. | `server.headlessUsageRefreshInterval` | on (10 minutes); read only | [#42](https://github.com/scuttletoby/t3code/pull/42) | -| Usage pacing: a reserve of each limit window kept free for the person, and an even pace for agents. | Agents should not use up a subscription before its owner needs it. | `server.usagePacing` | off | [#52](https://github.com/scuttletoby/t3code/pull/52) | -| The pacing gate on `delegate_task`: advise only, or move, lower the effort of, or hold a task on an account that is ahead of pace. | Delegated work follows the pace without the agent having to check. | `server.usagePacingGate`, `server.usagePacingLowerEffort` | off | [#73](https://github.com/scuttletoby/t3code/pull/73) | -| Agent slots: `delegate_task` is refused above a number of running agents per machine or account, or below a memory floor. | Too many agents at once can overload a small machine. | `server.agentSlots` | off; rule only observes | [#84](https://github.com/scuttletoby/t3code/pull/84), [#89](https://github.com/scuttletoby/t3code/pull/89) | -| Waits: agents can wait for a pull request, a file, a system unit, a named check command or a free agent slot (`wait_for`, `wait_list`, `wait_cancel`), and are woken when it happens. | An agent no longer polls or sleeps while it waits. | `server.waitFor`, `server.waitChecks`, `projectOverride.waitChecks` | off | [#84](https://github.com/scuttletoby/t3code/pull/84), [#90](https://github.com/scuttletoby/t3code/pull/90), [#91](https://github.com/scuttletoby/t3code/pull/91), [#103](https://github.com/scuttletoby/t3code/pull/103), [#106](https://github.com/scuttletoby/t3code/pull/106), [#109](https://github.com/scuttletoby/t3code/pull/109), [#110](https://github.com/scuttletoby/t3code/pull/110), [#116](https://github.com/scuttletoby/t3code/pull/116), [#119](https://github.com/scuttletoby/t3code/pull/119) | -| The Settings page **Fork** (`/settings/fork`): every fork setting in one place, with one line per feature at the top saying whether it is on: waits, wait checks, agent slots, usage pacing (per account by name), model policy, review rules, automatic updates, handoff and playtest. MCP profiles are not on it yet. | Change every fork setting without editing a file. | `server.waitFor`, `server.waitChecks`, `server.agentSlots`, `server.usagePacing`, `server.usagePacingGate`, `server.usagePacingLowerEffort`, `server.reviewRules`, `server.modelPolicy`, `server.automaticUpdates`, `server.automaticUpdatesQuietCommand`, `server.handoff`, `server.playtest` | off | [#104](https://github.com/scuttletoby/t3code/pull/104), [#543](https://github.com/scuttletoby/t3code/pull/543), [#544](https://github.com/scuttletoby/t3code/pull/544) | -| The fork's checks on `delegate_task` (pacing, slots) sit behind one fork module. | Fewer fork lines in upstream's code, so syncs conflict less. No change in behaviour. | none | always | [#174](https://github.com/scuttletoby/t3code/pull/174) | -| The fork's service layers in the server are wired from one fork file (`apps/server/src/fork/forkLayers.ts`). | Fewer fork lines in upstream's code, so syncs conflict less. No change in behaviour. | none | always | [#384](https://github.com/scuttletoby/t3code/pull/384) | -| A pure capacity rule for "dynamic slots" (one budget per machine), with shared test cases. Not wired into anything yet. | A later step replaces fixed agent counts with what the machine can take. | none | not used | [#203](https://github.com/scuttletoby/t3code/pull/203) | -| File links in chat open in the files viewer, full size, or in an editor on a remote machine, and can be downloaded, from their menu. | File links were only copyable on remote machines and had no menu on touch screens. | none | always | [#17](https://github.com/scuttletoby/t3code/pull/17) | -| A hidden read-only command, `t3 __fork-report --base-dir `, prints what a T3 home holds and which fork features are on, and where. | The go-live kit checks a build and a home before switching, without writing anything. | none | only when run | [#172](https://github.com/scuttletoby/t3code/pull/172), [#184](https://github.com/scuttletoby/t3code/pull/184) | -| A local, read-only "is T3 busy?" answer: `GET /api/fork/activity` on loopback, and `t3 __fork-report --base-dir --activity`, which asks it and exits 0 only when nothing is working. | A host quiet check reads T3's own turns and approvals instead of guessing from processes. | `server.activityProbe` | off | [#547](https://github.com/scuttletoby/t3code/pull/547) | -| `t3 serve --hold-work` starts a server on a copy of a T3 home with all agent work and T3 Connect held. | Look at a new build's threads and migrations on a copy before it replaces a running T3. | none | only with the flag | [#175](https://github.com/scuttletoby/t3code/pull/175) | -| Docs: one machine for new work through upstream's load-balancing weights; settings saved with an early draft's client setting for it still load. | Upstream already has the feature, so the fork's own setting was removed. | none | - | [#25](https://github.com/scuttletoby/t3code/pull/25) | -| MCP profiles: named profiles (which outside MCP servers and T3 tool groups a thread gets), a default, a sub-agent rule, a pick per thread and the rules that pick one. Claude applies the outside servers and refuses a turn when its profile cannot. | Fewer tools per thread saves tokens and keeps agents from tools they should not use. | `server.mcpProfiles`, `server.mcpProfile`, `server.mcpSubagentProfile`, `server.mcpThreadProfiles`, `projectOverride.mcpProfile`, `projectOverride.mcpSubagentProfile` | off | [#227](https://github.com/scuttletoby/t3code/pull/227), [#528](https://github.com/scuttletoby/t3code/pull/528) | -| Playtest mode, part 1: settings for a project's playtest adapter (claim, renew, push and release steps only the person sets) and the contract those steps follow. Nothing starts a session yet. | A person tries an agent's unfinished work on a test target, round after round, no PR per round. | `server.playtest`, `projectOverride.playtest` | off | [#260](https://github.com/scuttletoby/t3code/pull/260) | -| Playtest mode, part 2: the session store and service: start (claim), rounds of push steps, end (release), kept across restarts. A session keeps the steps it started with. No tool calls it yet. | A person tries an agent's unfinished work on a test target, round after round, no PR per round. | `server.playtest`, `projectOverride.playtest` | off | [#261](https://github.com/scuttletoby/t3code/pull/261) | -| Playtest mode, part 3: the worker renews claims, ends idle, too-long and gone-thread sessions, finishes what a restart cut, and tells the thread once when T3 ended a session. | A person tries an agent's unfinished work on a test target, round after round, no PR per round. | `server.playtest`, `projectOverride.playtest` | off | [#262](https://github.com/scuttletoby/t3code/pull/262) | -| Handoff instead of compaction, part 1: the setting and the pure "should this thread hand off now?" decision. Nothing acts on it yet. | Long threads keep their children, PRs and waits instead of compacting. | `server.handoff`, `projectOverride.handoff` | off | [#282](https://github.com/scuttletoby/t3code/pull/282) | -| Review rules, part 1: the setting for how this machine runs a review lens (runner, state question), off by default; `{"preset": "recommended"}` turns the fork's defaults on. A bad hand edit reads as off and is kept. Nothing reads it yet. | The repo says what a pull request needs and CI enforces it; the machine says how reviews run. | `server.reviewRules`, `projectOverride.reviewRules` | off | [#291](https://github.com/scuttletoby/t3code/pull/291) | -| An interval setting the settings file cannot hold (negative, not finite, too large) is refused when saved; one already in the file reads as its default instead of resetting every setting. A test finds every interval setting from the schema. | A `1e400` saved as `null` made the next start reject the whole settings file. | none | always | [#323](https://github.com/scuttletoby/t3code/pull/323) | -| Handoff instead of compaction, part 2: the handoff service hands a big or idle thread to a fresh session in the same thread, with T3's state and the recent conversation, only while the thread is quiet and still at the run that triggered it. | Long threads keep their children, PRs and waits instead of compacting. | `server.handoff`, `projectOverride.handoff` | off | [#320](https://github.com/scuttletoby/t3code/pull/320) | -| A task delegated to a linked environment is still followed if the delegating call is cut off, and still delivers its reply when that environment runs another T3 version. | Its parent was never told: the task stayed running here and its result never arrived. | none | always | [#412](https://github.com/scuttletoby/t3code/pull/412), [#540](https://github.com/scuttletoby/t3code/pull/540) | -| Run records: one line per agent run that ends (model, effort, role, kind, estimate, times, end status, provider-reported tokens) in `/run-records/runs.jsonl`, append-only, and a weekly summary command. | Later choices of model, effort and task size need what earlier runs cost and took. | none | always (writes only its own files) | [#588](https://github.com/scuttletoby/t3code/pull/588) | -| Model policy, part 1: the setting for which models (and efforts) agents may pick, per provider driver: off, advise or enforce, and the allowed rows. A bad hand edit reads as nothing allowed. Nothing reads the setting yet. | Agents should use the models the person chose for each kind of work. | `server.modelPolicy` | off | [#268](https://github.com/scuttletoby/t3code/pull/268) | +| Change | Why | Setting | Default | PR | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| The server, the CLI and the desktop app look for updates in this fork's releases, not upstream's. | A fork build must never "update" to an upstream build and lose the fork's changes. | none | always | [#1](https://github.com/scuttletoby/t3code/pull/1) | +| A server installed as a boot service can update itself once no thread is working, after an optional "is it quiet?" command agrees. | Updates without someone watching, and without cutting off running work. | `server.automaticUpdates`, `server.automaticUpdatesQuietCommand` | off | [#5](https://github.com/scuttletoby/t3code/pull/5) | +| Every fork setting has a pinned default, and lives in the fork's own file (`forkSettings.ts`). | An upstream sync can never switch a fork feature on, and a new setting touches no upstream file. | none | always | [#171](https://github.com/scuttletoby/t3code/pull/171), [#185](https://github.com/scuttletoby/t3code/pull/185) | +| The server keeps a history of each provider account's usage-limit readings. | Pacing needs to know how fast a limit is being used, not only where it stands. | none | always (read only) | [#24](https://github.com/scuttletoby/t3code/pull/24) | +| The server reads usage limits every 10 minutes while no app is open. | Pacing has current numbers on a machine nobody is looking at. | `server.headlessUsageRefreshInterval` | on (10 minutes); read only | [#42](https://github.com/scuttletoby/t3code/pull/42) | +| Usage pacing: a reserve of each limit window kept free for the person, and an even pace for agents. | Agents should not use up a subscription before its owner needs it. | `server.usagePacing` | off | [#52](https://github.com/scuttletoby/t3code/pull/52) | +| The pacing gate on `delegate_task`: advise only, or move, lower the effort of, or hold a task on an account that is ahead of pace. | Delegated work follows the pace without the agent having to check. | `server.usagePacingGate`, `server.usagePacingLowerEffort` | off | [#73](https://github.com/scuttletoby/t3code/pull/73) | +| Agent slots: `delegate_task` is refused above a number of running agents per machine or account, or below a memory floor. | Too many agents at once can overload a small machine. | `server.agentSlots` | off; rule only observes | [#84](https://github.com/scuttletoby/t3code/pull/84), [#89](https://github.com/scuttletoby/t3code/pull/89) | +| Waits: agents can wait for a pull request, a file, a system unit, a named check command or a free agent slot (`wait_for`, `wait_list`, `wait_cancel`), and are woken when it happens. | An agent no longer polls or sleeps while it waits. | `server.waitFor`, `server.waitChecks`, `projectOverride.waitChecks` | off | [#84](https://github.com/scuttletoby/t3code/pull/84), [#90](https://github.com/scuttletoby/t3code/pull/90), [#91](https://github.com/scuttletoby/t3code/pull/91), [#103](https://github.com/scuttletoby/t3code/pull/103), [#106](https://github.com/scuttletoby/t3code/pull/106), [#109](https://github.com/scuttletoby/t3code/pull/109), [#110](https://github.com/scuttletoby/t3code/pull/110), [#116](https://github.com/scuttletoby/t3code/pull/116), [#119](https://github.com/scuttletoby/t3code/pull/119) | +| Delegated results: `task_list` lists a thread's delegated tasks whose result it has not read, and its running children per place from live status; a parent whose finished child's result never arrived gets one reminder. | A finished child's result can no longer sit unseen. | `server.waitFor` (the reminder) | off; `task_list` always (read only) | [#622](https://github.com/scuttletoby/t3code/pull/622) | +| The Settings page **Fork** (`/settings/fork`): every fork setting in one place, with one line per feature at the top saying whether it is on: waits, wait checks, agent slots, usage pacing (per account by name), model policy, review rules, automatic updates, handoff and playtest. MCP profiles are not on it yet. | Change every fork setting without editing a file. | `server.waitFor`, `server.waitChecks`, `server.agentSlots`, `server.usagePacing`, `server.usagePacingGate`, `server.usagePacingLowerEffort`, `server.reviewRules`, `server.modelPolicy`, `server.automaticUpdates`, `server.automaticUpdatesQuietCommand`, `server.handoff`, `server.playtest` | off | [#104](https://github.com/scuttletoby/t3code/pull/104), [#543](https://github.com/scuttletoby/t3code/pull/543), [#544](https://github.com/scuttletoby/t3code/pull/544) | +| The fork's checks on `delegate_task` (pacing, slots) sit behind one fork module. | Fewer fork lines in upstream's code, so syncs conflict less. No change in behaviour. | none | always | [#174](https://github.com/scuttletoby/t3code/pull/174) | +| The fork's service layers in the server are wired from one fork file (`apps/server/src/fork/forkLayers.ts`). | Fewer fork lines in upstream's code, so syncs conflict less. No change in behaviour. | none | always | [#384](https://github.com/scuttletoby/t3code/pull/384) | +| A pure capacity rule for "dynamic slots" (one budget per machine), with shared test cases. Not wired into anything yet. | A later step replaces fixed agent counts with what the machine can take. | none | not used | [#203](https://github.com/scuttletoby/t3code/pull/203) | +| File links in chat open in the files viewer, full size, or in an editor on a remote machine, and can be downloaded, from their menu. | File links were only copyable on remote machines and had no menu on touch screens. | none | always | [#17](https://github.com/scuttletoby/t3code/pull/17) | +| A hidden read-only command, `t3 __fork-report --base-dir `, prints what a T3 home holds and which fork features are on, and where. | The go-live kit checks a build and a home before switching, without writing anything. | none | only when run | [#172](https://github.com/scuttletoby/t3code/pull/172), [#184](https://github.com/scuttletoby/t3code/pull/184) | +| A local, read-only "is T3 busy?" answer: `GET /api/fork/activity` on loopback, and `t3 __fork-report --base-dir --activity`, which asks it and exits 0 only when nothing is working. | A host quiet check reads T3's own turns and approvals instead of guessing from processes. | `server.activityProbe` | off | [#547](https://github.com/scuttletoby/t3code/pull/547) | +| `t3 serve --hold-work` starts a server on a copy of a T3 home with all agent work and T3 Connect held. | Look at a new build's threads and migrations on a copy before it replaces a running T3. | none | only with the flag | [#175](https://github.com/scuttletoby/t3code/pull/175) | +| Docs: one machine for new work through upstream's load-balancing weights; settings saved with an early draft's client setting for it still load. | Upstream already has the feature, so the fork's own setting was removed. | none | - | [#25](https://github.com/scuttletoby/t3code/pull/25) | +| MCP profiles: named profiles (which outside MCP servers and T3 tool groups a thread gets), a default, a sub-agent rule, a pick per thread and the rules that pick one. Claude applies the outside servers and refuses a turn when its profile cannot. | Fewer tools per thread saves tokens and keeps agents from tools they should not use. | `server.mcpProfiles`, `server.mcpProfile`, `server.mcpSubagentProfile`, `server.mcpThreadProfiles`, `projectOverride.mcpProfile`, `projectOverride.mcpSubagentProfile` | off | [#227](https://github.com/scuttletoby/t3code/pull/227), [#528](https://github.com/scuttletoby/t3code/pull/528) | +| Playtest mode, part 1: settings for a project's playtest adapter (claim, renew, push and release steps only the person sets) and the contract those steps follow. Nothing starts a session yet. | A person tries an agent's unfinished work on a test target, round after round, no PR per round. | `server.playtest`, `projectOverride.playtest` | off | [#260](https://github.com/scuttletoby/t3code/pull/260) | +| Playtest mode, part 2: the session store and service: start (claim), rounds of push steps, end (release), kept across restarts. A session keeps the steps it started with. No tool calls it yet. | A person tries an agent's unfinished work on a test target, round after round, no PR per round. | `server.playtest`, `projectOverride.playtest` | off | [#261](https://github.com/scuttletoby/t3code/pull/261) | +| Playtest mode, part 3: the worker renews claims, ends idle, too-long and gone-thread sessions, finishes what a restart cut, and tells the thread once when T3 ended a session. | A person tries an agent's unfinished work on a test target, round after round, no PR per round. | `server.playtest`, `projectOverride.playtest` | off | [#262](https://github.com/scuttletoby/t3code/pull/262) | +| Handoff instead of compaction, part 1: the setting and the pure "should this thread hand off now?" decision. Nothing acts on it yet. | Long threads keep their children, PRs and waits instead of compacting. | `server.handoff`, `projectOverride.handoff` | off | [#282](https://github.com/scuttletoby/t3code/pull/282) | +| Review rules, part 1: the setting for how this machine runs a review lens (runner, state question), off by default; `{"preset": "recommended"}` turns the fork's defaults on. A bad hand edit reads as off and is kept. Nothing reads it yet. | The repo says what a pull request needs and CI enforces it; the machine says how reviews run. | `server.reviewRules`, `projectOverride.reviewRules` | off | [#291](https://github.com/scuttletoby/t3code/pull/291) | +| An interval setting the settings file cannot hold (negative, not finite, too large) is refused when saved; one already in the file reads as its default instead of resetting every setting. A test finds every interval setting from the schema. | A `1e400` saved as `null` made the next start reject the whole settings file. | none | always | [#323](https://github.com/scuttletoby/t3code/pull/323) | +| Handoff instead of compaction, part 2: the handoff service hands a big or idle thread to a fresh session in the same thread, with T3's state and the recent conversation, only while the thread is quiet and still at the run that triggered it. | Long threads keep their children, PRs and waits instead of compacting. | `server.handoff`, `projectOverride.handoff` | off | [#320](https://github.com/scuttletoby/t3code/pull/320) | +| A task delegated to a linked environment is still followed if the delegating call is cut off, and still delivers its reply when that environment runs another T3 version. | Its parent was never told: the task stayed running here and its result never arrived. | none | always | [#412](https://github.com/scuttletoby/t3code/pull/412), [#540](https://github.com/scuttletoby/t3code/pull/540) | +| Run records: one line per agent run that ends (model, effort, role, kind, estimate, times, end status, provider-reported tokens) in `/run-records/runs.jsonl`, append-only, and a weekly summary command. | Later choices of model, effort and task size need what earlier runs cost and took. | none | always (writes only its own files) | [#588](https://github.com/scuttletoby/t3code/pull/588) | +| Model policy, part 1: the setting for which models (and efforts) agents may pick, per provider driver: off, advise or enforce, and the allowed rows. A bad hand edit reads as nothing allowed. Nothing reads the setting yet. | Agents should use the models the person chose for each kind of work. | `server.modelPolicy` | off | [#268](https://github.com/scuttletoby/t3code/pull/268) | ## Repository, CI and releases diff --git a/apps/server/src/fork/delegatedResults/DelegatedResultReminder.test.ts b/apps/server/src/fork/delegatedResults/DelegatedResultReminder.test.ts new file mode 100644 index 0000000000000..4c924267aeba2 --- /dev/null +++ b/apps/server/src/fork/delegatedResults/DelegatedResultReminder.test.ts @@ -0,0 +1,367 @@ +/** + * Fork (issue 386): the reminder of delegated results that never reached their parent. The state + * question: it runs again (no second message), the server restarted (a fresh service finds the + * message it sent), the parent is busy or was stopped, and old rows from before delivery records. + */ +import { describe, expect, it } from "@effect/vitest"; +import { + type CommandId, + MessageId, + type OrchestrationV2Subagent, + type OrchestrationV2ThreadShell, + ThreadId, +} from "@t3tools/contracts"; +import * as Duration from "effect/Duration"; +import * as Effect from "effect/Effect"; +import * as Fiber from "effect/Fiber"; +import * as Layer from "effect/Layer"; +import * as Option from "effect/Option"; +import * as PubSub from "effect/PubSub"; +import * as Stream from "effect/Stream"; +import { TestClock } from "effect/testing"; + +import * as McpToolAccessTestkit from "../../mcp/McpToolAccess.testkit.ts"; +import * as CommandReceiptStore from "../../orchestration-v2/CommandReceiptStore.ts"; +import * as ProjectionStore from "../../orchestration-v2/ProjectionStore.ts"; +import * as ThreadManagement from "../../orchestration-v2/ThreadManagementService.ts"; +import { ServerSettingsService } from "../../serverSettings.ts"; +import { REMINDER_MAX_AGE_MS } from "./delegatedResults.ts"; +import { at, delegatedTask, NOW, PARENT } from "./delegatedResults.testkit.ts"; +import * as DelegatedResultReminder from "./DelegatedResultReminder.ts"; + +const OPTIONS: DelegatedResultReminder.ReminderOptions = { + delay: Duration.minutes(2), + retryDelay: Duration.minutes(1), + attempts: 3, +}; + +const idleShell = ( + id: ThreadId, + overrides: Partial = {}, +): OrchestrationV2ThreadShell => ({ + ...McpToolAccessTestkit.liveThreadShell(id, { activeRunId: null }), + status: "idle", + ...overrides, +}); + +interface Sent { + readonly threadId: ThreadId; + readonly messageId: string; + readonly commandId: string; + readonly text: string; +} + +/** + * The parent's records as the server keeps them, in memory. `sent` is shared across services, as + * the database is across a restart; the services themselves keep nothing. + */ +const makeWorld = (waitFor: { readonly enabled: boolean } = { enabled: true }) => { + const shells = new Map([[PARENT, idleShell(PARENT)]]); + const tasks = new Map>([[PARENT, [delegatedTask()]]]); + const sent: Array = []; + const receipts = new Map(); + let failSends = 0; + return { + shells, + tasks, + sent, + failNextSends: (count: number) => { + failSends = count; + }, + layer: (events: Stream.Stream<{ type: string; threadId: ThreadId; payload: unknown }>) => + Layer.mergeAll( + ServerSettingsService.layerTest({ waitFor }), + Layer.mock(ThreadManagement.ThreadManagementService)({ + getThreadShell: (threadId) => Effect.succeed(shells.get(threadId) ?? null), + getThreadRecords: ((threadId: ThreadId) => + Effect.succeed({ subagents: tasks.get(threadId) ?? [] })) as never, + getMessagesById: (threadId, ids) => + Effect.succeed( + new Map( + sent + .filter((message) => message.threadId === threadId) + .filter((message) => ids.includes(MessageId.make(message.messageId))) + .map((message) => [MessageId.make(message.messageId), {} as never]), + ), + ), + sendToThread: (input) => + Effect.gen(function* () { + if (failSends > 0) { + failSends -= 1; + return yield* Effect.fail({ + _tag: "ThreadManagementProjectionLoadError", + message: "busy database", + } as never); + } + sent.push({ + threadId: input.threadId, + messageId: input.messageId!, + commandId: input.commandId, + text: input.text, + }); + receipts.set(input.commandId, input.threadId); + return {} as never; + }), + streamDomainEvents: events as never, + }), + Layer.mock(CommandReceiptStore.CommandReceiptStoreV2)({ + getByCommandId: (commandId: CommandId) => + Effect.succeed( + receipts.has(commandId) + ? Option.some({ status: "accepted", threadId: receipts.get(commandId)! } as never) + : Option.none(), + ), + }), + Layer.mock(ProjectionStore.ProjectionStoreV2)({ + getRecoveryThreadIds: () => Effect.succeed([...tasks.keys()]), + }), + ), + }; +}; + +/** A service as the server makes one, over the world's records; each call is a fresh one. */ +type Event = { readonly type: string; readonly threadId: ThreadId; readonly payload: unknown }; + +const service = ( + world: ReturnType, + events: Stream.Stream = Stream.empty, +) => DelegatedResultReminder.make(OPTIONS).pipe(Effect.provide(world.layer(events))); + +const setClock = TestClock.setTime(NOW); + +/** Lets forked fibers reach their next sleep (or finish) before the clock moves or a check. */ +const settle = Effect.repeat(Effect.yieldNow, { times: 20 }); + +describe("DelegatedResultReminder.check", () => { + it.effect("sends one reminder naming an owed result to an idle parent", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + const reminder = yield* service(world); + const outcome = yield* reminder.check(PARENT); + expect(outcome).toEqual({ type: "sent", taskIds: ["task-1"] }); + expect(world.sent).toHaveLength(1); + expect(world.sent[0]!.text).toContain("task-1"); + expect(world.sent[0]!.text).toContain("task_status"); + }), + ); + + it.effect("sends nothing while waits are off (a machine that never names the setting)", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld({ enabled: false }); + const reminder = yield* service(world); + expect(yield* reminder.check(PARENT)).toEqual({ type: "off" }); + expect(world.sent).toHaveLength(0); + }), + ); + + it.effect("runs again without a second message (the same result is never reminded twice)", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + const reminder = yield* service(world); + yield* reminder.check(PARENT); + yield* reminder.check(PARENT); + expect(world.sent).toHaveLength(1); + }), + ); + + it.effect("after a restart, a fresh service finds the message it sent and sends none", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + yield* (yield* service(world)).check(PARENT); + const restarted = yield* service(world); + expect(yield* restarted.check(PARENT)).toMatchObject({ type: "sent" }); + expect(world.sent).toHaveLength(1); + }), + ); + + it.effect("sends a new reminder when another result becomes owed", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + const reminder = yield* service(world); + yield* reminder.check(PARENT); + world.tasks.set(PARENT, [delegatedTask(), delegatedTask({ id: "task-2" })]); + yield* reminder.check(PARENT); + expect(world.sent).toHaveLength(2); + expect(world.sent[1]!.messageId).not.toBe(world.sent[0]!.messageId); + }), + ); + + it.effect("sends nothing for a delivered or read result", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + world.tasks.set(PARENT, [ + delegatedTask({ completionDelivery: { state: "delivered", observedByRunId: null } }), + delegatedTask({ + id: "task-2", + completionDelivery: { state: "acknowledged", observedByRunId: null }, + }), + ]); + expect(yield* (yield* service(world)).check(PARENT)).toEqual({ type: "none" }); + expect(world.sent).toHaveLength(0); + }), + ); + + it.effect("never wakes a parent for old rows: no delivery record, or ended over a day ago", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + world.tasks.set(PARENT, [ + delegatedTask({ completionDelivery: undefined }), + delegatedTask({ id: "task-2", completedAt: at(NOW - REMINDER_MAX_AGE_MS - 1) }), + delegatedTask({ id: "task-3", origin: "provider_native" }), + ]); + expect(yield* (yield* service(world)).check(PARENT)).toEqual({ type: "none" }); + expect(world.sent).toHaveLength(0); + }), + ); + + it.effect("waits while the parent works, and checks again when its run ends", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + world.shells.set( + PARENT, + idleShell(PARENT, { status: "running", activityRunStatus: "running" }), + ); + const reminder = yield* service(world); + expect(yield* reminder.check(PARENT)).toEqual({ type: "busy" }); + expect(world.sent).toHaveLength(0); + expect(reminder.watched.has(PARENT)).toBe(true); + expect( + reminder.triggers({ + type: "run.updated", + threadId: PARENT, + payload: { status: "completed" }, + }), + ).toBe(true); + world.shells.set(PARENT, idleShell(PARENT)); + expect(yield* reminder.check(PARENT)).toMatchObject({ type: "sent" }); + expect(reminder.watched.has(PARENT)).toBe(false); + // A run end of a parent with nothing owed is no trigger. + expect( + reminder.triggers({ + type: "run.updated", + threadId: PARENT, + payload: { status: "completed" }, + }), + ).toBe(false); + }), + ); + + it.effect("never restarts a parent a person stopped, and never writes to an archived one", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + world.shells.set(PARENT, idleShell(PARENT, { status: "interrupted" })); + const reminder = yield* service(world); + expect(yield* reminder.check(PARENT)).toEqual({ type: "held", detail: "stopped" }); + world.shells.set(PARENT, idleShell(PARENT, { archivedAt: at(NOW) })); + expect(yield* reminder.check(PARENT)).toEqual({ type: "gone" }); + world.shells.delete(PARENT); + expect(yield* reminder.check(PARENT)).toEqual({ type: "gone" }); + expect(world.sent).toHaveLength(0); + }), + ); + + it.effect("a send stopped half-way is tried again with the same ids, and lands once", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + world.failNextSends(1); + const reminder = yield* service(world); + expect(yield* reminder.check(PARENT)).toEqual({ type: "unreadable" }); + expect(yield* reminder.check(PARENT)).toMatchObject({ type: "sent" }); + expect(world.sent).toHaveLength(1); + }), + ); +}); + +describe("DelegatedResultReminder.start", () => { + it.effect("a task ending with its result owed brings one reminder after the delay", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + world.tasks.set(PARENT, []); + const pubsub = yield* PubSub.unbounded<{ + type: string; + threadId: ThreadId; + payload: unknown; + }>(); + const reminder = yield* service(world, Stream.fromPubSub(pubsub)); + const fiber = yield* reminder.start().pipe(Effect.andThen(Effect.never), Effect.forkScoped); + yield* Effect.yieldNow; + // The startup scan found nothing owed. + yield* TestClock.adjust(OPTIONS.delay); + expect(world.sent).toHaveLength(0); + + const ended = delegatedTask(); + world.tasks.set(PARENT, [ended]); + yield* PubSub.publish(pubsub, { type: "subagent.updated", threadId: PARENT, payload: ended }); + // A second event in the quiet window moves the one check later; it sends once. + yield* TestClock.adjust(Duration.minutes(1)); + yield* PubSub.publish(pubsub, { type: "subagent.updated", threadId: PARENT, payload: ended }); + yield* TestClock.adjust(Duration.minutes(1)); + expect(world.sent).toHaveLength(0); + yield* TestClock.adjust(Duration.minutes(1)); + expect(world.sent).toHaveLength(1); + yield* Fiber.interrupt(fiber); + }).pipe(Effect.scoped), + ); + + it.effect("after a restart, the startup scan reminds a result owed before it", () => + Effect.gen(function* () { + yield* setClock; + const world = makeWorld(); + const reminder = yield* service(world); + const fiber = yield* reminder.start().pipe(Effect.andThen(Effect.never), Effect.forkScoped); + yield* settle; + expect(world.sent).toHaveLength(0); + yield* TestClock.adjust(OPTIONS.delay); + yield* settle; + expect(world.sent).toHaveLength(1); + yield* Fiber.interrupt(fiber); + // The next start finds the same owed result and its message: nothing more is sent. + const again = yield* service(world); + const second = yield* again.start().pipe(Effect.andThen(Effect.never), Effect.forkScoped); + yield* settle; + yield* TestClock.adjust(OPTIONS.delay); + yield* settle; + expect(world.sent).toHaveLength(1); + yield* Fiber.interrupt(second); + }).pipe(Effect.scoped), + ); + + it.effect("ignores events that are no trigger", () => { + const world = makeWorld(); + return Effect.gen(function* () { + const reminder = yield* service(world); + expect( + reminder.triggers({ + type: "subagent.updated", + threadId: PARENT, + payload: delegatedTask({ status: "running" }), + }), + ).toBe(false); + expect( + reminder.triggers({ + type: "subagent.updated", + threadId: PARENT, + payload: delegatedTask({ origin: "provider_native" }), + }), + ).toBe(false); + expect( + reminder.triggers({ + type: "run.updated", + threadId: ThreadId.make("thread-other"), + payload: { status: "completed" }, + }), + ).toBe(false); + }); + }); +}); diff --git a/apps/server/src/fork/delegatedResults/DelegatedResultReminder.ts b/apps/server/src/fork/delegatedResults/DelegatedResultReminder.ts new file mode 100644 index 0000000000000..13025caa28eb1 --- /dev/null +++ b/apps/server/src/fork/delegatedResults/DelegatedResultReminder.ts @@ -0,0 +1,213 @@ +/** + * Fork (issue 386): backstop for upstream's completion wake. A parent's own events (a task ending + * owed, a run ending) start one delayed check; an idle parent with results still owed gets one queued + * message, ids from the task ids, so reruns and restarts never repeat it. Off with `server.waitFor`. + */ +import type { OrchestrationV2Subagent, ThreadId } from "@t3tools/contracts"; +import * as Clock from "effect/Clock"; +import * as Duration from "effect/Duration"; +import * as Effect from "effect/Effect"; +import * as FiberMap from "effect/FiberMap"; +import * as Layer from "effect/Layer"; +import * as Stream from "effect/Stream"; + +import * as CommandReceiptStore from "../../orchestration-v2/CommandReceiptStore.ts"; +import * as ProjectionStore from "../../orchestration-v2/ProjectionStore.ts"; +import * as ThreadManagement from "../../orchestration-v2/ThreadManagementService.ts"; +import { forkParked } from "../../serverActivation.ts"; +import { ServerSettingsService } from "../../serverSettings.ts"; +import * as WaitDelivery from "../../waits/WaitDelivery.ts"; +import { + isDelegatedTask, + isEndedTask, + isRemindable, + parentBusy, + parentGone, + reminderIds, + reminderText, + resultDelivery, +} from "./delegatedResults.ts"; + +export interface ReminderOptions { + /** From the event to the check: upstream's own wake goes first. */ + readonly delay: Duration.Input; + /** Between tries after a send that may succeed later. */ + readonly retryDelay: Duration.Input; + /** Tries of one check before it waits for the parent's next event. */ + readonly attempts: number; +} + +const DEFAULT_OPTIONS: ReminderOptions = { + delay: Duration.minutes(2), + retryDelay: Duration.minutes(1), + attempts: 5, +}; + +/** What one check did; tests read it. */ +export type CheckOutcome = + | { readonly type: "none" } + | { readonly type: "busy" } + | { readonly type: "off" } + | { readonly type: "gone" } + | { readonly type: "unreadable" } + | { readonly type: "sent"; readonly taskIds: ReadonlyArray } + | { readonly type: "held"; readonly detail: string }; + +const TERMINAL_RUN = new Set(["completed", "failed", "interrupted", "cancelled", "rolled_back"]); + +export const make = Effect.fn("DelegatedResultReminder.make")(function* ( + options: ReminderOptions = DEFAULT_OPTIONS, +) { + const threads = yield* ThreadManagement.ThreadManagementService; + const delivery = yield* WaitDelivery.make(); + const projections = yield* ProjectionStore.ProjectionStoreV2; + const settings = yield* ServerSettingsService; + // Parents that were busy (or could not be reached) while a result was owed: their next run end + // checks again. In memory; a restart rebuilds it from the startup scan. + const watched = new Set(); + + const check = Effect.fn("DelegatedResultReminder.check")(function* (parentThreadId: ThreadId) { + const enabled = yield* settings.getSettings.pipe( + Effect.map((current) => current.waitFor.enabled), + Effect.orElseSucceed(() => false), + ); + if (!enabled) return { type: "off" } satisfies CheckOutcome; + const shell = yield* threads + .getThreadShell(parentThreadId) + .pipe(Effect.orElseSucceed(() => undefined)); + if (shell === undefined) return { type: "unreadable" } satisfies CheckOutcome; + if (parentGone(shell)) { + watched.delete(parentThreadId); + return { type: "gone" } satisfies CheckOutcome; + } + if (parentBusy(shell!)) { + watched.add(parentThreadId); + return { type: "busy" } satisfies CheckOutcome; + } + const records = yield* threads + .getThreadRecords(parentThreadId, ["subagents"], { omitSettledSubagents: true }) + .pipe(Effect.orElseSucceed(() => undefined)); + if (records === undefined) return { type: "unreadable" } satisfies CheckOutcome; + const nowMs = yield* Clock.currentTimeMillis; + const owed = records.subagents.filter((task) => isRemindable(task, nowMs)); + if (owed.length === 0) { + watched.delete(parentThreadId); + return { type: "none" } satisfies CheckOutcome; + } + const ids = reminderIds( + parentThreadId, + owed.map((task) => task.id), + ); + const result = yield* delivery.deliver({ + threadId: parentThreadId, + messageId: ids.messageId, + commandId: ids.commandId, + text: reminderText(owed), + }); + switch (result.type) { + case "delivered": + watched.delete(parentThreadId); + return { type: "sent", taskIds: owed.map((task) => task.id) } satisfies CheckOutcome; + case "gone": + watched.delete(parentThreadId); + return { type: "gone" } satisfies CheckOutcome; + case "stopped": + // A person stopped it: never restart it from here. Its next run end checks again. + watched.add(parentThreadId); + return { type: "held", detail: "stopped" } satisfies CheckOutcome; + case "rejected": + watched.delete(parentThreadId); + yield* Effect.logWarning("delegated result reminder refused", { + threadId: parentThreadId, + detail: result.detail, + }); + return { type: "held", detail: result.detail } satisfies CheckOutcome; + case "retry": + watched.add(parentThreadId); + return { type: "unreadable" } satisfies CheckOutcome; + } + }); + + /** One check after the delay; tries again a few times only while the parent cannot be read. */ + const checkLater = Effect.fn("DelegatedResultReminder.checkLater")( + function* (parentThreadId: ThreadId) { + yield* Effect.sleep(options.delay); + for (let attempt = 1; attempt <= options.attempts; attempt += 1) { + const outcome = yield* check(parentThreadId); + if (outcome.type !== "unreadable") return; + yield* Effect.sleep(options.retryDelay); + } + }, + (effect, parentThreadId) => + effect.pipe( + Effect.catchCause((cause) => + Effect.logWarning("delegated result reminder check failed", { + threadId: parentThreadId, + cause, + }), + ), + ), + ); + + /** A task that ended with its result owed, or a run end of a watched parent. */ + const triggers = (event: { + readonly type: string; + readonly threadId: ThreadId; + readonly payload: unknown; + }): boolean => { + if (event.type === "subagent.updated") { + const task = event.payload as OrchestrationV2Subagent; + return isDelegatedTask(task) && isEndedTask(task) && resultDelivery(task) === "owed"; + } + if (event.type === "run.updated" && watched.has(event.threadId)) { + const run = event.payload as { readonly status?: string }; + return run.status !== undefined && TERMINAL_RUN.has(run.status); + } + return false; + }; + + const start = Effect.fn("DelegatedResultReminder.start")(function* () { + // Keyed by parent: a newer event moves its check later, so one quiet moment gets one check. + const timers = yield* FiberMap.make(); + yield* forkParked( + Stream.runForEach(threads.streamDomainEvents, (event) => + triggers(event) + ? FiberMap.run(timers, event.threadId, checkLater(event.threadId)).pipe(Effect.asVoid) + : Effect.void, + ), + ); + // Results owed when the server stopped: upstream offers their wakes again at start; a parent + // that still has one owed after the delay gets its reminder. One parent at a time. + yield* forkParked( + Effect.gen(function* () { + const parents = yield* projections + .getRecoveryThreadIds("delegated-completions") + .pipe(Effect.orElseSucceed((): ReadonlyArray => [])); + yield* Effect.sleep(options.delay); + for (const parentThreadId of new Set(parents)) { + // An event already moved this parent's check later. + if (yield* FiberMap.has(timers, parentThreadId)) continue; + yield* check(parentThreadId).pipe( + Effect.catchCause((cause) => + Effect.logWarning("delegated result reminder check failed", { + threadId: parentThreadId, + cause, + }), + ), + ); + } + }), + ); + }); + + return { check, start, triggers, watched }; +}); + +/** Started with the server's other workers, from `ForkLayers.layerRuntimeServices` (fork/forkLayers.ts). */ +export const layer = Layer.effectDiscard( + make().pipe(Effect.flatMap((service) => service.start())), +).pipe( + Layer.provide( + Layer.merge(ProjectionStore.layer, CommandReceiptStore.layerFromApplicationReceipts), + ), +); diff --git a/apps/server/src/fork/delegatedResults/delegatedResults.test.ts b/apps/server/src/fork/delegatedResults/delegatedResults.test.ts new file mode 100644 index 0000000000000..a1a5cc05042d4 --- /dev/null +++ b/apps/server/src/fork/delegatedResults/delegatedResults.test.ts @@ -0,0 +1,142 @@ +import { describe, expect, it } from "@effect/vitest"; +import { EnvironmentId, NodeId, ThreadId } from "@t3tools/contracts"; + +import { + countByPlace, + isRemindable, + localActivity, + parentBusy, + REMINDER_MAX_AGE_MS, + reminderIds, + reminderText, + remoteActivity, + resultDelivery, +} from "./delegatedResults.ts"; +import { at, delegatedTask, NOW, PARENT } from "./delegatedResults.testkit.ts"; + +describe("resultDelivery", () => { + it("is null while the task runs", () => { + expect(resultDelivery(delegatedTask({ status: "running" }))).toBeNull(); + }); + it("owes pending and claimed results", () => { + for (const state of ["pending", "claimed"] as const) { + expect( + resultDelivery(delegatedTask({ completionDelivery: { state, observedByRunId: null } })), + ).toBe("owed"); + } + }); + it("settles delivered, acknowledged and disposed results", () => { + for (const state of ["delivered", "acknowledged", "disposed"] as const) { + expect( + resultDelivery(delegatedTask({ completionDelivery: { state, observedByRunId: null } })), + ).toBe("settled"); + } + }); + it("says none for an old row with no delivery record", () => { + expect(resultDelivery(delegatedTask({ completionDelivery: undefined }))).toBe("none"); + }); +}); + +describe("isRemindable", () => { + it("reminds a recent owed result, local or remote", () => { + expect(isRemindable(delegatedTask(), NOW)).toBe(true); + const remote = delegatedTask({ + childThreadId: null, + remoteChild: { + environmentId: EnvironmentId.make("env-b"), + threadId: ThreadId.make("remote-1"), + label: "env b", + }, + }); + expect(isRemindable(remote, NOW)).toBe(true); + }); + it("never reminds an old row from before delivery records", () => { + expect(isRemindable(delegatedTask({ completionDelivery: undefined }), NOW)).toBe(false); + }); + it("never reminds a result older than a day (an abandoned thread)", () => { + const old = delegatedTask({ completedAt: at(NOW - REMINDER_MAX_AGE_MS - 1) }); + expect(isRemindable(old, NOW)).toBe(false); + }); + it("never reminds a delivered or read result (delivered twice is not owed twice)", () => { + expect( + isRemindable( + delegatedTask({ completionDelivery: { state: "delivered", observedByRunId: null } }), + NOW, + ), + ).toBe(false); + expect( + isRemindable( + delegatedTask({ completionDelivery: { state: "acknowledged", observedByRunId: null } }), + NOW, + ), + ).toBe(false); + }); + it("never reminds the provider's own subagents", () => { + expect(isRemindable(delegatedTask({ origin: "provider_native" }), NOW)).toBe(false); + }); + it("never reminds a task still running", () => { + expect(isRemindable(delegatedTask({ status: "running" }), NOW)).toBe(false); + }); +}); + +describe("reminderIds", () => { + it("is the same for the same owed tasks in any order, so a repeat is deduped", () => { + const one = reminderIds(PARENT, [NodeId.make("a"), NodeId.make("b")]); + const two = reminderIds(PARENT, [NodeId.make("b"), NodeId.make("a"), NodeId.make("a")]); + expect(two).toEqual(one); + expect(one.messageId.startsWith(`delegated-reminder:${PARENT}:`)).toBe(true); + expect(one.commandId).toBe(`${one.messageId}:0`); + }); + it("differs when another result becomes owed", () => { + expect(reminderIds(PARENT, [NodeId.make("a")])).not.toEqual( + reminderIds(PARENT, [NodeId.make("a"), NodeId.make("b")]), + ); + }); +}); + +describe("activity and slots", () => { + it("reads a remote child that does not answer as unknown, never as live", () => { + expect(remoteActivity(null)).toBe("unknown"); + expect(remoteActivity("running")).toBe("live"); + expect(remoteActivity("waiting")).toBe("live"); + expect(remoteActivity("completed")).toBe("ended"); + }); + it("reads a missing local child as unknown", () => { + expect(localActivity(null)).toBe("unknown"); + }); + it("counts live and unknown children per place, from activity only", () => { + const envB = EnvironmentId.make("env-b"); + expect( + countByPlace([ + { where: "local", activity: "live" }, + { where: "local", activity: "ended" }, + { where: envB, activity: "unknown" }, + { where: envB, activity: "live" }, + { where: EnvironmentId.make("env-c"), activity: "ended" }, + ]), + ).toEqual([ + { where: "local", live: 1, unknown: 0 }, + { where: envB, live: 1, unknown: 1 }, + ]); + expect(countByPlace([])).toEqual([]); + }); + it("treats a parent with a queued run or open question as busy", () => { + const idle = { + activityRunStatus: null, + status: "idle", + pendingRuntimeRequest: null, + pendingBackgroundTasks: [], + } as const; + expect(parentBusy(idle)).toBe(false); + expect(parentBusy({ ...idle, status: "queued" })).toBe(true); + }); +}); + +describe("reminderText", () => { + it("names each task and the tool that reads it", () => { + const text = reminderText([delegatedTask({ id: "t-2" }), delegatedTask({ id: "t-1" })]); + expect(text).toContain("2 delegated tasks"); + expect(text.indexOf("t-1")).toBeLessThan(text.indexOf("t-2")); + expect(text).toContain("task_status"); + }); +}); diff --git a/apps/server/src/fork/delegatedResults/delegatedResults.testkit.ts b/apps/server/src/fork/delegatedResults/delegatedResults.testkit.ts new file mode 100644 index 0000000000000..eff8810911512 --- /dev/null +++ b/apps/server/src/fork/delegatedResults/delegatedResults.testkit.ts @@ -0,0 +1,26 @@ +/** Fork (issue 386): delegated task rows for the tests of `task_list` and the reminder. */ +import { NodeId, type OrchestrationV2Subagent, ThreadId } from "@t3tools/contracts"; +import * as DateTime from "effect/DateTime"; + +export const NOW = Date.parse("2026-11-20T12:00:00.000Z"); +export const PARENT = ThreadId.make("thread-parent"); +export const at = (ms: number) => DateTime.makeUnsafe(ms); + +/** A local task that ended a minute ago with its result claimed but never delivered. */ +export const delegatedTask = ( + overrides: Omit, "id"> & { readonly id?: string } = {}, +): OrchestrationV2Subagent => + ({ + threadId: PARENT, + runId: null, + origin: "app_owned", + childThreadId: ThreadId.make(`child-${overrides.id ?? "task-1"}`), + title: "a task", + status: "completed", + result: "done", + completedAt: at(NOW - 60_000), + updatedAt: at(NOW - 60_000), + completionDelivery: { state: "claimed", observedByRunId: null }, + ...overrides, + id: NodeId.make(overrides.id ?? "task-1"), + }) as unknown as OrchestrationV2Subagent; diff --git a/apps/server/src/fork/delegatedResults/delegatedResults.ts b/apps/server/src/fork/delegatedResults/delegatedResults.ts new file mode 100644 index 0000000000000..34cacc82c3b47 --- /dev/null +++ b/apps/server/src/fork/delegatedResults/delegatedResults.ts @@ -0,0 +1,154 @@ +/** + * Fork (issue 386): pure rules of `task_list` and the delegated result reminder. Upstream's + * `completionDelivery.state`: `pending`/`claimed` = owed, `delivered`/`acknowledged`/`disposed` = + * settled, no state = a row from before it was kept (nothing says whether it was seen). + */ +// @effect-diagnostics-next-line nodeBuiltinImport:off -- The reminder id must be a deterministic digest. +import * as NodeCrypto from "node:crypto"; + +import { + isLiveSubagentShell, + type EnvironmentId, + type NodeId, + type OrchestrationV2RunStatus, + type OrchestrationV2Subagent, + type OrchestrationV2ThreadShell, + type ThreadId, +} from "@t3tools/contracts"; +import * as DateTime from "effect/DateTime"; + +/** Where a task's result stands for its parent. `none`: no delivery state was recorded. */ +export type ResultDelivery = "owed" | "settled" | "none"; + +/** Live: running there now. Ended: not running. Unknown: its environment could not be read. */ +export type ChildActivity = "live" | "ended" | "unknown"; + +const TERMINAL_TASK_STATUSES: ReadonlySet = new Set([ + "completed", + "failed", + "cancelled", + "interrupted", +]); + +const LIVE_RUN_STATUSES: ReadonlySet = new Set([ + "preparing", + "queued", + "starting", + "running", + "waiting", +]); + +/** A task this thread delegated with `delegate_task`: a child here or a thread in a linked one. */ +export const isDelegatedTask = (task: OrchestrationV2Subagent): boolean => + task.origin === "app_owned" && (task.childThreadId !== null || task.remoteChild !== undefined); + +export const isEndedTask = (task: Pick): boolean => + TERMINAL_TASK_STATUSES.has(task.status); + +export const resultDelivery = ( + task: Pick, +): ResultDelivery | null => { + if (!isEndedTask(task)) return null; + const state = task.completionDelivery?.state; + if (state === undefined) return "none"; + return state === "pending" || state === "claimed" ? "owed" : "settled"; +}; + +/** A local child from its thread shell, upstream's own rule for a live child row. */ +export const localActivity = ( + shell: Parameters[0] | null, +): ChildActivity => (shell === null ? "unknown" : isLiveSubagentShell(shell) ? "live" : "ended"); + +/** A remote child from its run's status there; null: the environment did not answer. */ +export const remoteActivity = (status: OrchestrationV2RunStatus | null): ChildActivity => + status === null ? "unknown" : LIVE_RUN_STATUSES.has(status) ? "live" : "ended"; + +/** Where a task runs: `local` for a child thread here, else the linked environment's id. */ +export const taskWhere = (task: OrchestrationV2Subagent): "local" | EnvironmentId => + task.childThreadId !== null ? "local" : (task.remoteChild?.environmentId ?? "local"); + +export interface PlaceCount { + readonly where: "local" | EnvironmentId; + readonly live: number; + readonly unknown: number; +} + +/** Live and unread children per place, from their live activity only (never from any notes). */ +export const countByPlace = ( + rows: ReadonlyArray<{ + readonly where: "local" | EnvironmentId; + readonly activity: ChildActivity; + }>, +): ReadonlyArray => { + const counts = new Map(); + for (const row of rows) { + const count = counts.get(row.where) ?? { live: 0, unknown: 0 }; + if (row.activity === "live") count.live += 1; + if (row.activity === "unknown") count.unknown += 1; + counts.set(row.where, count); + } + return [...counts.entries()] + .filter(([, count]) => count.live > 0 || count.unknown > 0) + .map(([where, count]) => ({ where: where as "local" | EnvironmentId, ...count })) + .toSorted((a, b) => + a.where === "local" ? -1 : b.where === "local" ? 1 : a.where < b.where ? -1 : 1, + ); +}; + +/** A reminder looks back this far; an older owed result belongs to an abandoned thread. */ +export const REMINDER_MAX_AGE_MS = 24 * 60 * 60 * 1000; + +/** + * Owed and recent enough to remind: ended with an owed delivery state in the last day. Tasks with + * no delivery state (rows from before upstream kept one) are never reminded. + */ +export const isRemindable = (task: OrchestrationV2Subagent, nowMs: number): boolean => { + if (!isDelegatedTask(task) || resultDelivery(task) !== "owed") return false; + const ended = task.completedAt ?? task.updatedAt; + return nowMs - DateTime.toEpochMillis(ended) <= REMINDER_MAX_AGE_MS; +}; + +/** Busy: the parent is working or holds a question, so its own turn end comes first. */ +export const parentBusy = (shell: Parameters[0]): boolean => + isLiveSubagentShell(shell); + +/** Gone: archived or deleted. It is never reminded. */ +export const parentGone = ( + shell: Pick | null, +): boolean => shell === null || shell.archivedAt !== null || shell.deletedAt !== null; + +const REMINDER_PREFIX = "delegated-reminder:"; + +/** + * The same owed tasks always give the same id, so a check that runs again, after a restart too, is + * answered from the message or its receipt instead of sending a second reminder. + */ +export const reminderIds = ( + parentThreadId: ThreadId, + taskIds: ReadonlyArray, +): { readonly messageId: string; readonly commandId: string } => { + const digest = NodeCrypto.createHash("sha256") + .update([...new Set(taskIds)].toSorted().join("\n")) + .digest("hex") + .slice(0, 16); + const messageId = `${REMINDER_PREFIX}${parentThreadId}:${digest}`; + return { messageId, commandId: `${messageId}:0` }; +}; + +const MAX_LISTED = 10; + +export const reminderText = (tasks: ReadonlyArray): string => { + const sorted = tasks.toSorted((a, b) => (a.id < b.id ? -1 : 1)); + const lines = sorted.slice(0, MAX_LISTED).map((task) => { + const title = task.title ?? task.remoteChild?.title ?? task.remoteChild?.label ?? "untitled"; + return `- ${task.id} "${title}" (${task.status})`; + }); + const more = sorted.length > MAX_LISTED ? [`- and ${sorted.length - MAX_LISTED} more`] : []; + const count = sorted.length === 1 ? "1 delegated task" : `${sorted.length} delegated tasks`; + return [ + `[delegated results] ${count} ended, but the result never reached this thread:`, + ...lines, + ...more, + "Read each with task_status (that marks it read). task_list lists every task with an unread result.", + ].join("\n"); +}; diff --git a/apps/server/src/fork/forkLayers.ts b/apps/server/src/fork/forkLayers.ts index e60e00a694b67..673ab46eba951 100644 --- a/apps/server/src/fork/forkLayers.ts +++ b/apps/server/src/fork/forkLayers.ts @@ -16,6 +16,7 @@ import * as UsagePacer from "../usage/UsagePacer.ts"; import * as UsageReadingsHistory from "../usage/UsageReadingsHistory.ts"; import * as WorkAdmission from "../workAdmission.ts"; import * as ActivityProbe from "./activityProbe.ts"; +import * as DelegatedResultReminder from "./delegatedResults/DelegatedResultReminder.ts"; import * as HandoffOwner from "./handoff/handoffOwner.ts"; import * as HandoffService from "./handoff/HandoffService.ts"; import * as RunRecorder from "./runRecords/RunRecorder.ts"; @@ -36,7 +37,7 @@ export const holdWorkCloudLink = ( * In the runtime's core dependencies (server.ts's `layerRuntimeCoreDependenciesBase`, not its local * `layerRuntimeServices`): usage pacer (and its history), slot gate, usage refresher, the handoff * service (fork/handoff), which upstream's own `mergeAll` builds concurrently beside its idle - * compaction worker, and the run records follower (fork/runRecords). + * compaction worker, the run records follower (fork/runRecords) and the delegated result reminder. */ export const layerRuntimeServices = Layer.mergeAll( UsagePacer.layer.pipe(Layer.provideMerge(UsageReadingsHistory.layer)), @@ -44,6 +45,7 @@ export const layerRuntimeServices = Layer.mergeAll( HeadlessUsageRefresher.layer, HandoffService.layer, RunRecorder.layer, + DelegatedResultReminder.layer, ); /** diff --git a/apps/server/src/mcp/toolkits/waits/handlers.ts b/apps/server/src/mcp/toolkits/waits/handlers.ts index 2384dc68afbc5..d2f8f9066fa92 100644 --- a/apps/server/src/mcp/toolkits/waits/handlers.ts +++ b/apps/server/src/mcp/toolkits/waits/handlers.ts @@ -11,6 +11,7 @@ import { loadCaller, unavailable, } from "../../threadAccess.ts"; +import { listTasks } from "./taskList.ts"; import { WaitsToolkit } from "./tools.ts"; /** @@ -101,6 +102,7 @@ const handlers = { return yield* waits.cancel(request); }), ), + task_list: McpToolAccess.readsAsCaller((input) => listTasks(input)), } satisfies McpToolAccess.Handlers; export const layer = McpToolAccess.toLayer(WaitsToolkit, handlers); diff --git a/apps/server/src/mcp/toolkits/waits/taskList.test.ts b/apps/server/src/mcp/toolkits/waits/taskList.test.ts new file mode 100644 index 0000000000000..a27eb7a67a2d5 --- /dev/null +++ b/apps/server/src/mcp/toolkits/waits/taskList.test.ts @@ -0,0 +1,338 @@ +/** + * Fork (issue 386): `task_list` through the registered tool. The state question: a linked + * environment that went away (its tasks read as unknown, never as live slots), settled results + * (left out unless asked), old rows with no delivery record, and an empty thread. + */ +import * as NodeServices from "@effect/platform-node/NodeServices"; +import { assert, describe, expect, it } from "@effect/vitest"; +import { + EnvironmentId, + type OrchestrationV2Subagent, + type OrchestrationV2ThreadShell, + ProviderInstanceId, + ThreadId, +} from "@t3tools/contracts"; +import * as Effect from "effect/Effect"; +import * as Fiber from "effect/Fiber"; +import * as Layer from "effect/Layer"; +import { McpSchema, McpServer } from "effect/ai"; +import { TestClock } from "effect/testing"; + +import { delegatedTask, PARENT } from "../../../fork/delegatedResults/delegatedResults.testkit.ts"; +import * as ThreadManagement from "../../../orchestration-v2/ThreadManagementService.ts"; +import * as PeerForwarding from "../../../peer/PeerForwarding.ts"; +import * as RemoteDelegation from "../../../peer/RemoteDelegation.ts"; +import { ServerSettingsService } from "../../../serverSettings.ts"; +import { Waits } from "../../../waits/WaitWorker.ts"; +import * as McpHttpServer from "../../McpHttpServer.ts"; +import * as McpInvocationContext from "../../McpInvocationContext.ts"; +import * as McpToolAccessTestkit from "../../McpToolAccess.testkit.ts"; +import { REMOTE_READ_TIMEOUT } from "./taskList.ts"; + +const HERE = EnvironmentId.make("env-here"); +const ENV_UP = EnvironmentId.make("env-up"); +const ENV_GONE = EnvironmentId.make("env-gone"); + +const remoteTask = (id: string, environmentId: EnvironmentId, status = "running") => + delegatedTask({ + id, + childThreadId: null, + remoteChild: { + environmentId, + threadId: ThreadId.make(`remote-${id}`), + label: environmentId, + runId: `run-${id}` as never, + }, + status: status as OrchestrationV2Subagent["status"], + ...(status === "running" ? { completionDelivery: undefined, completedAt: null } : {}), + }); + +const shell = ( + id: ThreadId, + overrides: Partial = {}, +): OrchestrationV2ThreadShell => ({ + ...McpToolAccessTestkit.liveThreadShell(id, { activeRunId: null }), + status: "idle", + ...overrides, +}); + +const scope: McpInvocationContext.McpInvocationScope = { + environmentId: HERE, + requestNamespace: "task-list", + thread: { + threadId: PARENT, + providerSessionId: "task-list", + providerInstanceId: ProviderInstanceId.make("codex"), + }, + client: undefined, + issuedAt: 0, + capabilities: new Set(["orchestration"]), +}; + +const client = McpSchema.McpServerClient.of({ + clientId: 1, + protocolVersion: "2025-06-18", + clientCapabilities: {}, + clientInfo: { name: "task-list", version: "1" }, + initializePayload: { + protocolVersion: "2025-06-18", + capabilities: {}, + clientInfo: { name: "task-list", version: "1" }, + }, + getClient: Effect.die("unused"), +}); + +interface World { + readonly tasks: ReadonlyArray; + readonly shells: ReadonlyMap; + /** Local children of this thread, as upstream lists them (also those of settled tasks). */ + readonly children?: ReadonlyArray; +} + +const toolsLayer = (world: World) => { + let omitSettled: boolean | undefined; + const services = Layer.mergeAll( + ServerSettingsService.layerTest({}), + Layer.mock(ThreadManagement.ThreadManagementService)({ + getThreadShell: (threadId) => + Effect.succeed( + threadId === PARENT + ? McpToolAccessTestkit.liveThreadShell(PARENT) + : (world.shells.get(threadId) ?? null), + ), + getThreadRecords: (( + _threadId: ThreadId, + _fields: unknown, + filter?: { omitSettledSubagents?: true }, + ) => { + omitSettled = filter?.omitSettledSubagents === true; + return Effect.succeed({ + subagents: omitSettled + ? world.tasks.filter( + (task) => + !["delivered", "acknowledged", "disposed"].includes( + task.completionDelivery?.state ?? "", + ), + ) + : world.tasks, + }); + }) as never, + listAppOwnedChildThreadIds: () => + Effect.succeed( + world.children ?? + world.tasks.flatMap((task) => + task.origin === "app_owned" && task.childThreadId !== null + ? [task.childThreadId] + : [], + ), + ), + }), + Layer.mock(Waits)({}), + Layer.mock(RemoteDelegation.RemoteDelegation)({ + // The environment that went away never answers; the other one says the run is going. + runStatus: (_scope, task) => + task.remoteChild?.environmentId === ENV_GONE ? Effect.never : Effect.succeed("running"), + }), + Layer.mock(PeerForwarding.PeerForwarding)({ + links: () => + Effect.succeed({ + environmentId: HERE, + links: [ENV_UP, ENV_GONE].map((environmentId) => ({ + environmentId, + label: environmentId, + status: "reachable" as const, + access: {} as never, + expiresAt: "2027-01-01T00:00:00.000Z", + lastError: null, + })), + notListed: "", + }), + call: ((_scope: unknown, _tool: unknown, environmentId: EnvironmentId) => + environmentId === ENV_GONE + ? Effect.never + : Effect.succeed({ + projectId: null, + currentThreadId: null, + nextCursor: null, + total: 3, + threads: [ + // Followed by a task here. + { + threadId: ThreadId.make("remote-r-up"), + title: "known", + delegatedFrom: { environmentId: HERE, threadId: PARENT, title: "p" }, + }, + // Launched by this thread, but its record never ran. + { + threadId: ThreadId.make("remote-lost"), + title: "lost one", + delegatedFrom: { environmentId: HERE, threadId: PARENT, title: "p" }, + }, + // Another thread's child. + { + threadId: ThreadId.make("remote-other"), + title: "other", + delegatedFrom: { + environmentId: HERE, + threadId: ThreadId.make("thread-other"), + title: "o", + }, + }, + { threadId: ThreadId.make("remote-own"), title: "its own thread" }, + ], + })) as never, + }), + ).pipe(Layer.provideMerge(NodeServices.layer)); + return McpHttpServer.layerForkTools.pipe( + Layer.provideMerge(McpHttpServer.layerInProcessMcpServer), + Layer.provideMerge(services), + ); +}; + +const callTaskList = (args: Record) => + Effect.gen(function* () { + const server = yield* McpServer.McpServer; + const result = yield* server + .callTool({ name: "task_list", arguments: args }) + .pipe( + Effect.provideService(McpInvocationContext.McpInvocationContext, scope), + Effect.provideService(McpSchema.McpServerClient, client), + ); + assert.notStrictEqual(result.isError, true, JSON.stringify(result.content)); + return result.structuredContent as { + readonly tasks: ReadonlyArray>; + readonly owed: number; + readonly unrecorded: number; + readonly slots: ReadonlyArray<{ where: string; live: number; unknown: number }>; + readonly orphans?: ReadonlyArray<{ environmentId: string; threadId: string }>; + readonly unreachable?: ReadonlyArray; + readonly next: string; + }; + }); + +/** Runs the call while the clock moves past each remote read timeout (activity, then links). */ +const withTimeouts = (effect: Effect.Effect) => + Effect.gen(function* () { + const fiber = yield* Effect.forkChild(effect); + for (let step = 0; step < 4; step++) { + yield* Effect.repeat(Effect.yieldNow, { times: 20 }); + yield* TestClock.adjust(REMOTE_READ_TIMEOUT); + } + return yield* Fiber.join(fiber); + }); + +const mixedWorld: World = { + tasks: [ + // Local, ended, result claimed but never delivered. + delegatedTask({ id: "l-owed" }), + // Local, ended, read already. + delegatedTask({ + id: "l-read", + completionDelivery: { state: "acknowledged", observedByRunId: null }, + }), + // Local, ended before delivery records were kept. + delegatedTask({ id: "l-old", completionDelivery: undefined }), + // Local, still running. + delegatedTask({ id: "l-live", status: "running", completionDelivery: undefined }), + remoteTask("r-up", ENV_UP), + remoteTask("r-gone", ENV_GONE), + // Remote, ended with an owed result: its end is recorded, no read there. + remoteTask("r-done", ENV_GONE, "completed"), + // The provider's own subagent is never listed. + delegatedTask({ id: "native", origin: "provider_native" }), + ], + shells: new Map([ + [ThreadId.make("child-l-owed"), shell(ThreadId.make("child-l-owed"))], + [ThreadId.make("child-l-read"), shell(ThreadId.make("child-l-read"))], + [ThreadId.make("child-l-old"), shell(ThreadId.make("child-l-old"))], + [ + ThreadId.make("child-l-live"), + shell(ThreadId.make("child-l-live"), { status: "running", activityRunStatus: "running" }), + ], + ]), +}; + +describe("task_list", () => { + it.effect("lists unread results, local and remote, and live slots from live status", () => + Effect.gen(function* () { + const listed = yield* withTimeouts(callTaskList({})); + const byId = Object.fromEntries(listed.tasks.map((task) => [task.taskId, task])); + expect(Object.keys(byId).toSorted()).toEqual( + ["l-live", "l-old", "l-owed", "r-done", "r-gone", "r-up"].toSorted(), + ); + expect(byId["l-owed"]).toMatchObject({ result: "owed", activity: "ended", where: "local" }); + expect(byId["l-old"]).toMatchObject({ result: "none" }); + expect(byId["l-live"]).toMatchObject({ result: null, activity: "live" }); + expect(byId["r-up"]).toMatchObject({ activity: "live", where: ENV_UP }); + // The environment that went away: unknown, never counted as a running child. + expect(byId["r-gone"]).toMatchObject({ activity: "unknown", where: ENV_GONE }); + expect(byId["r-done"]).toMatchObject({ activity: "ended", result: "owed" }); + expect(listed.owed).toBe(2); + expect(listed.unrecorded).toBe(1); + expect(listed.slots).toEqual([ + { where: "local", live: 1, unknown: 0 }, + { where: ENV_GONE, live: 0, unknown: 1 }, + { where: ENV_UP, live: 1, unknown: 0 }, + ]); + expect(listed.orphans).toBeUndefined(); + expect(listed.next).toContain("task_status"); + }).pipe(Effect.provide(toolsLayer(mixedWorld))), + ); + + it.effect("lists settled results too when asked", () => + Effect.gen(function* () { + const listed = yield* withTimeouts(callTaskList({ includeSettled: true })); + expect(listed.tasks.find((task) => task.taskId === "l-read")).toMatchObject({ + result: "settled", + }); + expect(listed.owed).toBe(2); + }).pipe(Effect.provide(toolsLayer(mixedWorld))), + ); + + it.effect("counts a live child of a settled task as a slot", () => + Effect.gen(function* () { + const listed = yield* callTaskList({}); + expect(listed.tasks).toEqual([]); + expect(listed.slots).toEqual([{ where: "local", live: 1, unknown: 0 }]); + }).pipe( + Effect.provide( + toolsLayer({ + tasks: [ + delegatedTask({ + id: "sent-more", + completionDelivery: { state: "acknowledged", observedByRunId: null }, + }), + ], + shells: new Map([ + [ + ThreadId.make("child-sent-more"), + shell(ThreadId.make("child-sent-more"), { + status: "running", + activityRunStatus: "running", + }), + ], + ]), + }), + ), + ), + ); + + it.effect("finds a thread launched in a linked environment that no task follows", () => + Effect.gen(function* () { + const listed = yield* withTimeouts(callTaskList({ checkLinked: true })); + expect(listed.orphans).toEqual([ + { environmentId: ENV_UP, threadId: "remote-lost", title: "lost one" }, + ]); + expect(listed.unreachable).toEqual([ENV_GONE]); + expect(listed.next).toContain("t3_thread_read"); + }).pipe(Effect.provide(toolsLayer(mixedWorld))), + ); + + it.effect("an empty thread lists nothing and owes nothing", () => + Effect.gen(function* () { + const listed = yield* callTaskList({}); + expect(listed).toMatchObject({ tasks: [], owed: 0, unrecorded: 0, slots: [] }); + expect(listed.next).toContain("Nothing unread"); + }).pipe(Effect.provide(toolsLayer({ tasks: [], shells: new Map(), children: [] }))), + ); +}); diff --git a/apps/server/src/mcp/toolkits/waits/taskList.ts b/apps/server/src/mcp/toolkits/waits/taskList.ts new file mode 100644 index 0000000000000..d716cc980cb6d --- /dev/null +++ b/apps/server/src/mcp/toolkits/waits/taskList.ts @@ -0,0 +1,243 @@ +/** + * Fork (issue 386): `task_list`, the calling thread's delegated tasks with where each result stands + * and how many children run now per place, read from their live status. Read only: reading here + * marks nothing read; `task_status` does. The rules are in `fork/delegatedResults`. + */ +import { + EnvironmentId, + NodeId, + OrchestratorMcpFailure, + type OrchestrationV2Subagent, + ThreadId, +} from "@t3tools/contracts"; +import * as DateTime from "effect/DateTime"; +import * as Duration from "effect/Duration"; +import * as Effect from "effect/Effect"; +import * as Schema from "effect/Schema"; + +import { + countByPlace, + isDelegatedTask, + isEndedTask, + localActivity, + remoteActivity, + resultDelivery, + taskWhere, +} from "../../../fork/delegatedResults/delegatedResults.ts"; +import * as PeerForwarding from "../../../peer/PeerForwarding.ts"; +import * as RemoteDelegation from "../../../peer/RemoteDelegation.ts"; +import type { McpThreadInvocationScope } from "../../McpInvocationContext.ts"; +import { readCaller, unavailable } from "../../threadAccess.ts"; +import { OrchestratorToolkit } from "../orchestrator/tools.ts"; + +export const TaskListInput = Schema.Struct({ + includeSettled: Schema.optionalKey( + Schema.Boolean.annotate({ + description: "Also list read or delivered results.", + }), + ), + checkLinked: Schema.optionalKey( + Schema.Boolean.annotate({ + description: + "Also ask linked environments for threads launched from here with no task. Slower.", + }), + ), +}); +export type TaskListInput = typeof TaskListInput.Type; + +const Where = Schema.Union([Schema.Literal("local"), EnvironmentId]); + +export const TaskListEntry = Schema.Struct({ + taskId: NodeId, + title: Schema.NullOr(Schema.String), + where: Where, + threadId: ThreadId, + status: Schema.String, + /** live: running now; ended; unknown: its environment did not answer. */ + activity: Schema.Literals(["live", "ended", "unknown"]), + /** owed: ended, result not delivered; settled: delivered or read; none: no record; null: running. */ + result: Schema.NullOr(Schema.Literals(["owed", "settled", "none"])), + completedAt: Schema.NullOr(Schema.String), +}); + +export const TaskListResult = Schema.Struct({ + tasks: Schema.Array(TaskListEntry), + /** Ended tasks whose result never reached this thread. */ + owed: Schema.Number, + /** Ended tasks with no delivery record (from before it was kept): read them to be sure. */ + unrecorded: Schema.Number, + /** Children running now per place, from their live status; `unknown` could not be read. */ + slots: Schema.Array(Schema.Struct({ where: Where, live: Schema.Number, unknown: Schema.Number })), + /** With `checkLinked`: threads there that name this thread as their parent but have no task. */ + orphans: Schema.optionalKey( + Schema.Array( + Schema.Struct({ environmentId: EnvironmentId, threadId: ThreadId, title: Schema.String }), + ), + ), + /** With `checkLinked`: linked environments that did not answer. */ + unreachable: Schema.optionalKey(Schema.Array(EnvironmentId)), + next: Schema.String, +}); +export type TaskListResult = typeof TaskListResult.Type; + +/** The longest one linked environment may take to answer before it counts as unknown. */ +export const REMOTE_READ_TIMEOUT = Duration.seconds(5); +const REMOTE_READ_CONCURRENCY = 4; + +const iso = (time: DateTime.Utc | null) => (time === null ? null : DateTime.formatIso(time)); + +export const listTasks = Effect.fn("waits.taskList")(function* (input: TaskListInput) { + const { scope, threads, caller } = yield* readCaller(); + if (caller === undefined || scope.thread === undefined) { + return yield* new OrchestratorMcpFailure({ + code: "capability_denied", + message: "Tasks belong to a thread. This MCP client is not running inside a T3 thread.", + }); + } + const threadScope = scope as McpThreadInvocationScope; + const remote = yield* RemoteDelegation.RemoteDelegation; + const records = yield* threads + .getThreadRecords( + caller.id, + ["subagents"], + input.includeSettled === true || input.checkLinked === true + ? {} + : { omitSettledSubagents: true }, + ) + .pipe(Effect.mapError(unavailable)); + const all = records.subagents.filter( + (task) => isDelegatedTask(task) && task.threadId === caller.id, + ); + const listed = + input.includeSettled === true ? all : all.filter((task) => resultDelivery(task) !== "settled"); + + const activityOf = (task: OrchestrationV2Subagent) => + task.childThreadId !== null + ? threads.getThreadShell(task.childThreadId).pipe( + Effect.map(localActivity), + Effect.orElseSucceed(() => "unknown" as const), + ) + : // A follower already recorded the end of a remote task; only an open one is read there. + isEndedTask(task) + ? Effect.succeed("ended" as const) + : remote.runStatus(threadScope, task).pipe( + Effect.timeoutOption(REMOTE_READ_TIMEOUT), + Effect.map((status) => remoteActivity(status._tag === "Some" ? status.value : null)), + ); + + const tasks = yield* Effect.forEach( + listed, + (task) => + activityOf(task).pipe( + Effect.map((activity) => ({ + taskId: task.id, + title: task.title ?? task.remoteChild?.title ?? null, + where: taskWhere(task), + threadId: (task.childThreadId ?? task.remoteChild?.threadId)!, + status: task.status, + activity, + result: resultDelivery(task), + completedAt: iso(task.completedAt), + })), + ), + { concurrency: REMOTE_READ_CONCURRENCY }, + ); + + // Slots count every local child, also one of a settled task that was sent more work since. + const listedChildren = new Set(tasks.map((task) => task.threadId)); + const otherChildren = (yield* threads + .listAppOwnedChildThreadIds(caller.id) + .pipe(Effect.orElseSucceed((): ReadonlyArray => []))).filter( + (threadId) => !listedChildren.has(threadId), + ); + const otherActivity = yield* Effect.forEach( + otherChildren, + (threadId) => + threads.getThreadShell(threadId).pipe( + Effect.map((shell) => ({ where: "local" as const, activity: localActivity(shell) })), + Effect.orElseSucceed(() => ({ where: "local" as const, activity: "unknown" as const })), + ), + { concurrency: REMOTE_READ_CONCURRENCY }, + ); + + const owed = tasks.filter((task) => task.result === "owed").length; + const unrecorded = tasks.filter((task) => task.result === "none").length; + const linked: Partial> = + input.checkLinked === true ? yield* findOrphans(threadScope, caller.id, all) : {}; + const orphanCount = linked.orphans?.length ?? 0; + return { + tasks, + owed, + unrecorded, + slots: countByPlace([...tasks, ...otherActivity]), + ...linked, + next: + owed + unrecorded + orphanCount === 0 + ? "Nothing unread. Slots show the children running now." + : `Read each owed or unrecorded task with task_status; that marks it read.${ + orphanCount === 0 + ? "" + : " An orphan runs there with no task here: read it with t3_thread_read and its environmentId." + }`, + } satisfies TaskListResult; +}); + +/** + * Threads in linked environments whose `delegatedFrom` names this thread but that no task here + * follows: a launch whose record never ran. Reads the newest 100 threads of each environment. + */ +const findOrphans = Effect.fn("waits.taskList.orphans")(function* ( + scope: McpThreadInvocationScope, + callerId: ThreadId, + tasks: ReadonlyArray, +) { + // An agent of a linked environment would be pointed at this machine's links. + if (scope.linkedEnvironments === false || scope.client?.linked === true) { + return { orphans: [], unreachable: [] }; + } + const forwarding = yield* PeerForwarding.PeerForwarding; + const links = yield* forwarding.links(scope).pipe(Effect.orElseSucceed(() => null)); + if (links === null) return { orphans: [], unreachable: [] }; + const known = new Set( + tasks.flatMap((task) => + task.remoteChild === undefined + ? [] + : [`${task.remoteChild.environmentId}\n${task.remoteChild.threadId}`], + ), + ); + const unreachable: Array = []; + const orphans: Array<{ environmentId: EnvironmentId; threadId: ThreadId; title: string }> = []; + yield* Effect.forEach( + links.links.filter((link) => link.status !== "expired"), + (link) => + forwarding + .call(scope, OrchestratorToolkit.tools.t3_thread_list, link.environmentId, { + allProjects: true, + limit: 100, + }) + .pipe( + Effect.timeoutOption(REMOTE_READ_TIMEOUT), + Effect.map((listed) => { + if (listed._tag === "None") return void unreachable.push(link.environmentId); + for (const thread of listed.value.threads) { + const from = thread.delegatedFrom; + if ( + from !== undefined && + from.environmentId === scope.environmentId && + from.threadId === callerId && + !known.has(`${link.environmentId}\n${thread.threadId}`) + ) { + orphans.push({ + environmentId: link.environmentId, + threadId: thread.threadId, + title: thread.title, + }); + } + } + }), + Effect.orElseSucceed(() => void unreachable.push(link.environmentId)), + ), + { concurrency: REMOTE_READ_CONCURRENCY, discard: true }, + ); + return { orphans, unreachable }; +}); diff --git a/apps/server/src/mcp/toolkits/waits/tools.ts b/apps/server/src/mcp/toolkits/waits/tools.ts index a4db05133a19a..0b82cb1c4156d 100644 --- a/apps/server/src/mcp/toolkits/waits/tools.ts +++ b/apps/server/src/mcp/toolkits/waits/tools.ts @@ -17,8 +17,11 @@ import * as Schema from "effect/Schema"; import { McpSchema, Tool, Toolkit } from "effect/ai"; import * as ThreadManagementService from "../../../orchestration-v2/ThreadManagementService.ts"; +import * as PeerForwarding from "../../../peer/PeerForwarding.ts"; +import * as RemoteDelegation from "../../../peer/RemoteDelegation.ts"; import { Waits } from "../../../waits/WaitWorker.ts"; import * as McpInvocationContext from "../../McpInvocationContext.ts"; +import { TaskListInput, TaskListResult } from "./taskList.ts"; const dependencies = [ McpInvocationContext.McpInvocationContext, @@ -76,4 +79,26 @@ const WaitCancelTool = Tool.make("wait_cancel", { .annotate(Tool.Idempotent, true) .annotate(Tool.OpenWorld, false); -export const WaitsToolkit = Toolkit.make(WaitForTool, WaitListTool, WaitCancelTool); +// Fork (issue 386): the caller's delegated tasks with unread results and live slot use. +const TaskListTool = Tool.make("task_list", { + description: + "List this thread's delegated tasks with an unread result (local and linked) and its running children per place, from live status. Marks nothing read; task_status does.", + parameters: TaskListInput, + success: TaskListResult, + failure: OrchestratorMcpFailure, + failureMode: "return", + dependencies: [ + McpInvocationContext.McpInvocationContext, + ThreadManagementService.ThreadManagementService, + RemoteDelegation.RemoteDelegation, + PeerForwarding.PeerForwarding, + ], +}) + .annotate(Tool.Title, "List delegated tasks") + .annotate(Tool.Readonly, true) + .annotate(Tool.Destructive, false) + .annotate(Tool.Idempotent, true) + .annotate(Tool.OpenWorld, false) + .annotate(McpSchema.EnabledWhen, McpInvocationContext.hiddenFromOutsideClients); + +export const WaitsToolkit = Toolkit.make(WaitForTool, WaitListTool, WaitCancelTool, TaskListTool); diff --git a/apps/server/src/mcp/toolkits/waits/waits.test.ts b/apps/server/src/mcp/toolkits/waits/waits.test.ts index d8f3d7f988ef5..09bf5f2bb5140 100644 --- a/apps/server/src/mcp/toolkits/waits/waits.test.ts +++ b/apps/server/src/mcp/toolkits/waits/waits.test.ts @@ -17,6 +17,8 @@ import { McpSchema, McpServer, Tool } from "effect/ai"; import { TestClock } from "effect/testing"; import * as ThreadManagement from "../../../orchestration-v2/ThreadManagementService.ts"; +import * as PeerForwarding from "../../../peer/PeerForwarding.ts"; +import * as RemoteDelegation from "../../../peer/RemoteDelegation.ts"; import { ServerSettingsService } from "../../../serverSettings.ts"; import type { WaitDelivery } from "../../../waits/WaitDelivery.ts"; import * as WaitService from "../../../waits/WaitService.ts"; @@ -98,6 +100,9 @@ const toolsLayer = (input: { const base = Layer.mergeAll( ServerSettingsService.layerTest(input.settings ?? {}), threadsLayer(input.shells ?? new Map([[CALLER, callerShell()]])), + // task_list's services; these tests never call it. + Layer.mock(RemoteDelegation.RemoteDelegation)({}), + Layer.mock(PeerForwarding.PeerForwarding)({}), ); const made = WaitService.make({ filePath: input.filePath, @@ -691,14 +696,16 @@ it.layer(NodeServices.layer)("wait tools", (it) => { const { filePath } = yield* workspace; yield* Effect.gen(function* () { const server = yield* McpServer.McpServer; - const listed = server.tools.map(({ tool }) => ({ + const all = server.tools.map(({ tool }) => ({ name: tool.name, description: tool.description ?? "", })); assert.deepStrictEqual( - listed.map((tool) => tool.name), - ["wait_for", "wait_list", "wait_cancel"], + all.map((tool) => tool.name), + ["wait_for", "wait_list", "wait_cancel", "task_list"], ); + // task_list shares the toolkit; its own budget is ForkAgentContextBudget's. + const listed = all.filter((tool) => tool.name.startsWith("wait_")); const total = listed.reduce((sum, tool) => sum + tool.description.length, 0); assert.isAtMost(total, 900); const waitFor = listed[0]!.description; diff --git a/apps/server/src/provider/ForkAgentContextBudget.test.ts b/apps/server/src/provider/ForkAgentContextBudget.test.ts index 6e6719cdd68e0..ed5d338fe38e7 100644 --- a/apps/server/src/provider/ForkAgentContextBudget.test.ts +++ b/apps/server/src/provider/ForkAgentContextBudget.test.ts @@ -11,6 +11,8 @@ import { McpServer } from "effect/ai"; import * as McpHttpServer from "../mcp/McpHttpServer.ts"; import { ThreadManagementService } from "../orchestration-v2/ThreadManagementService.ts"; +import { PeerForwarding } from "../peer/PeerForwarding.ts"; +import { RemoteDelegation } from "../peer/RemoteDelegation.ts"; import { Waits } from "../waits/WaitWorker.ts"; import budget from "./forkAgentContextBudget.json" with { type: "json" }; @@ -18,6 +20,8 @@ import budget from "./forkAgentContextBudget.json" with { type: "json" }; const toolHandlerServices = Layer.mergeAll( Layer.mock(ThreadManagementService)({}), Layer.mock(Waits)({}), + Layer.mock(RemoteDelegation)({}), + Layer.mock(PeerForwarding)({}), ).pipe(Layer.provideMerge(NodeServices.layer)); const utf8Bytes = (text: string) => Buffer.byteLength(text, "utf8"); diff --git a/apps/server/src/provider/forkAgentContextBudget.json b/apps/server/src/provider/forkAgentContextBudget.json index ded23a2552aef..cbc6e8fba205d 100644 --- a/apps/server/src/provider/forkAgentContextBudget.json +++ b/apps/server/src/provider/forkAgentContextBudget.json @@ -1,3 +1,3 @@ { - "mcpToolListJsonBytes": 1979 + "mcpToolListJsonBytes": 2451 } diff --git a/docs/fork/README.md b/docs/fork/README.md index 02f9cfa76c65b..5e3ab8a168aaf 100644 --- a/docs/fork/README.md +++ b/docs/fork/README.md @@ -177,6 +177,13 @@ woken with the result. `wait_list` and `wait_cancel` list and end them. While of judged nor delivered; listing and cancelling still work. Other fields set deadlines, limits per thread and how often each kind is checked. +The same switch turns on the delegated result reminder: when a `delegate_task` child ends and its +result has still not reached the idle parent a couple of minutes later (upstream's own wake did not +land), the parent gets one queued message naming those tasks. It is sent once per set of owed tasks, +also across restarts, never to a busy, stopped or archived thread, and never for results older than +a day. `task_list` (always there, read only) lists a thread's tasks with an unread result and how +many of its children run now per place, from their live status. + Switch on: **Settings → Fork**, section Waits (shown while a server is connected), "Let agents wait for events", or in `settings.json`: `"waitFor": { "enabled": true }`.