Skip to content

Commit 82c4c11

Browse files
nullStack65claude
andauthored
feat(server): deliver messages by class, never into a compaction (#144)
* feat(server): deliver messages by class, never into a compaction A message arriving while the provider is compacting now waits for the compaction to finish instead of aborting it. Senders set a delivery class: interrupt, steer_at_tool_boundary, next_turn, when_idle, or digest. Digests never start a turn and ride with the next one. An undelivered message with the same coalesce key replaces the older one, and one past its expiry is dropped and logged. Alerts default to digest. t3_thread_send accepts the class, coalesce key and expiry, and t3_queue shows them. The pre-commit lint and knip hooks fail in this worktree because Node cannot load the repo's TypeScript oxlint plugin and knip schemas. Both fail the same way on the untouched base commit. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(server): a digest never starts a turn and is never dropped silently Review of the delivery classes found six holes. A slash-prefixed message now carries its digest instead of discarding it. A digest sent to a thread that has never run waits instead of starting a provider turn. An agent send that names no class rides in the digest. A steer waiting for the next tool result releases when that tool is interrupted or cancelled, not only when it completes or fails. An expired digest is dropped even when it is the only queued message. A turn that starts through a fork, merge-back or provider switch carries the digest too. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(server): a message to the sender's own thread keeps its mode The agent digest default keyed off the caller having any thread, so an agent sending its next instruction to its own thread had it held as a digest that never started a turn. The default now applies only when the sender and the target are different threads. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * fix(server): hold an alert behind a queued digest and retry with its expiry A flush read the watcher once and delivered every due alert together, and a queued digest is hidden from the shell, so the next alert never saw the turn the first one queued and queued behind it instead of waiting. Deliveries now go one at a time, and a watcher whose only turn is a waiting digest counts as busy. A retry recomputed the digest expiry from the clock, so it disagreed with the failed send and was not a replay. The expiry is fixed when the alert is queued. Co-Authored-By: Claude Code <noreply@anthropic.com> * fix(web): keep a timed-out draft test from breaking the next one The ARC voice job's first fake-timer test hit the 15s timeout, then the next test failed: that run was still going, so its cleanup uninstalled the clock or its late upload showed up in the next assertion. Import the modules on real timers, leave setImmediate and performance real so the clock can yield, and ignore a superseded test's cleanup. Model: grok-4.7 Task: fix Agent: Builder Lane: grok Harness: t3 --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
1 parent 1095e66 commit 82c4c11

22 files changed

Lines changed: 1047 additions & 135 deletions

‎apps/server/src/mcp/OrchestratorMcpService.ts‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3559,6 +3559,36 @@ const make = Effect.gen(function* () {
35593559
yield* assertTargetWithinLimits(limits, target.thread);
35603560

35613561
const mode = input.mode ?? "auto";
3562+
// A message to another thread rides in that thread's next digest unless the sender
3563+
// names a class. A message to the sender's own thread keeps the mode it asked for.
3564+
const deliveryClass =
3565+
input.deliveryClass ??
3566+
(input.digestCategory !== undefined ||
3567+
(parent !== undefined && parent.thread.id !== input.threadId)
3568+
? "digest"
3569+
: undefined);
3570+
if (
3571+
deliveryClass === undefined &&
3572+
(input.coalesceKey !== undefined || input.expiresAt !== undefined)
3573+
) {
3574+
return yield* failure("invalid_request", "coalesceKey and expiresAt need a deliveryClass.");
3575+
}
3576+
const expiresAt =
3577+
input.expiresAt === undefined
3578+
? undefined
3579+
: Option.getOrUndefined(DateTime.make(input.expiresAt));
3580+
if (input.expiresAt !== undefined && expiresAt === undefined) {
3581+
return yield* failure("invalid_request", "expiresAt must be an ISO date-time.");
3582+
}
3583+
const delivery =
3584+
deliveryClass === undefined
3585+
? undefined
3586+
: {
3587+
class: deliveryClass,
3588+
...(input.coalesceKey === undefined ? {} : { coalesceKey: input.coalesceKey }),
3589+
...(expiresAt === undefined ? {} : { expiresAt: DateTime.toUtc(expiresAt) }),
3590+
...(input.digestCategory === undefined ? {} : { category: input.digestCategory }),
3591+
};
35623592
const key = yield* requestKey(input.clientRequestId);
35633593
const messageId = stableOperationMessageId({
35643594
scope,
@@ -3584,6 +3614,7 @@ const make = Effect.gen(function* () {
35843614
attachments: [],
35853615
mode,
35863616
...(input.freshContext ? { freshContext: true } : {}),
3617+
...(delivery === undefined ? {} : { delivery }),
35873618
createdBy: "agent",
35883619
creationSource: "mcp",
35893620
})
@@ -3626,6 +3657,7 @@ const make = Effect.gen(function* () {
36263657
...(result.run.delegatedTaskId === undefined ? {} : { taskId: result.run.delegatedTaskId }),
36273658
status: result.run.status,
36283659
delivery: result.delivery,
3660+
...(deliveryClass === undefined ? {} : { deliveryClass }),
36293661
} satisfies OrchestratorMcpThreadSendResult;
36303662
}),
36313663
waitForThread: Effect.fn("OrchestratorMcpService.waitForThread")(function* (scope, input) {

‎apps/server/src/mcp/toolkits/orchestrator/tools.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -285,7 +285,7 @@ const ThreadUpdateTool = Tool.make("t3_thread_update", {
285285

286286
const ThreadSendTool = Tool.make("t3_thread_send", {
287287
description:
288-
"Send a message to any thread. The target cannot run with broader modes than yours. `mode:'auto'` starts an idle thread, steers a running turn, or queues. `queue`: new turn. `steer`: mid-turn update. `restart`: interrupt and restart. A send to a finished child thread starts a fresh task and returns its `taskId`. In your reply, name it as its `link` verbatim.",
288+
"Send a message to any thread. The target cannot run with broader modes than yours. `mode:'auto'` starts an idle thread, steers a running turn, or queues. `queue`: new turn. `steer`: mid-turn update. `restart`: interrupt and restart. `deliveryClass` overrides `mode`: `interrupt`, `steer_at_tool_boundary` (after the next tool result), `next_turn`, `when_idle` (once nothing runs), or `digest` (never starts a turn; rides with the next one, grouped by `digestCategory`). `coalesceKey` replaces an undelivered message with the same key; past `expiresAt` an undelivered one is dropped. A send to a finished child thread starts a fresh task and returns its `taskId`. In your reply, name it as its `link` verbatim.",
289289
parameters: OrchestratorMcpThreadSendInput,
290290
success: OrchestratorMcpThreadSendResult,
291291
failure: OrchestratorMcpFailure,

‎apps/server/src/mcp/toolkits/thread/handlers.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import {
77
OrchestratorMcpFailure,
88
type OrchestrationV2Command,
99
} from "@t3tools/contracts";
10+
import * as DateTime from "effect/DateTime";
1011
import * as Effect from "effect/Effect";
1112
import * as Option from "effect/Option";
1213
import { modelSelectionCommandType } from "@t3tools/shared/model";
@@ -42,6 +43,17 @@ function queueEntry(
4243
queuedRunId: run.id,
4344
text: characters.slice(0, limit).join(""),
4445
truncated: characters.length > limit,
46+
...(run.delivery === undefined
47+
? {}
48+
: {
49+
deliveryClass: run.delivery.class,
50+
...(run.delivery.coalesceKey === undefined
51+
? {}
52+
: { coalesceKey: run.delivery.coalesceKey }),
53+
...(run.delivery.expiresAt === undefined
54+
? {}
55+
: { expiresAt: DateTime.formatIso(run.delivery.expiresAt) }),
56+
}),
4557
};
4658
}
4759
const dispatch = Effect.fn("mcp.dispatchThreadCommand")(function* (

‎apps/server/src/mcp/toolkits/thread/tools.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import {
1515
IsoDateTime,
1616
OrchestratorMcpFailure,
1717
OrchestrationV2DispatchCommandResult,
18+
OrchestrationV2DeliveryClass,
1819
ThreadId,
1920
RunId,
2021
NonNegativeInt,
@@ -79,12 +80,15 @@ const queueEntry = Schema.Struct({
7980
queuedRunId: RunId,
8081
text: Schema.String,
8182
truncated: Schema.Boolean,
83+
deliveryClass: Schema.optional(OrchestrationV2DeliveryClass),
84+
coalesceKey: Schema.optional(Schema.String),
85+
expiresAt: Schema.optional(Schema.String),
8286
});
8387
const QUEUE_READ_ACTIONS = ["list", "read"] as const;
8488
const QueueTool = Tool.make("t3_queue", {
8589
...commandTool,
8690
description:
87-
"Manage a thread's queued messages. Omit `threadId` for this thread. list: send order, paged by `cursor` and `limit`. read: up to 16,000 chars of one. edit: replace its `text`; attachments stay. cancel. reorder: before `beforeRunId`, or to the end with null. promote_to_steer: send it now as steering to active `targetRunId`. Writes fail once the run is no longer queued.",
91+
"Manage a thread's queued messages. Omit `threadId` for this thread. list: send order, paged by `cursor` and `limit`. read: up to 16,000 chars of one. edit: replace its `text`; attachments stay. cancel. reorder: before `beforeRunId`, or to the end with null. promote_to_steer: send it now as steering to active `targetRunId`. Entries show their `deliveryClass`, `coalesceKey` and `expiresAt`; a `digest` entry never starts a turn and goes out with the next one. Writes fail once the run is no longer queued.",
8892
parameters: Schema.Struct({
8993
action: Schema.Literals([
9094
...QUEUE_READ_ACTIONS,

‎apps/server/src/orchestration-v2/CommandPolicy.ts‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,23 @@ export function followUpOwnerRun(
135135
);
136136
}
137137

138+
/** Steering a provider mid-compaction aborts the compaction, so steers into it wait. */
139+
export function compactionHoldsSteer(
140+
projection: Pick<
141+
OrchestrationV2ThreadProjection,
142+
"turnItems" | "providerThreads" | "providerSessions"
143+
>,
144+
run: OrchestrationV2Run,
145+
): boolean {
146+
return (
147+
projection.turnItems.some(
148+
(item) => item.runId === run.id && item.type === "compaction" && item.status === "running",
149+
) && providerTurnCapabilities(projection, run)?.activeSteeringDuringCompaction !== true
150+
);
151+
}
152+
138153
function providerTurnCapabilities(
139-
projection: OrchestrationV2ThreadProjection,
154+
projection: Pick<OrchestrationV2ThreadProjection, "providerThreads" | "providerSessions">,
140155
run: OrchestrationV2Run,
141156
) {
142157
const providerThread = projection.providerThreads.find(

‎apps/server/src/orchestration-v2/IdleCompaction.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { isListedThreadLineage, type OrchestrationV2ThreadProjection } from "@t3tools/contracts";
22
import { COMPACTION_SUGGESTION_MIN_TOKENS } from "@t3tools/shared/claudeCompaction";
33
import { derivePendingBackgroundWork } from "@t3tools/shared/orchestrationV2PendingBackgroundWork";
4+
import { isWaitingDigestRun } from "@t3tools/shared/orchestrationV2ThreadError";
45
import * as DateTime from "effect/DateTime";
56

67
// Claude Code's idleCompaction defaults to 200K and won't go below 100K. Below
@@ -53,7 +54,9 @@ export function idleCompactionWindow(
5354
if (!isListedThreadLineage(thread.lineage)) return null;
5455
// A removed or steered queued message leaves a cancelled run that never started.
5556
const latestRun = runs
56-
.filter((run) => run.status !== "cancelled" || run.startedAt !== null)
57+
.filter(
58+
(run) => (run.status !== "cancelled" || run.startedAt !== null) && !isWaitingDigestRun(run),
59+
)
5760
.reduce<(typeof runs)[number] | null>(
5861
(latest, run) => (latest === null || run.ordinal > latest.ordinal ? run : latest),
5962
null,

‎apps/server/src/orchestration-v2/IdleCompactionService.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import {
77
type RunId,
88
type ThreadId,
99
} from "@t3tools/contracts";
10+
import { isWaitingDigestRun } from "@t3tools/shared/orchestrationV2ThreadError";
1011
import * as DateTime from "effect/DateTime";
1112
import * as Duration from "effect/Duration";
1213
import * as Effect from "effect/Effect";
@@ -89,7 +90,10 @@ export const make = Effect.gen(function* () {
8990
)
9091
return;
9192
const latestRun = records.runs
92-
.filter((run) => run.status !== "cancelled" || run.startedAt !== null)
93+
.filter(
94+
(run) =>
95+
(run.status !== "cancelled" || run.startedAt !== null) && !isWaitingDigestRun(run),
96+
)
9397
.reduce<(typeof records.runs)[number] | undefined>(
9498
(latest, run) => (latest === undefined || run.ordinal > latest.ordinal ? run : latest),
9599
undefined,

‎apps/server/src/orchestration-v2/ModelAlertReactor.ts‎

Lines changed: 76 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import {
44
isListedThreadLineage,
55
MessageId,
66
type ModelAlertWatcher,
7+
type OrchestrationV2DigestCategory,
78
type OrchestrationV2DomainEvent,
89
type OrchestrationV2Run,
910
type OrchestrationV2ThreadShell,
@@ -124,6 +125,11 @@ const Delivery = Schema.Struct({
124125
attempts: Schema.Int,
125126
nextAttemptAt: Schema.Number,
126127
createdAt: Schema.Number,
128+
/**
129+
* When the digest expires, fixed when the alert is queued. A retry must carry the same time,
130+
* or the first send and the retry disagree and the retry is not a replay.
131+
*/
132+
digestExpiresAt: Schema.optional(Schema.Number),
127133
});
128134
type Delivery = typeof Delivery.Type;
129135

@@ -306,6 +312,7 @@ function enqueueAlert(
306312
attempts: 0,
307313
nextAttemptAt: nowMs,
308314
createdAt: nowMs,
315+
digestExpiresAt: nowMs + ALERT_DIGEST_TTL_MS,
309316
}));
310317
return {
311318
outbox: [...pruneOutbox(outbox, nowMs), ...added].slice(-MAX_OUTBOX),
@@ -577,6 +584,8 @@ const BUSY_WATCHER_STATUSES = new Set<string>([
577584
/** Matches the flush interval, so a held alert is looked at again on the next flush. */
578585
export const BUSY_WATCHER_RECHECK_MS = 30_000;
579586
export const BUSY_WATCHER_MAX_HOLD_MS = 10 * 60 * 1000;
587+
/** How long an undelivered alert stays in the watcher's digest before it is dropped. */
588+
const ALERT_DIGEST_TTL_MS = 24 * 60 * 60 * 1000;
580589

581590
/**
582591
* While the watcher is mid-turn, a sent alert only queues behind that turn and is judged stale or
@@ -682,6 +691,7 @@ export function recordUsageLimit(
682691
attempts: 0,
683692
nextAttemptAt: notBefore,
684693
createdAt: nowMs,
694+
digestExpiresAt: nowMs + ALERT_DIGEST_TTL_MS,
685695
},
686696
].slice(-MAX_OUTBOX);
687697
}
@@ -736,6 +746,32 @@ export const watcherAlertMessage = (alert: WatcherAlert) => {
736746
}
737747
};
738748

749+
750+
/** Alerts ride in the watcher's next digest; a newer alert on the same subject replaces an older one. */
751+
function alertDelivery(alert: WatcherAlert): {
752+
readonly coalesceKey: string;
753+
readonly category: OrchestrationV2DigestCategory;
754+
} {
755+
switch (alert.type) {
756+
case "peer-link-renewal":
757+
return { coalesceKey: `${alert.type}:${alert.linkedEnvironmentId}`, category: "alert" };
758+
case "usage-limit-incident":
759+
return { coalesceKey: `${alert.type}:${alert.groupKey}`, category: "alert" };
760+
case "model-degraded":
761+
return { coalesceKey: `${alert.type}:${alert.threadId}`, category: "alert" };
762+
case "thread-event":
763+
return {
764+
coalesceKey: `${alert.type}:${alert.kind}:${alert.threadId}`,
765+
category:
766+
alert.kind === "lane-ended"
767+
? "lane_result"
768+
: alert.kind === "needs-person"
769+
? "decision_needed"
770+
: "alert",
771+
};
772+
}
773+
}
774+
739775
/**
740776
* Puts an alert in the watcher thread, here or on the linked machine that owns it. The retry key is
741777
* one per alert and watcher, so a retry after a lost answer sends once.
@@ -749,11 +785,16 @@ export const deliverAlertToWatcher = Effect.fnUntraced(function* (
749785
alert: WatcherAlert,
750786
watcher: ModelAlertWatcher,
751787
here: EnvironmentId,
788+
expiresAtMs?: number,
752789
) {
753790
const message = watcherAlertMessage(alert);
754791
// Run ids can be long; a retry key must stay under the 256 characters the peer accepts.
755792
const identity = `${alert.id}\n${watcher.environmentId}\n${watcher.threadId}`;
756793
const key = `${alert.type === "model-degraded" ? "model-alert" : alert.type}:${yield* deps.sha256Hex(identity)}`;
794+
const { coalesceKey, category } = alertDelivery(alert);
795+
const expiresAt = DateTime.makeUnsafe(
796+
expiresAtMs ?? DateTime.toEpochMillis(yield* DateTime.now) + ALERT_DIGEST_TTL_MS,
797+
);
757798
if (watcher.environmentId === here) {
758799
yield* deps.dispatch({
759800
type: "message.dispatch",
@@ -763,6 +804,7 @@ export const deliverAlertToWatcher = Effect.fnUntraced(function* (
763804
text: message,
764805
attachments: [],
765806
dispatchMode: { type: "queue_after_active" },
807+
delivery: { class: "digest", coalesceKey, expiresAt, category },
766808
createdBy: "agent",
767809
creationSource: "server",
768810
});
@@ -771,7 +813,16 @@ export const deliverAlertToWatcher = Effect.fnUntraced(function* (
771813
yield* deps.call({
772814
environmentId: watcher.environmentId,
773815
tool: "t3_thread_send",
774-
arguments: { threadId: watcher.threadId, message, mode: "queue", clientRequestId: key },
816+
arguments: {
817+
threadId: watcher.threadId,
818+
message,
819+
mode: "queue",
820+
clientRequestId: key,
821+
deliveryClass: "digest",
822+
coalesceKey,
823+
expiresAt: DateTime.formatIso(expiresAt),
824+
digestCategory: category,
825+
},
775826
limits: { runtimeMode: "full-access", interactionMode: "default" },
776827
success: OrchestratorMcpThreadSendResult,
777828
});
@@ -851,12 +902,18 @@ export const make = Effect.gen(function* () {
851902
crypto
852903
.digest(Digest.sha256, new TextEncoder().encode(identity))
853904
.pipe(Effect.map(Hex.encode), Effect.orDie);
854-
const deliverToWatcher = (alert: WatcherAlert, watcher: ModelAlertWatcher, here: EnvironmentId) =>
905+
const deliverToWatcher = (
906+
alert: WatcherAlert,
907+
watcher: ModelAlertWatcher,
908+
here: EnvironmentId,
909+
expiresAtMs: number,
910+
) =>
855911
deliverAlertToWatcher(
856912
{ dispatch: engine.dispatch, call: peers.call, sha256Hex },
857913
alert,
858914
watcher,
859915
here,
916+
expiresAtMs,
860917
);
861918

862919
type WatcherView = {
@@ -993,8 +1050,15 @@ export const make = Effect.gen(function* () {
9931050
if (watcher.environmentId === here) {
9941051
const shell = yield* projections.getThreadShell(watcher.threadId);
9951052
if (shell === null) return null;
1053+
// A queued digest never starts a turn, so the shell hides it and reads idle. It is still a
1054+
// turn waiting to be judged, and another alert queueing behind it would be judged stale.
1055+
const digestWaiting =
1056+
shell.status === "idle" &&
1057+
(yield* projections.getThreadRecords(watcher.threadId, ["runs"])).runs.some(
1058+
(run) => run.status === "queued" && run.delivery?.class === "digest",
1059+
);
9961060
return {
997-
status: shell.status as string,
1061+
status: (digestWaiting ? "queued" : shell.status) as string,
9981062
providerInstanceId: shell.providerInstanceId as string | null,
9991063
usageLimited: shell.lastErrorClass === "usage_limit",
10001064
resetAt: shell.usageLimitResetAt ?? null,
@@ -1088,7 +1152,12 @@ export const make = Effect.gen(function* () {
10881152
holdUntil(view, alert, nowMs) ??
10891153
busyWatcherHold(view?.status ?? null, entry.createdAt, nowMs);
10901154
if (until !== null) return { _tag: "held", until } satisfies DeliveryResult;
1091-
yield* deliverToWatcher(alert, target, here);
1155+
yield* deliverToWatcher(
1156+
alert,
1157+
target,
1158+
here,
1159+
entry.digestExpiresAt ?? nowMs + ALERT_DIGEST_TTL_MS,
1160+
);
10921161
return { _tag: "sent" } satisfies DeliveryResult;
10931162
});
10941163

@@ -1104,6 +1173,8 @@ export const make = Effect.gen(function* () {
11041173
return yield* readWatcher(watcher, yield* environment.getEnvironmentId);
11051174
}).pipe(Effect.catchCause(() => Effect.succeed(null))),
11061175
);
1176+
// One at a time: a delivery queues a turn on the watcher, and the next alert in this
1177+
// flush has to see that turn to hold instead of queueing behind it.
11071178
yield* Effect.forEach(
11081179
due,
11091180
(entry) =>
@@ -1131,7 +1202,7 @@ export const make = Effect.gen(function* () {
11311202
: afterDeliveryAttempt(current, entry, true, attemptedAt),
11321203
]);
11331204
}),
1134-
{ concurrency: 4, discard: true },
1205+
{ concurrency: 1, discard: true },
11351206
);
11361207
});
11371208

0 commit comments

Comments
 (0)