Skip to content

Commit 2beec08

Browse files
Vikhyath Mondreticlaude
andcommitted
fix(billing): meter the sandbox a cloud Pi session runs in
Pi's own sandbox was never metered. withPiSandbox called createSandbox without the meterUsage argument, so only sandboxes created through executeFunctionRequest were charged — and Pi's is the larger consumer by an order of magnitude. A Function block holds one for seconds; a Pi session holds one for a minimum lifetime of 31 minutes. The gap was worst exactly where it was least visible. A Pi coding agent normally runs BYOK, so its model cost is zero by definition, and the ledger bills a model row on total > 0. With the sandbox unmetered, such a run produced a zero-cost model_unbilled row and Sim collected only the flat execution fee while paying its provider for the whole session. Threads a cost sink through PiRunContext, which is the seam backends already receive and the only one that reaches all four cloud modes. The handler owns one sink covering both sandbox sources — Function tools in local mode, the agent's own sandbox in cloud mode — so neither can be dropped where the cost is folded into the block's output. It rides in toolCost for the same reason the Function tool cost already does: that is what survives the BYOK zeroing. Unlike the Function path this charges on creation rather than on a completed session. A Function run is seconds long, so absorbing one the provider failed to deliver is cheap and reads as fair; tens of minutes of Pi compute is consumed whether the agent finished, errored, or was cancelled, and billing only clean endings would mean paying for every other one. A create that throws still costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 933a4bc commit 2beec08

10 files changed

Lines changed: 120 additions & 30 deletions

File tree

‎apps/sim/executor/handlers/pi/cloud/authoring/backend.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -435,7 +435,10 @@ async function runCloudAuthoringPi(
435435
const lifetimeMs = resolvePiRunLifetimeMs(context.signal)
436436
const piTimeoutMs = resolvePiTimeoutMs(lifetimeMs)
437437

438-
const authored = await withPiSandbox<AuthoringPhaseResult>({ lifetimeMs }, async (runner) => {
438+
// Bound to a local so the call stays on one line: inlining the second option
439+
// reflows this whole callback body and buries the change in re-indentation.
440+
const sandboxOptions = { lifetimeMs, cost: context.sandboxCost }
441+
const authored = await withPiSandbox<AuthoringPhaseResult>(sandboxOptions, async (runner) => {
439442
try {
440443
const clone = await raceAbort(
441444
runner.run(params.mode === 'cloud' ? CREATE_PR_CLONE_SCRIPT : UPDATE_BRANCH_CLONE_SCRIPT, {

‎apps/sim/executor/handlers/pi/cloud/babysit/backend.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -784,7 +784,7 @@ export async function runBabysitPiWithOptions(
784784
const lifetimeMs = resolvePiRunLifetimeMs(context.signal)
785785
const piTimeoutMs = resolvePiTimeoutMs(lifetimeMs)
786786

787-
return await withPiSandbox({ lifetimeMs }, async (runner) => {
787+
return await withPiSandbox({ lifetimeMs, cost: context.sandboxCost }, async (runner) => {
788788
const clone = await raceAbort(
789789
runner.run(BABYSIT_CLONE_SCRIPT, {
790790
envs: {

‎apps/sim/executor/handlers/pi/cloud/plan/backend.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ export const runCloudPlanPi: PiBackendRun<PiCloudPlanRunParams> = async (params,
8080
const thinking = mapThinkingLevel(params.thinkingLevel) ?? 'medium'
8181
const lifetimeMs = resolvePiRunLifetimeMs(context.signal)
8282

83-
return withPiSandbox({ lifetimeMs }, async (runner) => {
83+
return withPiSandbox({ lifetimeMs, cost: context.sandboxCost }, async (runner) => {
8484
try {
8585
const clone = await raceAbort(
8686
runner.run(PLAN_CLONE_SCRIPT, {

‎apps/sim/executor/handlers/pi/cloud/review/backend.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ export const runCloudReviewPi: PiBackendRun<PiCloudReviewRunParams> = async (par
218218
const lifetimeMs = resolvePiRunLifetimeMs(context.signal)
219219

220220
try {
221-
return await withPiSandbox({ lifetimeMs }, async (runner) => {
221+
return await withPiSandbox({ lifetimeMs, cost: context.sandboxCost }, async (runner) => {
222222
await runner.writeFile(GIT_ASKPASS_PATH, GIT_ASKPASS_SCRIPT)
223223
const fetched = await raceAbort(
224224
runner.run(FETCH_PR_SCRIPT, {

‎apps/sim/executor/handlers/pi/core/backend.ts‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
*/
1010

1111
import type { TSchema } from 'typebox'
12+
import type { SandboxCostSink } from '@/lib/execution/remote-sandbox/types'
1213
import type { SSHConnectionConfig } from '@/lib/internal/ssh/client'
1314
import type { Message } from '@/executor/handlers/agent/types'
1415
import type { PiEvent, PiRunTotals } from '@/executor/handlers/pi/core/events'
@@ -172,6 +173,16 @@ export type PiRunParams =
172173
export interface PiRunContext {
173174
onEvent: (event: PiEvent) => void
174175
signal?: AbortSignal
176+
/**
177+
* Where a backend reports the cost of Sim-provisioned compute it used.
178+
*
179+
* Only the cloud modes have any: they run the agent in a Sim-paid sandbox,
180+
* while local mode drives the caller's own machine over SSH and costs Sim
181+
* nothing. The handler folds whatever lands here into the block's `toolCost`,
182+
* which is what keeps a BYOK Pi run — model unbilled by definition — from
183+
* reporting no cost at all for a session that ran for tens of minutes.
184+
*/
185+
sandboxCost?: SandboxCostSink
175186
}
176187

177188
/** Final result of a Pi run. */

‎apps/sim/executor/handlers/pi/local/sim-tools.ts‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99

1010
import { createLogger } from '@sim/logger'
1111
import { getErrorMessage } from '@sim/utils/errors'
12+
import type { SandboxCostSink } from '@/lib/execution/remote-sandbox/types'
1213
import {
1314
readWorkflowInputFieldsForTool,
1415
readWorkflowMetadataForTool,
@@ -47,10 +48,6 @@ type PiToolResultProjection =
4748
| { safe: true; result: PiToolResult }
4849
| { safe: false; result: PiToolResult }
4950

50-
export interface PiFunctionToolCostAccumulator {
51-
total: number
52-
}
53-
5451
function projectToolResult(
5552
result: ToolResponse,
5653
registry: ResolvedSecretTraceRegistry | undefined
@@ -112,7 +109,7 @@ function buildSimToolSpec(
112109
inputTools: ToolInput[],
113110
provider: ProviderToolConfig,
114111
toolIndex: number,
115-
functionToolCost?: PiFunctionToolCostAccumulator
112+
sandboxCost?: SandboxCostSink
116113
): PiToolSpec {
117114
const toolId = provider.canonicalId ?? provider.id
118115
const preseededParams = provider.params || {}
@@ -182,12 +179,12 @@ function buildSimToolSpec(
182179
: undefined
183180
if (
184181
toolId === 'function_execute' &&
185-
functionToolCost &&
182+
sandboxCost &&
186183
typeof resultCostTotal === 'number' &&
187184
Number.isFinite(resultCostTotal) &&
188185
resultCostTotal > 0
189186
) {
190-
functionToolCost.total += resultCostTotal
187+
sandboxCost.total += resultCostTotal
191188
}
192189
const projection = projectToolResult(result, toolCallRegistry?.forkForPropagatedEntries())
193190
if (projection.safe && registry && toolCallRegistry?.isComplete()) {
@@ -219,7 +216,7 @@ function buildSimToolSpec(
219216
export async function buildSimToolSpecs(
220217
ctx: ExecutionContext,
221218
inputTools: unknown,
222-
functionToolCost?: PiFunctionToolCostAccumulator
219+
sandboxCost?: SandboxCostSink
223220
): Promise<PiToolSpec[]> {
224221
if (!Array.isArray(inputTools)) return []
225222

@@ -263,6 +260,6 @@ export async function buildSimToolSpecs(
263260
await annotateDuplicateToolBindings(ctx, providers)
264261
assignProviderToolIdentities(providers)
265262
return configuredTools.map(({ provider, toolIndex }) =>
266-
buildSimToolSpec(ctx, inputTools, provider, toolIndex, functionToolCost)
263+
buildSimToolSpec(ctx, inputTools, provider, toolIndex, sandboxCost)
267264
)
268265
}

‎apps/sim/executor/handlers/pi/pi-handler.test.ts‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,7 @@ vi.mock('@/blocks/utils', () => ({
111111
},
112112
}))
113113

114+
import type { PiRunContext } from '@/executor/handlers/pi/core/backend'
114115
import { PiBlockHandler, parsePiReviewMentions } from '@/executor/handlers/pi/pi-handler'
115116
import type { ExecutionContext, StreamingExecution } from '@/executor/types'
116117
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
@@ -291,6 +292,41 @@ describe('PiBlockHandler', () => {
291292
expect(output.cost).toEqual({ input: 0, output: 0, toolCost: 0.125, total: 0.125 })
292293
})
293294

295+
it('bills the cloud sandbox a Pi session ran in, even when the model is BYOK', async () => {
296+
// The regression this guards: the agent's own sandbox runs on Sim's provider
297+
// account, so a BYOK run whose model cost is zero by definition would
298+
// otherwise report no cost at all for tens of minutes of paid compute.
299+
mockRunCloud.mockImplementation(async (_params: unknown, context: PiRunContext) => {
300+
if (context.sandboxCost) context.sandboxCost.total += 0.0842
301+
return { totals: { finalText: 'done', inputTokens: 0, outputTokens: 0 } }
302+
})
303+
304+
const output = (await handler.execute(ctx(), block, {
305+
mode: 'cloud',
306+
task: 'do it',
307+
model: 'claude',
308+
owner: 'o',
309+
repo: 'r',
310+
githubToken: 'ghp',
311+
})) as { cost: unknown }
312+
313+
expect(output.cost).toEqual({ input: 0, output: 0, toolCost: 0.0842, total: 0.0842 })
314+
})
315+
316+
it('leaves a cloud run that provisioned no sandbox uncharged', async () => {
317+
const output = (await handler.execute(ctx(), block, {
318+
mode: 'cloud',
319+
task: 'do it',
320+
model: 'claude',
321+
owner: 'o',
322+
repo: 'r',
323+
githubToken: 'ghp',
324+
})) as { cost: Record<string, unknown> }
325+
326+
expect(output.cost.toolCost).toBeUndefined()
327+
expect(output.cost.total).toBe(0)
328+
})
329+
294330
it('routes Create PR to the cloud backend and surfaces PR output', async () => {
295331
const output = (await handler.execute(ctx(), block, {
296332
mode: 'cloud',

‎apps/sim/executor/handlers/pi/pi-handler.ts‎

Lines changed: 31 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
import { createLogger } from '@sim/logger'
1010
import { projectResolvedModelInput } from '@/lib/execution/model-input-provenance'
11+
import type { SandboxCostSink } from '@/lib/execution/remote-sandbox/types'
1112
import type { BlockOutput } from '@/blocks/types'
1213
import { parseOptionalNumberInput } from '@/blocks/utils'
1314
import {
@@ -45,10 +46,7 @@ import {
4546
resolvePiSearchKey,
4647
} from '@/executor/handlers/pi/core/keys'
4748
import { runLocalPi } from '@/executor/handlers/pi/local/backend'
48-
import {
49-
buildSimToolSpecs,
50-
type PiFunctionToolCostAccumulator,
51-
} from '@/executor/handlers/pi/local/sim-tools'
49+
import { buildSimToolSpecs } from '@/executor/handlers/pi/local/sim-tools'
5250
import { buildPiSearchToolSpec } from '@/executor/handlers/pi/search/tool'
5351
import type {
5452
BlockHandler,
@@ -270,8 +268,8 @@ export class PiBlockHandler implements BlockHandler {
270268
}
271269
const usePrivateKey = inputs.authMethod === 'privateKey'
272270
const port = parseOptionalNumberInput(inputs.port, 'port', { integer: true, min: 1 }) ?? 22
273-
const functionToolCost: PiFunctionToolCostAccumulator = { total: 0 }
274-
const tools = await buildSimToolSpecs(ctx, inputs.tools, functionToolCost)
271+
const sandboxCost: SandboxCostSink = { total: 0 }
272+
const tools = await buildSimToolSpecs(ctx, inputs.tools, sandboxCost)
275273
const params: PiLocalRunParams = {
276274
...contextualBase,
277275
mode: 'local',
@@ -286,7 +284,7 @@ export class PiBlockHandler implements BlockHandler {
286284
passphrase: usePrivateKey ? asRawString(inputs.passphrase) : undefined,
287285
},
288286
}
289-
return this.runPi(ctx, block, runLocalPi, params, memoryConfig, functionToolCost)
287+
return this.runPi(ctx, block, runLocalPi, params, memoryConfig, sandboxCost)
290288
}
291289

292290
const owner = asOptString(inputs.owner)
@@ -478,17 +476,23 @@ export class PiBlockHandler implements BlockHandler {
478476
isBYOK: boolean,
479477
startTime: number,
480478
startTimeISO: string,
481-
functionToolCost = 0
479+
sandboxCost = 0
482480
): NormalizedBlockOutput {
483481
const { totals } = result
484482
const endTime = Date.now()
485483
const modelCost = computePiCost(model, totals.inputTokens, totals.outputTokens, isBYOK)
484+
/*
485+
* Sandbox compute rides in `toolCost` so it survives a BYOK run: the model
486+
* side is zero by definition there, and the ledger bills a model row on
487+
* `total > 0`. Folding it in is what makes a BYOK Pi session bill for the
488+
* E2B time it actually consumed instead of nothing at all.
489+
*/
486490
const cost =
487-
functionToolCost > 0
491+
sandboxCost > 0
488492
? {
489493
...modelCost,
490-
toolCost: functionToolCost,
491-
total: modelCost.total + functionToolCost,
494+
toolCost: sandboxCost,
495+
total: modelCost.total + sandboxCost,
492496
}
493497
: modelCost
494498
return {
@@ -534,7 +538,14 @@ export class PiBlockHandler implements BlockHandler {
534538
backend: PiBackendRun<P>,
535539
params: P,
536540
memoryConfig?: PiMemoryConfig,
537-
functionToolCost?: PiFunctionToolCostAccumulator
541+
/**
542+
* One sink for every Sim-paid sandbox this block touches. Local mode fills it
543+
* from the Function tools it runs host-side; cloud modes fill it from the
544+
* sandbox the agent itself runs in. They are mutually exclusive in practice,
545+
* and sharing one total means neither can be forgotten at the point the cost
546+
* is folded into the block's output.
547+
*/
548+
sandboxCost: SandboxCostSink = { total: 0 }
538549
): Promise<BlockOutput | StreamingExecution> {
539550
const startTime = Date.now()
540551
const startTimeISO = new Date(startTime).toISOString()
@@ -560,6 +571,7 @@ export class PiBlockHandler implements BlockHandler {
560571
if (text) controller.enqueue(encoder.encode(text))
561572
},
562573
signal: ctx.abortSignal,
574+
sandboxCost,
563575
})
564576
if (result.totals.errorMessage) {
565577
controller.error(new Error(result.totals.errorMessage))
@@ -577,7 +589,7 @@ export class PiBlockHandler implements BlockHandler {
577589
params.isBYOK,
578590
startTime,
579591
startTimeISO,
580-
functionToolCost?.total
592+
sandboxCost.total
581593
)
582594
)
583595
if (memoryConfig) {
@@ -608,7 +620,11 @@ export class PiBlockHandler implements BlockHandler {
608620
}
609621
}
610622

611-
const result = await backend(params, { onEvent: () => {}, signal: ctx.abortSignal })
623+
const result = await backend(params, {
624+
onEvent: () => {},
625+
signal: ctx.abortSignal,
626+
sandboxCost,
627+
})
612628
if (result.totals.errorMessage) {
613629
throw new Error(result.totals.errorMessage)
614630
}
@@ -627,7 +643,7 @@ export class PiBlockHandler implements BlockHandler {
627643
params.isBYOK,
628644
startTime,
629645
startTimeISO,
630-
functionToolCost?.total
646+
sandboxCost.total
631647
)
632648
}
633649
}

‎apps/sim/lib/execution/remote-sandbox/index.ts‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ import type {
4545
SandboxCodeResult,
4646
SandboxCollectedFile,
4747
SandboxCommandResult,
48+
SandboxCostSink,
4849
SandboxDirectoryEntry,
4950
SandboxExecutionCost,
5051
SandboxExecutionRequest,
@@ -59,6 +60,7 @@ import type {
5960
} from '@/lib/execution/remote-sandbox/types'
6061

6162
export type {
63+
SandboxCostSink,
6264
SandboxExecutionRequest,
6365
SandboxExecutionResult,
6466
SandboxFile,
@@ -1117,12 +1119,13 @@ export interface PiSandboxRunner {
11171119
* caller's sandbox body, which would have buried the change in whitespace.
11181120
*/
11191121
export async function withPiSandbox<T>(
1120-
options: { lifetimeMs?: number },
1122+
options: { lifetimeMs?: number; cost?: SandboxCostSink },
11211123
fn: (runner: PiSandboxRunner) => Promise<T>
11221124
): Promise<T> {
11231125
const lifetimeMs =
11241126
options.lifetimeMs !== undefined ? options.lifetimeMs : resolvePiSandboxLifetimeMs()
1125-
const { sandbox } = await createSandbox('pi', { lifetimeMs })
1127+
const created = await createSandbox('pi', { lifetimeMs }, Boolean(options.cost))
1128+
const { sandbox } = created
11261129
logger.info('Started Pi sandbox', { sandboxId: sandbox.sandboxId, lifetimeMs })
11271130

11281131
const runner: PiSandboxRunner = {
@@ -1142,6 +1145,18 @@ export async function withPiSandbox<T>(
11421145
try {
11431146
return await fn(runner)
11441147
} finally {
1148+
/*
1149+
* Charged on creation rather than on a successful session, which is where
1150+
* this departs from the Function path — and deliberately. A Function run is
1151+
* seconds long, so absorbing one that the provider failed to deliver costs
1152+
* little and reads as fair. A Pi session holds its sandbox for tens of
1153+
* minutes, and that compute is consumed whether the agent finished, errored,
1154+
* or was cancelled. Billing only the clean endings would mean paying for
1155+
* every other one. A create that throws never reaches here, so the one case
1156+
* where nothing was provisioned is still free.
1157+
*/
1158+
const cost = calculateSandboxCost(created, Date.now())
1159+
if (cost && options.cost) options.cost.total += cost.total
11451160
try {
11461161
await sandbox.kill()
11471162
} catch {

‎apps/sim/lib/execution/remote-sandbox/types.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,18 @@ export interface SandboxExecutionCost {
109109
total: number
110110
}
111111

112+
/**
113+
* Running total a caller accumulates sandbox charges into.
114+
*
115+
* A long-lived sandbox reports its cost when it is torn down, which is after the
116+
* value its caller cares about has already been returned. Handing the layer a
117+
* sink lets the charge land without reshaping every return type between here and
118+
* the block that owns the bill.
119+
*/
120+
export interface SandboxCostSink {
121+
total: number
122+
}
123+
112124
export interface SandboxExecutionResult {
113125
result: unknown
114126
stdout: string

0 commit comments

Comments
 (0)