Skip to content

Commit 933a4bc

Browse files
Vikhyath Mondreticlaude
andcommitted
test(billing): check the metered sandbox amount against a real provider
The pricing unit test pins the arithmetic and the conformance suite proves a cost is produced, attached to the right outcomes, and routed — but that suite stubs the provider and mocks Date.now() with a counter advancing one millisecond per call. Under that clock `total > 0` is the strongest claim available, and it holds equally well if the metered window is anchored to the wrong instants or the resource constants are wrong. Bounds the charge between what the sleep must cost and what the wall clock could justify, so a wrong rate, a wrong vCPU/memory constant, and a mis-anchored window all fail. Provider-agnostic via resolveProvider, opt-in behind SANDBOX_BILLING_SMOKE=1 like the sibling smoke suites. Verified against both providers: E2B billed 9.031s of a 9.302s call at $0.1656/hr, Daytona 8.435s of 8.721s at $0.16668/hr — both matching published rates, both excluding ~275ms of Sim-side overhead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent dfc0e35 commit 933a4bc

1 file changed

Lines changed: 78 additions & 0 deletions

File tree

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
/**
2+
* @vitest-environment node
3+
*
4+
* Checks the metered amount against a real provider run.
5+
*
6+
* `sandbox-pricing.test.ts` pins the arithmetic and the conformance suite proves
7+
* a cost is produced, attached, and routed — but that suite stubs the provider
8+
* and mocks `Date.now()`, so its clock advances one millisecond per call. Under
9+
* those conditions `total > 0` is the strongest claim available, and it would
10+
* hold just as well if the metered window measured the wrong instants. Only a
11+
* real run can show that the window tracks the sandbox's actual lifetime.
12+
*
13+
* Enable with `SANDBOX_BILLING_SMOKE=1`. Runs against whichever provider
14+
* `SANDBOX_PROVIDER` selects, so point it at each in turn to cover both.
15+
*/
16+
import { describe, expect, it } from 'vitest'
17+
import { createSandboxPricing } from '@/lib/billing/sandbox-pricing'
18+
import { CodeLanguage } from '@/lib/execution/languages'
19+
import { executeInSandbox } from '@/lib/execution/remote-sandbox'
20+
import { resolveProvider } from '@/lib/execution/remote-sandbox/provider'
21+
22+
const smokeEnabled = process.env.SANDBOX_BILLING_SMOKE === '1'
23+
const CASE_TIMEOUT_MS = 5 * 60_000
24+
const RUN_TIMEOUT_MS = 4 * 60_000
25+
26+
/** Long enough that provisioning jitter cannot dominate the measured runtime. */
27+
const SLEEP_SECONDS = 5
28+
29+
describe.skipIf(!smokeEnabled)('sandbox billing smoke', () => {
30+
it(
31+
'bills the sandbox lifetime at the provider rate',
32+
async () => {
33+
const pricing = createSandboxPricing(resolveProvider().id)
34+
const usdPerSecond =
35+
pricing.resources.vcpu * pricing.rates.cpuUsdPerVcpuSecond +
36+
pricing.resources.memoryGiB * pricing.rates.memoryUsdPerGiBSecond +
37+
pricing.resources.diskGiB * pricing.rates.diskUsdPerGiBSecond
38+
const usdPerBilledSecond = usdPerSecond * pricing.multiplier
39+
40+
const wallClockStartedAtMs = Date.now()
41+
const result = await executeInSandbox({
42+
code: `import time\ntime.sleep(${SLEEP_SECONDS})\nprint("slept")`,
43+
language: CodeLanguage.Python,
44+
timeoutMs: RUN_TIMEOUT_MS,
45+
meterUsage: true,
46+
})
47+
const wallClockMs = Date.now() - wallClockStartedAtMs
48+
49+
expect(result.cost).toEqual({ input: 0, output: 0, total: expect.any(Number) })
50+
const billed = result.cost?.total ?? 0
51+
52+
/**
53+
* The window opens immediately before the provider create call and closes
54+
* before teardown, so it has to cover the sleep and cannot exceed the whole
55+
* call measured from out here. A rate error, a wrong resource constant, or a
56+
* window anchored to the wrong instant all land outside these bounds — which
57+
* an `expect.any(Number)` assertion cannot see.
58+
*/
59+
expect(billed).toBeGreaterThanOrEqual(SLEEP_SECONDS * usdPerBilledSecond)
60+
expect(billed).toBeLessThanOrEqual((wallClockMs / 1000) * usdPerBilledSecond)
61+
},
62+
CASE_TIMEOUT_MS
63+
)
64+
65+
it(
66+
'bills nothing when the caller did not ask for metering',
67+
async () => {
68+
const result = await executeInSandbox({
69+
code: 'print("unmetered")',
70+
language: CodeLanguage.Python,
71+
timeoutMs: RUN_TIMEOUT_MS,
72+
})
73+
74+
expect(result.cost).toBeUndefined()
75+
},
76+
CASE_TIMEOUT_MS
77+
)
78+
})

0 commit comments

Comments
 (0)