Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 72 additions & 17 deletions docs/modules/worker-loader.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ description: Load and run dynamic Workers at runtime with the worker_loaders bin
<BindingStatus
binding="worker-loader"
works="`get()`, `load()`, `fetch()`, and RPC, with `env` and `props` capabilities and `globalOutbound`"
differs="Each dynamic Worker runs in its own agentOS VM. A load usually takes a prewarmed VM; when none is ready, it waits about 0.3 seconds for one to boot. Compatibility settings and `limits` are ignored."
differs="Each dynamic Worker runs in its own agentOS VM. A load usually takes a prewarmed VM; when none is ready, it boots one, which takes about 0.3 seconds. Compatibility settings and `limits` are ignored."
missing="`getDurableObjectClass()`, `connect()` to a dynamic Worker, Python modules, and tail events from dynamic Workers"
/>

Expand Down Expand Up @@ -89,8 +89,9 @@ Hello from a dynamic Worker

- Dynamic Worker code is untrusted, so each dynamic Worker gets its own VM,
as it gets its own isolate on Cloudflare. It never runs on the host.
- Orange keeps a few VMs booted ahead of use, so a new dynamic Worker only
writes its own modules. Each VM runs one dynamic Worker and is never reused.
- Orange keeps VMs booted ahead of use, and more after a burst of loads, so a
new dynamic Worker only writes its own modules. Each VM runs one dynamic
Worker and is never reused.
- A dynamic Worker stops when it is idle or nothing references it. Its VM is
disposed, never returned to the pool.
- Your Worker and the dynamic Worker talk Workers RPC, so stubs and bindings
Expand Down Expand Up @@ -125,13 +126,44 @@ agentOS's limits apply:
| One module file | 16 MiB | The load fails with a `RangeError` that names the limit. Split the module. |
| All files in the VM | 64 MiB by default | agentOS's limit. Orange adds no check. |
| `env` | No set limit. In a test with agentOS 0.4.1, 5 MiB worked. | At 10 MiB, the dynamic Worker exited with code 1. |
| Running dynamic Workers, across all Worker Loader bindings | 32. Set `ORANGE_WORKER_LOADER_MAX_WORKERS` to change it. | A new dynamic Worker stops the least recently used idle one. If none is idle, `get()` or `load()` throws a `WorkerLoaderLimitError` that names the limit. At 80% of the limit, `orange dev` logs a warning. |
| Running dynamic Workers, across all Worker Loader bindings | 32. Set `ORANGE_WORKER_LOADER_MAX_WORKERS` to change it. | A new dynamic Worker stops the least recently used idle one. If none is idle, it waits in line for a slot, first come first served, and its calls wait with it. At 80% of the limit, Orange logs a warning. |
| Time a dynamic Worker waits for a slot | 5 seconds. Set `ORANGE_WORKER_LOADER_QUEUE_TIMEOUT_MS` to change it, from 1 to 2147483647. | Its calls reject with a `WorkerLoaderLimitError` whose `code` is `queue_timeout`. |
| Dynamic Workers that wait for a slot | 256. Set `ORANGE_WORKER_LOADER_MAX_QUEUED` to change it; `0` refuses at once. | Its calls reject with a `WorkerLoaderLimitError` whose `code` is `queue_full`. At 80% of the limit, Orange logs a warning. |
| agentOS VMs, counting running Workers, prewarmed VMs, and VMs that boot or stop | `ORANGE_WORKER_LOADER_MAX_WORKERS` plus `ORANGE_WORKER_LOADER_POOL_MAX`, 48 by default. Set `ORANGE_WORKER_LOADER_MAX_VMS` to change it; it must not be less than `ORANGE_WORKER_LOADER_MAX_WORKERS`. | Prewarmed VMs stop first, and the pool stops refilling. A Worker that needs a new VM waits for one to stop. At 80% of the budget, Orange logs a warning. |
| Idle time before a dynamic Worker stops | 30 seconds. Set `ORANGE_WORKER_LOADER_IDLE_TIMEOUT_MS` to change it, from 1 to 2147483647. | The dynamic Worker stops and frees its VM. The next call loads it again. A value out of range stops `orange dev` at startup with a `RangeError` that names the setting. |
| Prewarmed VMs | 2 for each network mode by default, from 0 to 16 with `ORANGE_WORKER_LOADER_POOL_SIZE` | A load that finds the pool empty boots its own VM. |

Each running dynamic Worker uses about 54 MiB. Prewarmed VMs that no Worker
has claimed do not count toward `ORANGE_WORKER_LOADER_MAX_WORKERS`, and the
limit never stops them.
| Prewarmed VMs when no burst is recent | 2 for each network mode by default, from 0 to 64 with `ORANGE_WORKER_LOADER_POOL_SIZE` | A load that finds the pool empty boots its own VM. |
| Prewarmed VMs after a burst | 16 for each network mode by default, from 0 to 64 with `ORANGE_WORKER_LOADER_POOL_MAX`. It must not be less than `ORANGE_WORKER_LOADER_POOL_SIZE`. `0` boots every VM on demand. | The pool stops growing. A value out of range stops `orange dev` at startup with a `RangeError` that names the setting. |
| VMs that prewarm at once | 4, across both network modes, while no Worker loads. While Workers load, only as many as keep up with a steady rate of loads. | The other prewarms wait. Loads never wait for this limit. |

Each running dynamic Worker uses about 54 MiB. Each prewarmed VM uses 26 to
54 MiB, depending on the agentOS version. After a burst of 16 or more loads,
a network mode can hold 16 prewarmed VMs, 0.4 to 0.9 GiB, until the idle
timeout passes. Prewarmed VMs do not count toward
`ORANGE_WORKER_LOADER_MAX_WORKERS`, but they count toward
`ORANGE_WORKER_LOADER_MAX_VMS`, which bounds the memory of all VMs.

##### Refusals

- A `WorkerLoaderLimitError` is a `RangeError` with `code` (`queue_full` or
`queue_timeout`), `overloaded: true`, `retryable: true`, and
`retryAfterSeconds: 1`, like the overloaded error of a Cloudflare Durable
Object. `get()` and `load()` never throw it; the Worker's calls reject with it.
- If your Worker does not catch it, the response is `503 Service Unavailable`
with `Retry-After: 1`, not 500. A WebSocket upgrade gets the same 503. On
Cloudflare, an uncaught exception is a 500 (error 1101). Cloudflare has no
per-runner Worker limit, so it has no such refusal.
- A held stub whose Worker was refused asks for a slot again on its next call,
and `get()` with the same name calls `getCode` again.
- Under load that stays above what the runner can start, the line stays full:
a Worker that gets a slot has waited close to the queue timeout, and the
rest are refused. Lower `ORANGE_WORKER_LOADER_QUEUE_TIMEOUT_MS` to trade
refusals for latency, and scale out on the queue wait (see
[Monitoring](../operations/monitoring#autoscaling)).
- agentOS bounds the requests in flight to its shared sidecar at 128, and
each running Worker keeps some open. In a test that started 80 Workers at
once with `ORANGE_WORKER_LOADER_MAX_WORKERS` at 96, 33 to 38 started; the
rest failed with agentOS's `ERR_AGENTOS_IN_FLIGHT_REQUEST_LIMIT`. When a
running Worker's process fails this way, its call fails with a 500.

#### Isolation

Expand All @@ -149,12 +181,32 @@ limit never stops them.
Worker loads, so an app without dynamic Workers boots none. A prewarmed VM
has its Node.js process started with Orange's runtime imported. It has seen
no Worker code or `env`.
- With an agentOS client that supports `snapshotEntrypoints`, each Node.js
process boots from a V8 snapshot of Orange's runtime, built once per agentOS
sidecar, instead of importing it. Each process gets its own copy of the
runtime's state. agentOS 0.4.8 lacks the option, so there each process
imports the runtime.
- A load takes a prewarmed VM, writes its modules, and starts the Worker. If
the pool is empty, it waits for the VM that is booting, or boots its own.
The pool refills one VM at a time, shortly after the load finishes.
the pool is empty, it takes a VM that is booting and that no other load
waits for, or boots its own at once, beside the other loads.
- Each network mode's pool aims for the most loads seen within one prewarm
time, from `ORANGE_WORKER_LOADER_POOL_SIZE` to
`ORANGE_WORKER_LOADER_POOL_MAX`. So the next burst of the same size finds
its VMs ready, and a steady stream of loads keeps the pool ahead.
- The pool refills at full speed as soon as no dynamic Worker is loading,
because a prewarm that runs beside a load slows it. Workers that load at the
same time also hold the refill until their first calls finish. One Worker
holds the refill back for at most 1 second.
- While Workers keep loading, the pool still refills, with as many prewarms
as the rate of loads needs. Only loads that keep arriving count, so a burst
does not refill beside its own loads.
- When the idle timeout (`ORANGE_WORKER_LOADER_IDLE_TIMEOUT_MS`) passes
without a burst as large, the pool shrinks to
`ORANGE_WORKER_LOADER_POOL_SIZE` and disposes the other prewarmed VMs.
- A VM that fails to prewarm is reported as a background error, and the next
load boots its own VM. Set `ORANGE_WORKER_LOADER_POOL_SIZE` to `0` to boot
every VM on demand.
load boots its own VM. The pool tries to prewarm again only after its next
load. Set `ORANGE_WORKER_LOADER_POOL_MAX` to `0` to boot every VM on
demand.
- If the VM cannot boot, `get()` and `load()` calls fail with the boot error.
Orange never falls back to the host.
- A dynamic Worker is idle when no call to it is in flight, your Worker holds
Expand Down Expand Up @@ -250,10 +302,13 @@ These add to the [Compatibility](#compatibility) table, the
`tests/bindings/worker-loader.test.ts` drives a real `orange dev` over HTTP:
fetch and RPC, env and props capabilities, isolation, `globalOutbound: null`,
name caching, syntax errors, process crashes, module types, a `node:http`
server and an Express app in a dynamic Worker, and two bursts of loads larger
than the VM pool, the second after the first burst's Workers stop idle, with
each Worker isolated from the others and from the read-only runtime mount. A
second `orange dev` with a limit of 2 and a 1.5-second idle timeout checks that
server and an Express app in a dynamic Worker, and bursts of loads larger than
the VM pool whose Workers each wait in a call for all the others. The VM pool
grows to the last burst and shrinks to its idle size after the idle timeout,
and each Worker gets a fresh VM, isolated from the others and from the
read-only runtime mount, with its own runtime module state and randomness
when its process boots from the agentOS snapshot. A second `orange dev` with
a limit of 2 and a 1.5-second idle timeout checks that
idle Workers stop and load again, that calls longer than the timeout finish,
that busy Workers fail a new load with `WorkerLoaderLimitError`, and that more
`load()` calls than the limit succeed.
Expand Down
64 changes: 63 additions & 1 deletion docs/operations/monitoring.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,12 @@ $ docker run \
-e RIVET_ENDPOINT=... my-worker
```

## Autoscale dynamic Workers

Scale runners on `orange_worker_loader_slot_utilization` and the slot queue
wait, not on CPU. See [Dynamic Worker metrics](#dynamic-worker-metrics) for an
example.

## Learn more

- [OpenTelemetry](https://rivet.dev/docs/tracing/): span names, sampling, and application spans
Expand Down Expand Up @@ -95,7 +101,7 @@ RivetKit serves Prometheus metrics at `http://127.0.0.1:$RIVETKIT_METRICS_PORT/m
them in the Rivet dashboard. `orange deploy` refuses to set it with `--var`.
- **Self-hosted:** the endpoint listens on `127.0.0.1` only, so `docker run -p` does not expose
it. Scrape it from the same network namespace, such as a sidecar in the same Kubernetes pod.
- `orange dev` serves no metrics.
- `orange dev` serves metrics when `RIVETKIT_METRICS_PORT` is set, from its first Worker.
- With several Workers, only the first Worker's runtime serves metrics, so they cover that
Worker's objects only.

Expand All @@ -105,6 +111,62 @@ RivetKit serves Prometheus metrics at `http://127.0.0.1:$RIVETKIT_METRICS_PORT/m
| `rivet_rivetkit_actor_http_requests_active` | Requests and RPC calls in progress |
| `rivet_rivetkit_actor_inbox_depth` | Work waiting to run |

#### Dynamic Worker metrics

A Worker with a `worker_loaders` binding adds these to the same endpoint.

| Metric | Type | What it shows |
|---|---|---|
| `orange_worker_loader_workers` | gauge | Dynamic Workers that hold a slot |
| `orange_worker_loader_workers_max` | gauge | `ORANGE_WORKER_LOADER_MAX_WORKERS` |
| `orange_worker_loader_slot_utilization` | gauge | Slots in use divided by slots, from 0 to 1 |
| `orange_worker_loader_queue_depth` | gauge | Dynamic Workers that wait for a slot |
| `orange_worker_loader_queue_wait_seconds` | histogram | How long each dynamic Worker waited for a slot |
| `orange_worker_loader_start_seconds` | histogram | From a slot to a loaded dynamic Worker |
| `orange_worker_loader_vm_boot_seconds` | histogram | How long one agentOS VM takes to boot |
| `orange_worker_loader_pool_claims_total` | counter | VM claims by `mode` (`network`, `deny`) and `result`: `hit` (prewarmed), `warming` (waited for a prewarm), `miss` (booted on demand) |
| `orange_worker_loader_pool_ready` | gauge | Prewarmed VMs by `mode` |
| `orange_worker_loader_vms` | gauge | agentOS VMs by `state`: `live`, `pooled`, `booting`, `stopping` |
| `orange_worker_loader_vms_max` | gauge | `ORANGE_WORKER_LOADER_MAX_VMS` |
| `orange_worker_loader_vm_queue_depth` | gauge | Dynamic Workers with a slot that wait for room in the VM budget |
| `orange_worker_loader_rejections_total` | counter | `WorkerLoaderLimitError` refusals by `code`: `queue_full`, `queue_timeout` |
| `orange_runtime_resident_memory_bytes` | gauge | RSS by `process`: `runner`, and on Linux `children`, which includes the agentOS sidecar |
| `orange_runtime_memory_limit_bytes` | gauge | The runner's cgroup memory limit, when it has one |

Useful queries:

```promql
# Pool miss rate
sum(rate(orange_worker_loader_pool_claims_total{result="miss"}[5m]))
/ sum(rate(orange_worker_loader_pool_claims_total[5m]))
# Slot queue wait, p95
histogram_quantile(0.95, sum by (le) (rate(orange_worker_loader_queue_wait_seconds_bucket[5m])))
# Memory headroom
sum(orange_runtime_resident_memory_bytes) / orange_runtime_memory_limit_bytes
```

The agentOS sidecar's own request queue is not visible to Orange, so it has no metric.

#### Autoscaling

With a Kubernetes HorizontalPodAutoscaler and a Prometheus adapter:

- Scale on `orange_worker_loader_slot_utilization`, with a target of about
0.7, and on the p95 of `orange_worker_loader_queue_wait_seconds`. A busy
slot can mean a dynamic Worker that waits on I/O, so CPU does not show
dynamic Worker demand. Do not scale on CPU.
- Set `behavior.scaleDown.stabilizationWindowSeconds` longer than
`ORANGE_WORKER_LOADER_IDLE_TIMEOUT_MS` (30 s by default), such as 300 s. An
idle dynamic Worker holds its slot until the timeout, so a shorter window
scales down while slots still look busy, and then scales up again.
- Alert on `orange_worker_loader_rejections_total`. Each refusal is a 503 or
a `WorkerLoaderLimitError` your Worker caught.
- Size the pod's memory limit for `ORANGE_WORKER_LOADER_MAX_VMS` VMs, and
watch `orange_runtime_resident_memory_bytes` against
`orange_runtime_memory_limit_bytes`.
- Orange has no readiness endpoint, so a full runner still receives
requests. Its 503s carry `Retry-After`.

#### Logs

- `orange dev` prints each request and your Worker's `console` output.
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "open-orange",
"version": "1.0.3",
"version": "1.0.4-rc.2",
"description": "Cloudflare Workers compatibility layer for Rivet",
"packageManager": "pnpm@10.34.9",
"engines": {
Expand Down Expand Up @@ -50,7 +50,7 @@
"license": "Apache-2.0",
"dependencies": {
"@aws-sdk/client-s3": "3.1153.0",
"@rivet-dev/agentos-core": "0.4.8",
"@rivet-dev/agentos-core": "0.4.9-rc.1",
"@rivet-dev/durable-object-migrator": "1.0.3",
"@rivet-dev/sandbox-adapter": "0.5.3",
"capnweb": "0.12.0",
Expand Down
2 changes: 1 addition & 1 deletion packages/runtime-node/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
},
"dependencies": {
"@aws-sdk/client-s3": "3.1153.0",
"@rivet-dev/agentos-core": "0.4.8",
"@rivet-dev/agentos-core": "0.4.9-rc.1",
"@rivet-dev/durable-object-migrator": "1.0.3",
"@rivet-dev/sandbox-adapter": "0.5.3",
"capnweb": "0.12.0",
Expand Down
16 changes: 11 additions & 5 deletions packages/runtime-node/src/bindings/queues/workerd-wire.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,19 @@ const maxHeadersGuard = 3;

// The index is workerd's ErrorTag. Unknown (0) carries its own name.
// https://github.com/cloudflare/workerd/blob/b73ce3ae105c35bafaaad864095079524573a465/src/workerd/jsg/ser.c%2B%2B#L23
const webAssemblyError = (name: string): ErrorConstructor =>
Reflect.get(Reflect.get(globalThis, "WebAssembly") as object, name) as ErrorConstructor;
const errorClasses: readonly (ErrorConstructor | undefined)[] = [
// A string names a WebAssembly class, read on use: V8 has no WebAssembly
// global while agentOS builds a module snapshot of the dynamic Worker child.
const errorClasses: readonly (ErrorConstructor | string | undefined)[] = [
Error, Error, TypeError, RangeError, ReferenceError, SyntaxError,
webAssemblyError("CompileError"), webAssemblyError("LinkError"), webAssemblyError("RuntimeError"),
"CompileError", "LinkError", "RuntimeError",
undefined, EvalError, URIError, undefined, undefined,
];
const errorClass = (tag: number): ErrorConstructor | undefined => {
const entry = errorClasses[tag];
return typeof entry === "string"
? (Reflect.get(Reflect.get(globalThis, "WebAssembly") as object, entry) as ErrorConstructor)
: entry;
};
const errorNames: Readonly<Record<number, string>> = {
9: "SuspendError",
12: "AggregateError",
Expand Down Expand Up @@ -111,7 +117,7 @@ class WorkerdWireDeserializer extends Deserializer {
const name: unknown = errorTag === 0 ? this.readValue() : errorNames[errorTag];
const message: unknown = this.readValue();
if (typeof message !== "string") throw cloneError("error message is not a string");
const error = new (errorClasses[errorTag] ?? Error)(message);
const error = new (errorClass(errorTag) ?? Error)(message);
if (name !== undefined) Object.defineProperty(error, "name", { configurable: true, value: name, writable: true });
const properties: unknown = this.readValue();
if (isObject(properties)) {
Expand Down
4 changes: 2 additions & 2 deletions packages/runtime-node/src/bindings/unsafe-eval.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { compileFunction, runInThisContext } from "node:vm";

import { WebAssembly } from "../globals/webassembly.js";
import { nativeWebAssembly } from "../globals/webassembly.js";

function bytesFrom(source: unknown): Uint8Array {
if (source instanceof ArrayBuffer) return new Uint8Array(source);
Expand Down Expand Up @@ -48,7 +48,7 @@ export class UnsafeEval {
}

newWasmModule(source: unknown): unknown {
return new WebAssembly.Module(bytesFrom(source));
return new (nativeWebAssembly().Module)(bytesFrom(source));
}
}

Expand Down
Loading
Loading