feat: add experimental Code Mode tools (run-code + get-code-docs)

[MQ37]

The Actor is deployed https://apify.com/apify/code-runtime and lives in github.com/apify/actor-code-runtime/ (currently PR apify/actor-code-runtime#1)

What

Add two experimental tools. run-code runs a JS/TS script in a sandboxed apify/code-runtime Actor exposing an apify binding (search/run Actors, read/write datasets & key-value stores); it reuses the existing call-actor start→wait→respond pipeline and returns the run plus a nextStep pointing at get-dataset-items for the script's { stdout, stderr } output. get-code-docs serves a paginated guide for writing those scripts. Selecting either Code Mode tool loads both (loader pairing), and when present the run/storage helpers are auto-injected (same as call-actor).

Why

Multi-step Actor work (chain Actors, then filter/aggregate) otherwise costs many separate tool calls and round-trips every intermediate dataset through the model. One Code Mode script does it in a single run, keeping large intermediate data out of context. Implements #1017; the sandbox is the apify/code-runtime Actor (repo apify/actor-code-runtime), which runs with limited permissions (no filesystem, no imports, outbound network limited to *.apify.com).

Note on concurrency

The guide shows a plain sequential loop for running an Actor over many inputs and does not hint at parallel fan-out. This avoids nudging agents into spinning up many concurrent runs by default, which free-tier accounts can hit as concurrent-run/memory limits. It does not forbid parallelism either — the guide stays neutral.

Testing

• Unit tests: tools.run_code.test.ts, tools.get_code_docs.test.ts, loader-pairing cases in utils.tools_loader.test.ts; updated tools.mode_contract.test.ts and integration suite.ts (experimental tool count).
• Full gate green: type-check, lint (0/0), test:unit 1018 pass / 2 skip, format, check:agents.
• Backing Actor deployed and exercised end-to-end separately (18/18 binding smoke).

Configurator wiring (web tool selector): apify/apify-mcp-server-internal#628 — merges after the package bump that includes this PR.

[MQ37]

Configurator wiring (adds run-code + get-code-docs to the web tool selector): apify/apify-mcp-server-internal#628 — merges after the package bump that includes this PR.

[jirispilka]

I would love to test it, I'll try to test it on Friday

[vojtechj-apify]

We are running only TS/JS, what about Python? That is usually the type of language language models go for when writing a small script.

[vojtechj-apify]

Tested this branch locally — built from feat/code-mode-tools and driven via mcpc against dist/stdio.js with a real token. Summary of what we exercised and what we found.

What we tested

get-code-docs

• All three pages (overview, api, recipes) + the default page resolve correctly.
• Invalid page value and wrong type are both rejected cleanly (-32602 validation error).

Tool loading / pairing (tools_loader.ts)

• --tools=run-code alone → auto-injects get-code-docs + run/storage helpers.
• --tools=get-code-docs alone → auto-injects run-code + helpers.
• Both produce the identical 6-tool set: run-code, get-code-docs, get-actor-run, get-dataset-items, get-key-value-store-record, abort-actor-run.

run-code

• Minimal script → SUCCEEDED, { stdout, stderr } written to the run's dataset, nextStep points at get-dataset-items (followed it — contents correct).
• Input validation probes: waitSecs:60 → must be <= 45; empty code → must NOT have fewer than 1 characters; waitSecs:-5 → must be >= 0. No run started in any of these.
• Throwing script (throw new Error(...)) → run still SUCCEEDED, exitCode 0, error captured to stderr, pre-throw stdout preserved. Matches the documented contract.
• Long-run path: a script that outlives waitSecs returns RUNNING with a poll nextStep; get-actor-run polling → SUCCEEDED.

End-to-end chaining + "keep big data out of context" (the core motivation)
One script: scrape 40 Prague coffee shops via compass/crawler-google-places → rank / aggregate / pick rows in-sandbox → chain the winner's website into apify/rag-web-browser. Two chained Actor runs, all aggregation done server-side. The raw Maps dataset was ~182 KB (40 × 57 fields); only a ~1.2 KB summary came back through run-code. Worked exactly as intended.

Findings

1. run-code reports SUCCEEDED / exit-0 even when the script throws or a nested run fails to start — and there's no reliable way to detect it. This is the documented "throw → stderr → still SUCCEEDS" behavior, but for an agent-facing tool it means the primary signal (status: SUCCEEDED + nextStep: read the dataset) is decoupled from whether the task actually did anything. Concrete case we hit on our first real run: a nested run was rejected with 400 max-total-charge-usd-below-minimum → script threw → run-code returned status: SUCCEEDED, exitCode: 0, stdout: "", error only in stderr. And there's no clean signal to catch this programmatically: console.error / console.warn are a legitimate logging channel (our successful run also had a non-empty stderr), and the runtime collapses the throw into a SUCCEEDED / exit-0 run. A robust fix likely needs apify/code-runtime itself to surface the error distinctly (fail the run, non-zero exit code, or a structured { ok, error } field), not just a heuristic in run_code.ts. This is the one we'd want resolved before trusting the tool for autonomous multi-step use.

2. No effective USD cap on a run-code call. The tool is paymentRequired and can start arbitrary nested billable Actor runs from a script, but callOptions.maxTotalChargeUsd is ignored — apify/code-runtime is paid per platform usage, not pay-per-event — and it doesn't bound the nested runs either (they're independent runs that can outlive the code-runtime run's timeout). So there's no hard cost ceiling on an invocation. Combined with 1, a runaway / expensive script also fails silently.

3. Minor: $0.50 is the platform minimum for maxTotalChargeUsd, so a naive small cap on a nested pay-per-event run fails the whole run (that's what triggered the case in 1). A one-liner in the recipes page could save the surprise.

Overall: the feature works and the core value (server-side orchestration, big data kept out of context) is real. But two issues stand out and should be resolved before this is trusted for autonomous use: (1) run-code reports SUCCEEDED / exit-0 even when the script throws or a nested run fails — and there's no reliable signal to detect it, since console.error is a legitimate channel and the runtime erases the error; likely needs a fix in apify/code-runtime (fail the run, non-zero exit, or a structured error field), not just here. (2) no effective USD cap on a run-code call (code-runtime is pay-per-usage so maxTotalChargeUsd is ignored, and it doesn't bound nested runs).

[MQ37]

Thanks for the thorough testing. Verified findings 2 and 3 against apify-core:

Finding 2 — confirmed. maxTotalChargeUsd is only validated/enforced for pay-per-event Actors (checkPaidActorEffectiveOptions / getPaidActorEffectiveOptions in run_limits_helper.ts — for non-paid, platform-usage Actors it returns {} and the option is dropped). Since apify/code-runtime is platform-usage, the cap is silently ignored, and nested runs are independent with their own limits.

Rather than ship a knob that does nothing, we'll remove the cost cap from run-code for now and re-introduce it once the platform supports real caps for pay-per-usage Actors. Making code-runtime pay-per-event just to get a cap would mean bolting billing onto the Actor — more complication and hassle than it's worth for this.

Finding 3 — real gotcha, but it's not a fixed $0.50 platform floor. The minimum is per-Actor: currentPricingInfo.minimalMaxTotalChargeUsd, set by each PPE Actor's author in the monetization form (defaults to 0). The $0.50 was just the specific minimum of the Actor you nested; other PPE Actors differ. Since we're dropping the cap (finding 2) this stops being a run-code surprise, but I'll add a recipes note for nested runs anyway: each PPE Actor defines its own minimum for maxTotalChargeUsd; a cap below it fails with max-total-charge-usd-below-minimum.

Finding 1 — we want to keep the run as SUCCEEDED on a user-code throw. Failing the run would skew the Actor's success/failure statistics even though the runtime itself worked exactly as intended (the throw is a user-script-level event, not an infra failure). We already surface the error via stderr, but I agree that's not a clean programmatic signal — so we'll add an explicit exitCode field to the output dataset item alongside { stdout, stderr } so callers can reliably detect a failed script without heuristics on stderr.

[vojtechj-apify]

Thanks Jakub — the plan makes sense on all three. Ran a deeper test pass since; validation against your responses plus the extra coverage below.

On your responses

Finding 1 — the exitCode output field resolves it. Agreed that keeping the run SUCCEEDED on a user-code throw is correct (script-level event, not infra). One detail from the extended run so the field actually closes the loop: failure currently surfaces in two different places depending on the failure type —

• Compile / syntax error (e.g. const x = ;) → the run goes FAILED, exitCode 1, empty dataset, diagnostic nextStep. Detectable via run.status.
• Runtime throw / nested-run start failure → the run stays SUCCEEDED, run exitCode 0; the error only lands in stderr.

So for the new field to be a reliable signal it needs to be the user script's effective exit status (non-zero on a runtime throw even though the run is SUCCEEDED / exit-0), and callers will need to check both run.status (compile/infra failures) and the item's exitCode (runtime throws). If that's the intent, 👍.

Finding 2 — agree with removing the cap. Just flagging that removal leaves the code-runtime run's timeout as the only bound on an invocation (confirmed: a 30s script with timeout:5 → TIMED-OUT); nested runs stay independent/unbounded. Fine given the reasoning — no knob beats a knob that silently does nothing.

Finding 3 — thanks for the minimalMaxTotalChargeUsd correction. The per-Actor recipes note sounds right.

Extended test pass — full apify binding surface (all against the live Actor)

• dataset.*: create, pushItems (2500 rows), iterate (auto-paged all 2500), listItems field projection ✅
• kvs.*: create, set (JSON→object, text/plain→string), missing-key → null, list ✅
• run.*: start→get→wait→getLog ✅; abort → ABORTING ✅
• actor.*: search, getDetails (full input schema), run / runAndGetItems ✅
• Chaining: the get-code-docs "Chain Actors" recipe (google-search-scraper → website-content-crawler) run verbatim ✅
• Tool mechanics: waitSecs:0 fire-and-forget returns ~1s with READY+runId ✅; --task mode ✅; memory:64 → must be >= 128 ✅

Two minor new findings

• dataset.getSchema reports itemCount: 0 immediately after pushItems of 2500 rows (the fields sample is correct, and iterate sees all 2500). Eventual-consistency lag on the count — misleading if a script branches on itemCount.
• get-code-docs requires a token to start. It's fully static (no token, no network), but it isn't in unauthEnabledTools, and the pairing logic injects run-code — so a docs-only, tokenless server won't start. Minor, but the static guide could arguably be readable unauthenticated.

Separately, I have a couple of sandbox-hardening notes for the apify/code-runtime Actor itself (its outbound-network model and what the run environment exposes to user code) that I'll pass to that Actor's owners directly rather than in this thread — not blockers for this PR's wiring.

[vojtechj-apify]

@MQ37 Here are the sandbox-hardening notes for apify/code-runtime. I kept them in private Github issue.

[MQ37]

@vojtechj-apify thanks again for the thorough test passes — all the actionable findings are now resolved. Summary of what landed:

In this PR (feat/code-mode-tools)

Finding 2 — no effective USD cap (confirmed, cap removed). maxTotalChargeUsd/maxItems are only enforced for pay-per-event/result Actors; code-runtime is platform-usage, so they were silently dropped. Rather than ship a knob that does nothing, I narrowed run-code's callOptions to just memory + timeout (the options that actually apply). As you noted, the run's timeout remains the bound on an invocation; nested runs stay independent. We'll reintroduce a real cap once the platform supports one for pay-per-usage Actors.

Finding 1 / exitCode signal — resolved via a dedicated output field. The output dataset item is now { stdout, stderr, exitCode }. exitCode is the user script's effective status (0 = returned normally, 1 = threw), distinct from the Actor run's status — the run still ends SUCCEEDED on a user-script throw. So the reliable failed-script signal is exactly the two you described:

• run.status → compile/infra failures (the run goes FAILED), and
• the item's exitCode → runtime throws (SUCCEEDED run, exitCode: 1).

stderr is no longer needed as a failure heuristic (it's still a legitimate console.error/console.warn channel). The tool description and the Code Mode guide (get-code-docs) document this.

Finding 3 — per-Actor minimalMaxTotalChargeUsd. Moot for run-code now that the cap is gone.

Minor: get-code-docs required a token to start — fixed. It's now in unauthEnabledTools, so a docs-only tokenless server starts. run-code (paired with it by the loader) still needs a token, and the token gate keys off the explicitly requested tools, so requesting get-code-docs alone stays unauth. Added auth tests + README note.

Minor: dataset.getSchema itemCount: 0 right after pushItems. Platform eventual-consistency on the count; the fields sample and iterate are correct. Runtime-side, left as-is (documented behavior).

In the backing Actor (apify/actor-code-runtime)

Your sandbox-hardening notes (apify/ai-team#216) — fixed in apify/actor-code-runtime#1. Removing workerd's nodejs_compat was the root fix for all three findings:

• A — node:net (the raw-socket egress path that bypassed the *.apify.com fetch allowlist) is gone.
• B — the run token is no longer readable: process/require are undefined.
• C — no imports at all now (neither npm nor node:*), so the docs are accurate.

No regression — the apify binding is all fetch-based (binding smoke 18/18); a new isolation probe asserts the boundary (node blocked, process undefined, allowlist admits apify.com + subdomains and blocks lookalikes/metadata IP — 15/15). Full remediation write-up is in the issue.

[vojtechj-apify]

Thanks for the fix!

[jirispilka]

I'm really sorry this review took a while. This is an important feature and I wanted to do it properly rather than yolo an approval :).

And I'm really sorry for this long text 🤦🏻 but I wanted to make it clear.

I had pretty long session with Claude, putting in together, thoughts my mine but I asked Claude to wrap it up.

I'm glad I did: a few things came out of it that I think should hold the release, not because the code is bad, but because we're about to ship something we can't yet judge.

And let me be blunt, in a friendly way: the PR as it stands didn't really help me judge whether Code Mode is worth it. No examples, no metrics, no evals, and no follow-up plan for how we'll measure success once it's out. For a feature whose entire pitch is "this is cheaper and faster than manual tool calls," that's the one thing I most needed to see — so I went and measured it myself (below).

The good part up front: the code itself is genuinely fine. Running untrusted code in a separate sandboxed Actor instead of in-process, reusing the call-actor pipeline, keeping the bulky data in the sandbox — all correct. The binding is justified. And the earlier round with Vojta already resolved the big runtime issues (the exitCode signal, dropping the dead cost cap, the sandbox hardening). Thank you for that work.

My main point isn't the code. It's that we reached for two dedicated tools where one Actor would be simpler, and I think we missed the chance to make this smaller instead of bigger. Let me make that case concretely.

---

1. I measured it: 5 tasks, manual vs Code Mode — manual won 3–2

It is also possible that I picked up wrong examples but I had to invent it because I did not have anything better.

Full detail in the artifact: https://claude.ai/code/artifact/8052d747-6c68-4cba-848a-e25b8291316b?via=auto_preview

Metric	Manual	Code Mode
Wins (of 5)	3	2
Total tokens	373,331	414,477
Total time	20.8 min	30.2 min

What I tested

I picked five tasks that scale from trivial to hard, to see where Code Mode's advantage kicks in:

1. Top-rated restaurants — 1 actor (compass/crawler-google-places). Search "Italian restaurants," return top 5 by rating (min 50 reviews), sorted by rating then review count. Basic filter+sort on one dataset — baseline, small expected edge since there's nothing to chain. → Manual (57.9k tok / 81s vs 65.8k / 224.6s).
2. Tech YouTuber lookup — 2 actors, sequential (apify/rag-web-browser → streamers/youtube-scraper). Web-search "top tech YouTube channels 2026," extract top 3 channel names, then look up each channel's subscribers and latest views. Parse-then-lookup dependency; one actor's output seeds the
   next. → Manual (67.2k / 127.1s vs 102.6k / 583.7s; Code Mode needed 4 attempts).
3. Restaurant review sentiment — 2 actors, fan-out (compass/crawler-google-places → compass/Google-Maps-Reviews-Scraper). Top 5 restaurants, then 20 recent reviews each (5 parallel Actor calls), compute rating trend + top keywords locally. Real fan-out + bulk local text
   aggregation. → Code Mode (66.0k / 151.1s vs 93.3k / 311.6s).
4. Local business lead-gen — 3 actors, 2-hop chain (compass/crawler-google-places → renzomacar/website-contact-finder → harvestapi/linkedin-profile-search). 5 boutique gyms → crawl each site for emails → find the owner on LinkedIn → join into one lead per business. Join-by-key
   across differently-shaped datasets. → Split: Code Mode won every efficiency number but picked worse search terms (duplicate chain gyms + park facilities instead of real boutique studios) — cheaper/faster, not better.
5. Competitive product intel — 3 actors, fan-out + join (apify/rag-web-browser → junglee/Amazon-crawler → scraper_one/x-posts-search). Web-search "best wireless earbuds 2026," extract top 3, get Amazon price/rating/reviews for each, search X for buzz, compute a composite
   score and rank. Entity extraction + two fan-out branches per product + heterogeneous join + derived metric. → Manual (69.7k / 290.2s vs 99.6k / 543.2s; Code Mode needed 2 attempts, one feeding garbage like "1. Home, 2. Headphones" into a paid Amazon search).

The pattern (matters more than the scoreline)

• Code Mode's one clean win (Exp 3) was pure structured-data work: real parallel fan-out (5 Actor calls via Promise.all, launched within ~15ms of each other) and 100 raw reviews reduced to a summary before any model saw them. There it's genuinely better — cheaper and faster.
• Every loss involved pulling a fact out of unstructured text — a ranked list in an article, a channel name, a product name — with a regex and no judgment. A wrong guess fails silently until the whole script finishes, and the fix is a full container retry.
• Manual is slower, but the model reads the messy text and course-corrects on the spot — which is
  exactly what models are good at. Code Mode's core trade (skip the model mid-script to save tokens)
  backfires precisely when the mid-script step needed the model's judgment.

So the ask that follows from the data: the description opens with "USE THIS FIRST FOR ANY MULTI-STEP TASK," and the numbers say the opposite — it's a narrow win, not a first resort, and that line will push agents straight into the free-text cases where it's most fragile. I'd flip the framing: name the sweet spot (genuine fan-out + reducing bulk structured data), and say plainly when
not to reach for it (any step that needs reading or understanding free text — keep the model in the loop with individual calls). Same tool, honest steering.

---

2. The case for one Actor instead of two dedicated tools

I want to argue this on verified facts. I checked each of these by running both variants side by side through mcpc.

The proposal: drop the internal run-code tool and the get-code-docs tool, and expose apify/code-runtime as a curated direct Actor tool — the same mechanism we already use for apify/rag-web-browser(whitelisted input + an extra description line). Docs move into the Actor and come back throughfetch-actor-details`.

2.1 — It deletes ~380 lines of hand-written, security-adjacent code, replaced by a few lines of config. run_code.ts (144 lines) + code_docs_content.ts (165) + get_code_docs.ts (72) = 381 source lines, plus ~200 lines of bespoke tests (tools.run_code.test.ts 127, tools.get_code_docs.test.ts
55, loader-pairing cases ~20). rag-web-browser's whole curation is a whitelist array and a description constant.

2.2 — The core behavior is identical. I loaded both run-code and apify--code-runtime in one mcpc session and compared: happy path, waitSecs, waitSecs:0 fire-and-forget, --task mode, error path, and the get-dataset-items follow-up are all byte-for-byte the same envelope. Helper auto-injection (get-actor-run, get-dataset-items, get-key-value-store-record, abort-actor-run)
triggers for both.

2.3 — It fixes two real bugs for free — bugs that otherwise need manual patching in run_code.ts:

• Progress. The Actor dispatch path creates a progress tracker unconditionally (server.ts:1218),
  gated only by whether the client asked for progress. The internal run-code path hardcodes it to null (server.ts:1092-1093 only opts in call-actor and get-actor-run), so a blocking wait of up to 45s is silent today.
• Telemetry. Actor tools carry actorId/actorName into telemetry on every call (server.ts:921-925; extractActorId covers TOOL_TYPE.ACTOR). run-code sets neither — so today every code-mode call is logged with no actor id, unlike call-actor (call_actor.ts:658,677,683).

2.4 — No whitelist gymnastics needed. apify/code-runtime's own published input schema is already just { code } (confirmed live from the Actor-tool schema), so the direct-tool surface is a clean { code, waitSecs } out of the box.

2.5 — Docs stop being a tool. get-code-docs only ever served static text, and the name never sat right next to run-code because the thing itself is off — documentation isn't an action. fetch-actor-details already does exactly this job, it's already the call the guide tells the agent to make for every other Actor, and moving the guide into the Actor gives us one source of truth. Today the
guide lives here (code_docs_content.ts) while the implementation lives in the Actor repo, and it's already drifting — a dead docs/API.md link, and several binding methods documented but never exampled. Deleting get-code-docs also removes the loader-pairing hack and its unauth carve-out.

2.6 — The only thing lost is the per-call memory/timeout knob — and that knob is unusable as shipped. A direct Actor tool exposes only waitSecs, not callOptions. But the model has no way to know how much memory a script needs or how long it'll run; nothing in the docs guides the value; and a
timeout or OOM doesn't come back as "raise this and retry" — it fails opaquely and slips past the exitCode signal entirely (that signal only catches script throws, not resource-limit kills). So the right move isn't to keep an un-actionable knob — it's to set sensible defaults on the Actor, including a real default timeout (the Actor doesn't set one today). Net: simpler and no worse.

Put together: same behavior, two fewer bugs, ~580 fewer lines to maintain, one source of truth for the docs. If we want to keep the run-code name specifically, we can name the Actor accordingly — it isn't
published yet.

---

3. Before we ship: Code Mode hides the sub-Actor runs from MCP's attribution

This is the one I'm least sure about and most want us to check, because it changes how we'll read our own numbers. It's a property of Code Mode itself, not of the keep-or-drop decision above.

When a run-code script runs an Actor, it does so from inside the sandbox via the apify binding. The MCP server never starts those runs — it starts exactly one run, apify/code-runtime (run_code.ts:102-104 starts that Actor and nothing else). Every Actor the script then triggers is launched by the binding with the run's token, so it's recorded as an ordinary Actor run, not as a run
the MCP server initiated.

---

4. Small code notes

Quick, and most of these disappear if we go the Actor-tool route:

• No progress notifications on a blocking wait — server.ts:1092-1093 (see 2.3).
• No actorId on telemetry — run_code.ts sets none vs call_actor.ts:658,677,683 (see 2.3).
• run_code.ts:99-142 re-implements the executeCallActor pipeline by hand — that's where both of the
  above crept in; @ already spotted it. If we keep a dedicated tool,
  parameterizing executeCallActor fixes all three at once.
• run_code.ts:70 — the ${APIFY_CODE_RUNTIME_ACTOR} in the JSDoc prints literally (plain comment, not
  a template string).
• src/tools/AGENTS.md doesn't list the three new files.

---

5. Follow-up I'd want before/around release

This ties back to the top. The 5-task comparison I ran is the kind of thing that should live be present during the review so that I don't have to do it:

• A small standing eval — a few fan-out tasks, a few text-extraction tasks, pass/fail on the result — so we know whether the steering works and catch it when the binding drifts.
• A success metric we actually agree on. Right now "is this useful?" has no answer we can point to, and §3 means one obvious candidate (MCP-driven runs) may not even count Code Mode correctly.

Let's write it down.

None of this is a knock on the implementation. We should have caught in the spec earlier, it's my fault, sorry.

[MQ37]

Superseded by a self-contained design: no dedicated run-code/get-code-docs MCP tools needed. The generic call-actor (already a default tool) plus the Actor's own actor.json description/schema is sufficient — getNormalActorsAsTools reads that verbatim to build the tool surface, and call-actor's existing response builder already supplies the nextStep → get-dataset-items hint generically, for any Actor.

All the design/guide content here (sandbox limits, output contract, recipes) now lives in the Actor's own README/docs: apify/actor-code-runtime#1. That PR also closes the critical realFetch-exfil and redirect-following allowlist bypasses that predate this one, migrates to TypeScript, and adds MCP-run attribution (meta.origin) for sub-runs — none of which existed in this design.

Closing without merge.