Sandboxes: comparison mode can never complete; compare is unmetered; cleanup job hard-requires google-cloud-run; instance tiers unwired

[TonsOfFun]

Part of an ActiveAgent + actionagent dashboard functional review (multi-agent, adversarially verified). Severity: 🟠 Major.

The engine sandbox surface has several dead ends: comparison mode subscribes to a platform-only SandboxChannel that the engine doesn't ship (and the polling fallback is dead code), so it can never complete in the UI; #compare enforces the execution quota but never records usage (comparisons are free); SandboxCleanupJob bypasses the backend registry and hard-requires google-cloud-run (a non-dependency, so LoadError escapes its rescue); and the instance-tier feature is unwired end to end (selector never mounted, tier never accepted by create, never passed to provisioning).

Findings

Engine sandbox comparison mode can never complete in the UI: SandboxChannel is platform-only and the polling fallback is dead code

• Where: actionagent/frontend/components/dashboard/SandboxRunner.jsx:172 · severity: major · kind: missing · repo: activeagent
• What breaks: SandboxRunner.jsx subscribes to ActionCable channel 'SandboxChannel' for run_started/run_complete/run_error, but the engine ships no channel classes at all (no app/channels anywhere in /home/user/activeagent) and the dummy host has no cable mount, so no message ever arrives. The only non-cable fallback for comparison mode is a single setTimeout after 60s guarded by if (runningProviders.length > 0) — a stale closure that captured the render-time value [] (set before setRunningProviders commits) — so pollForCompletion() never fires. Result: POST /api/sandboxes/compare executes and records runs server-side (proven live), but the engine UI shows every provider stuck at 'Processing...' and isRunning stays true forever; the Run button never re-enables. Single-provider runs are unaffected because runSingleProvider uses the 1s pollRunStatus loop. This also answers 'anything else platform-only called by the engine copy': besides the known /api/usage 404, SandboxChannel is the other platform-only dependency (checkout.js was already engine-ified via upgradeUrl). For /api/usage the right fix is to add an engine GET /api/usage (route + controller) returning the platform shape {usage:{runs_used,runs_limit,runs_remaining,can_run,plan}} sourced from the host app's quota hooks (ActionAgent.quota_denial / a new usage resolver), with an unlimited/null shape when the host configures none — both SandboxRunner.jsx:186 and OrganizationView.jsx:21 read that exact shape.
• Evidence: actionagent/frontend/components/dashboard/SandboxRunner.jsx:171-176 (useActionCable('SandboxChannel',...)), :353-358 (dead fallback: setTimeout closure over stale runningProviders), :313-323 (setRunningProviders after render capture). No channel classes: grep -rn "SandboxChannel|ApplicationCable" /home/user/activeagent --include=*.rb returns nothing; app copy has /home/user/activeagents/app/channels/sandbox_channel.rb. Live: curl --noproxy localhost http://localhost:3001/cable -> 404. Live compare works server-side: POST /activeagents/api/sandboxes/compare returned 202 with 2 runs; GET /activeagents/api/sandboxes/ 4s later showed both providers status 'completed' — results only reachable by the polling the comparison UI never performs. Engine SandboxRunJob broadcasts to 'sandbox_<session_id>' (actionagent/app/jobs/action_agent/sandbox_run_job.rb:243-283) with no subscriber possible.
• Suggested fix: In the engine (repo activeagent): make comparison mode poll like single runs — replace the dead 60s setTimeout (SandboxRunner.jsx:353-358) with a 1s interval that GETs /api/sandboxes/:session_id (pollForCompletion already does the reconciliation) until every selected provider has a completed/failed run, then setIsRunning(false)/setRunningProviders([]); this removes the ActionCable dependency and the stale closure at once (or drive the check from a useEffect on runningProviders / a ref if the timeout is kept). If cable support is desired, note a documented cable mount alone is insufficient — the engine must also ship app/channels/action_agent/sandbox_channel.rb streaming sandbox_<session_id> (port from /home/user/activeagents/app/channels/sandbox_channel.rb), since no SandboxChannel class exists engine-side for any consumer to subscribe to. Also worth porting the polling fix back to the app copy, which carries the same dead fallback and would break identically if its cable ever fails. The /api/usage suggestion is valid but duplicates known finding 3.

---

Engine compare enforces the execution quota but never records usage — comparisons are free

• Where: actionagent/app/controllers/action_agent/api/sandboxes_controller.rb:55 · severity: minor · kind: bug · repo: activeagent
• What breaks: Api::SandboxesController#run calls record_execution_usage before enqueueing its single SandboxRunJob, so single runs count against the host app's quota. #compare gained enforce_execution_quota! as a before_action (so a caller already at the limit is denied), but the action spawns one SandboxRunJob per provider (2-3 LLM generations) without ever calling record_execution_usage — so comparisons never increment usage and a caller can run unlimited N-provider comparisons while staying at 0 recorded executions, making the quota gate on compare itself ineffective over time. The check-without-record asymmetry indicates the record call was simply missed when the quota hooks were added.
• Evidence: actionagent/app/controllers/action_agent/api/sandboxes_controller.rb:14-15 (require_execution_enabled!/enforce_execution_quota! on :compare), :55-64 (compare enqueues one job per provider, no record_execution_usage anywhere in the action) vs :137 (#run calls record_execution_usage); actionagent/app/controllers/action_agent/api/base_controller.rb:86-101 (enforce_quota!/record_execution_usage definitions). Code path read end-to-end; compare verified executing live on :3001.
• Suggested fix: In #compare, after the can_run? guard and before enqueueing, record usage once per provider: providers.size.times { record_execution_usage } (or extend ActionAgent.record_usage to accept a count and call it once). Place it before the providers.map block so usage is counted even if a later enqueue raises, mirroring #run's record-before-enqueue ordering.

---

Engine SandboxCleanupJob bypasses the backend registry and hard-requires google-cloud-run, which is not a dependency — LoadError escapes its rescue

• Where: actionagent/app/jobs/action_agent/sandbox_cleanup_job.rb:34 · severity: minor · kind: drift · repo: activeagent
• What breaks: The engine's sandbox stack was refactored so all infrastructure goes through SandboxOrchestrator/ActionAgent.sandbox_backends ('the engine carries none of those SDKs' — sandbox_orchestrator.rb:24-30), and SandboxProvisionJob follows that. SandboxCleanupJob was not adapted: delete_cloud_run_job still does require "google/cloud/run/v2" and calls the Google SDK directly. actionagent.gemspec declares no google-cloud gem, so in any non-development engine install the require raises LoadError — which is a ScriptError, NOT caught by the method's rescue => e (StandardError only) — and the job fails and retries. The trigger is routine: SandboxSession#expire! (the DELETE /api/sandboxes/:id path and cleanup_expired!) enqueues the job whenever cloud_run_job_id is present, and even the built-in mock backend populates cloud_run_job_id (provision stores result[:sandbox_id] = container_name 'mock-sandbox-...'). The registered backend's own terminate is meanwhile never invoked, so a real host backend leaks containers.
• Evidence: actionagent/app/jobs/action_agent/sandbox_cleanup_job.rb:14-16 (guard skips development only), :33-40 (require + rescue => e which cannot catch LoadError); grep -n google actionagent/actionagent.gemspec actionagent/Gemfile -> no matches (the platform app has gem "google-cloud-run-v2" at activeagents/Gemfile:66); actionagent/app/models/action_agent/sandbox_session.rb:116-120 (expire! enqueues when cloud_run_job_id present); actionagent/app/jobs/action_agent/sandbox_provision_job.rb:22-27 (mark_ready! stores orchestrator sandbox_id as cloud_run_job_id, mock backend included).
• Suggested fix: In delete_cloud_run_job (or inline in perform), replace the direct Google SDK call with SandboxOrchestrator.new.terminate(sandbox.cloud_run_job_id), rescuing StandardError and logging as today. This drops the google/cloud/run/v2 require from the engine entirely, matches SandboxProvisionJob's already-migrated pattern, and invokes the registered backend's terminate/terminate_pod/cancel_job via ADAPTER_METHODS so host backends actually reclaim containers.

---

Instance-tier feature is unwired end to end: selector never mounted, tier never accepted by create, never passed to provisioning

• Where: actionagent/app/controllers/action_agent/api/sandboxes_controller.rb:167 · severity: minor · kind: missing · repo: activeagent
• What breaks: The tier API works (verified live: index/show/recommend/pricing all 200 on :3001, unknown id -> 404 JSON) and SandboxOrchestrator#create_sandbox supports instance_tier — but nothing connects the pieces: InstanceTierSelector.jsx is imported by no component in either repo (dead code in both frontends), Api::SandboxesController#sandbox_params permits only :sandbox_type so a chosen tier could never reach the model, and SandboxProvisionJob calls SandboxOrchestrator.new.create_sandbox(sandbox) without instance_tier, so every sandbox provisions on the backend default regardless. The 'Colab/HuggingFace-style hardware selection' the component and endpoints advertise cannot happen. The app sibling is identically dead, with an extra wrinkle: its Api::InstanceTiersController inherits ApplicationController without allow_unauthenticated_access, so the anonymous free-tier sandbox flow it is styled for would get a 302 HTML redirect to /session/new instead of JSON (verified live on :3000).
• Evidence: grep -rn InstanceTierSelector over both frontends matches only the component's own file in each repo (no importer); actionagent/app/controllers/action_agent/api/sandboxes_controller.rb:166-168 (params.permit(:sandbox_type) only — same at activeagents app/controllers/api/sandboxes_controller.rb:163-165); actionagent/app/jobs/action_agent/sandbox_provision_job.rb:22 (create_sandbox with no instance_tier despite sandbox_orchestrator.rb:73-95 supporting it). Live: GET http://localhost:3001/activeagents/api/instance_tiers -> 200; .../recommend?sandbox_type=playwright_mcp -> 200 cpu_medium; .../pricing -> 200; .../bogus -> 404 JSON. App anon: GET http://localhost:3000/api/instance_tiers -> 302 to /session/new.
• Suggested fix: In the engine: mount InstanceTierSelector in SandboxRunner's setup UI; add :instance_tier to sandbox_params (sandboxes_controller.rb:167) and persist it on SandboxSession; have SandboxProvisionJob pass it through (create_sandbox(sandbox, instance_tier: sandbox.instance_tier) at sandbox_provision_job.rb:22). Port the same to the app copy (app/controllers/api/sandboxes_controller.rb sandbox_params), and additionally add allow_unauthenticated_access to app/controllers/api/instance_tiers_controller.rb (or rebase it on the Api::BaseController pattern used by SandboxesController) so the anonymous free-tier flow gets JSON instead of a 302 to /session/new. Alternatively, if hardware selection is not shipping, delete InstanceTierSelector.jsx from both frontends and the unused instance_tiers routes/controllers.

Verification

Each finding above was produced by a dedicated per-feature review agent, then confirmed by an independent adversarial verifier (all rated high-confidence; zero rejected in this set). File:line citations are against the current main/HEAD of each repo; many were reproduced live against a booted dashboard.