fix: keep blocking docker and subprocess calls off hypervisor-agent and agent-spawner event loops

[welshDog]

Fourth of the sync-I/O-on-an-async-loop fixes (#309, #310, #311). Neither service is deployed, so neither has misbehaved — but agent-spawner carries the exact pair of ingredients that took the healer down in #309.

hypervisor-agent

auto_scale_containers() and heal_back() called containers.list(), containers.get(), c.stop(timeout=15), c.start() and c.remove() — all synchronous docker-py — directly on the event loop.

Fixed by extracting the blocking bodies into _auto_scale_sync / _heal_back_sync and driving them with asyncio.to_thread, matching what _refresh_container_cache already does for _collect_container_metrics. DRY-RUN semantics, the ENFORCE_SCALING guards, and the returned action strings are unchanged.

It holds no pub/sub connection, so the symptom was latency only.

I previously called this file "clean" and held it up as the pattern to copy. It does use asyncio.to_thread — at line 321, for the metrics sweep. The auto-scale path never did. One correct call site is not evidence about the rest of a file.

agent-spawner — the interesting one

spawn_agent() ran subprocess.run(timeout=30) — up to 30 seconds of frozen loop — and shutdown_agent() / spawn_listener() called containers.get() and container.stop(timeout=10).

This is dormant but not benign. spawn_listener holds a pubsub whose read carries a 5s socket_timeout, and check_idle_agents calls shutdown_agent concurrently. A 10s blocking stop() while that read is pending expires the timeout, spawn_listener raises, and because it re-raises into asyncio.gather the whole service dies rather than retrying.

That's the healer's failure exactly — minus the retry loop that turned the healer's into a connection leak. Here it would just crash the process.

Verification

Neither service has a container, so I extracted the real functions from source with ast.get_source_segment, ran them against a fake slow docker/subprocess, and measured event-loop stall with a 50ms ticker task. The same harness was run against the pre-fix code from main as a control:

function	control	fixed
auto_scale_containers	1982.1 ms	17.2 ms
heal_back	1292.6 ms	15.8 ms
spawn_agent	1999.4 ms	16.9 ms
shutdown_agent	1692.7 ms	28.8 ms

(The fake docker sleeps ~2s; the real subprocess.run ceiling is 30s.)

Behaviour assertions pass identically on control and fixed — DRY-RUN tags preserved, zombie and non-critical rules intact, critical containers untouched, stopped_by_agent bookkeeping correct, spawn lock released, activity tracking updated. So the refactor changed timing, not semantics.

Not verified

Neither service has a container in the current stack. This is tested at the function level against fakes, not end-to-end.

Found by fixing the tool, not by reading

The AST sweep used in #311 only matched httpx/requests/docker.from_env/.containers.get|list. It missed subprocess.run entirely, and missed every c.stop() / c.start() / c.remove() because those are methods on container objects, not on .containers. Teaching it to track variables bound from containers.get()/.list() and to flag subprocess.* took it from 4 hits to 9 on these two files alone.

Re-sweeping with the better rules turns up two more, both in running containers, deliberately left out of this PR:

• agents/hyperhealth/main.py:111 — subprocess.run() inside async def (hyperhealth-api is running)
• agents/broski-bot/cogs/status_cog.py:39 — subprocess.run() inside async def (broski-bot is running)

And agents/test-agent/main.py:193 calls time.sleep() in a coroutine (no container).

The sweep's blind spot still stands: it only sees calls written lexically inside a coroutine, so it would not catch the broski-pets-bridge shape from #310, where the sync work sits in a helper def.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@welshDog, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 38 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c57cba74-4b70-4a98-9dc3-938509a6dad3

📥 Commits

Reviewing files that changed from the base of the PR and between 798a565 and a0e5981.

📒 Files selected for processing (2)

• agents/hypervisor-agent/hypervisor_agent.py
• services/agent-spawner/spawner.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/hypervisor-spawner-blocking-loop

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}