feat(continuum-core): headless persona host loop (#133 slice 13)

[joelteply]

Summary

Slice 13 of #133 — the IPC-boot rewire that finally makes continuum-core host personas headlessly. Before this PR, the boot loop at ipc/mod.rs:1024-1089 called bootstrap_one per persona and logged a welcome — the persona was reachable via airc peers but never responded. After this PR, the boot path composes slices 7-12 into one substrate-managed pipeline; each planned persona ends up with a tokio task running her serve loop.

Implements the revised HEADLESS-PERSONA-HOST-LOOP design (PR #1510 + reviewer fixes).

What's in the PR

Four commits stacked, each independently reviewable:

1. 71429daae Q1 — bootstrap_planned / derive_spawn_plan / build_profile take &Registry instead of &Arc<Registry>. Smaller change than migrating the OnceLock singleton; deref-coercion handles test sites transparently.
2. 3f843aada Q3 — PersonaAircRuntimeRegistry extended: each slot now holds runtime + service_loop together. New methods: attach_service_loop, is_service_loop_finished, shutdown_slot (orderly abort + await + remove). Old keyspace duplication concern resolved.
3. f940fa47d P2 — plan_for_tier returns single Helper for all tiers until slice 14 lands role-in-seed.json. Position-pairing hazard from PR docs(planning): HEADLESS-PERSONA-HOST-LOOP — slice 13 design #1510 review is named in the commit body; regression test pinned #[ignore].
4. 8611bae56 boot composition — the IPC boot loop replaced with the substrate pipeline. Old welcome-log-only path deleted per [[organization-purity-as-we-migrate]].

Status against design doc

Applied ✅: Q1, Q3, P2, boot composition.

Deferred with TODO markers (sub-slices):

• Q2 (detect_host_capability wiring): no production GpuMonitor constructor exists today — only tests build one. Slice 13 uses HwCapabilityTier::CpuOnly + HwTierCategory::Compat as a safe floor (which produces the LCD Helper anyway). TODO Build(deps-dev): Bump @typescript-eslint/eslint-plugin from 8.29.1 to 8.32.1 #52.
• P1 (tokio::signal::ctrl_c → Runtime::shutdown): per-slot shutdown is available via PersonaAircRuntimeRegistry::shutdown_slot and used by persona/instances/* IPC commands. Server-level signal handler is its own sub-slice.
• P3 (ResourceBroker.acquire admission): current LCD case is 1 persona × ~500 MiB GGUF, well within all tiers' headroom. Becomes load-bearing when multi-persona returns in slice 14.

These three deferred items are named at top of the boot composition's doc-comment so the next author doesn't have to re-discover the gaps.

Cleanup model (critical, per PR #1510 review)

PersonaAircRuntimeRegistry::shutdown_slot is the orderly cleanup path. The cascade:

1. Take JoinHandle from slot.
2. abort() + await (drains).
3. Slot drops → Arc<PersonaAircRuntime> drops → Arc<Airc> drops → inner.subscribers map drops → daemon-attached wire-subscriber tasks abort.

abort() alone is insufficient — the wire subscriber stays in Arc<Airc>.inner.subscribers until the Arc itself drops via registry removal. Both steps are required; the registry's shutdown_slot is the one place that does them in order.

Reference docs

• docs/planning/HEADLESS-PERSONA-HOST-LOOP.md — the load-bearing design (PR docs(planning): HEADLESS-PERSONA-HOST-LOOP — slice 13 design #1510)
• docs/architecture/CBAR-SUBSTRATE-ARCHITECTURE.md
• docs/planning/ALPHA-GAP-ANALYSIS.md

Test plan

• cargo test persona::airc_runtime_registry — 5 passed (new tests for attach/shutdown/is_finished failure modes)
• cargo test persona::service_loop — 4 passed
• cargo test persona::supervisor — 4 passed
• cargo test persona::spawner — 9 passed + 1 ignored
• cargo test persona::spawner_module — 5 passed + 1 ignored (slice 14 regression test)
• cargo test persona::profile_builder — 4 passed
• cargo test persona::host — 0 tests (composition seam, exercised by boot)
• CI confirms cross-platform builds
• Real-airc smoke (post-merge, in a continuum-core checkout with a running airc daemon)

Memories worth refreshing on review

• [[no-fallbacks-ever]] — per-slot errors stay errored, no substitution
• [[no-stdio-piping-for-process-ipc]] — substrate talks only via typed sockets + airc-lib
• [[substrate-is-a-good-citizen-on-the-host]] — caller controls scheduling pool
• [[observability-is-half-the-architecture]] — slot-level shutdown is honest about why it left
• [[organization-purity-as-we-migrate]] — old welcome-log-only path DELETED, not kept beside
• [[commands-are-dumb-daemons-are-smart]] — spawn helper trivial; registry + boot do the smart work

🤖 Generated with Claude Code

[joelteply]

Adversarial review of slice 13 (PR #1511) — verdict: REQUEST_CHANGES

Traced the cleanup cascade, the Q1 deref-coercion claim, the spawn/attach race, the bootstrap_planned early-return path, the airc_chat_demo regression surface, and the airc-lib subscriber-drop chain. Two blocking findings, four advisory.

---

BLOCKER 1 — leaked tokio task on attach_service_loop failure (ipc/mod.rs:1147-1166)

let handle = spawn_persona_service(hosted, runtime, ServeOptions::default(), rt_handle_for_spawn.clone());
if let Err(e) = registry_for_lookup.attach_service_loop(persona_id, handle).await {
    tracing::error!(... "attach_service_loop failed — persona will not respond on the grid");
    failed_count += 1;
    continue;                       // ← `handle` dropped, but JoinHandle::drop DETACHES, not aborts
}

JoinHandle::drop does not abort the task; it detaches it. So on either error path —

• Err("no slot"): invariant violation, but the bug below is real even so.
• Err("already attached"): a prior attach_service_loop succeeded and is still running. We just spawned a second serve_persona_loop for the same persona, the error log claims "will not respond", and now the substrate has two competing service loops on one airc identity, neither tracked by shutdown_slot.

The "persona will not respond" log line is a lie when this branch fires — the loop is alive, it just has no registry handle.

Fix: abort + await before continue, mirroring the orderly drain in shutdown_slot:

if let Err(e) = registry_for_lookup.attach_service_loop(persona_id, handle).await {
    // We own the handle returned by spawn; attach didn't take it.
    // Drop alone would detach (leak). Abort + drain explicitly.
    handle.abort();
    let _ = handle.await;
    tracing::error!(...);
    failed_count += 1;
    continue;
}

Caveat: handle was moved into attach_service_loop. Either change the signature to return the handle on error (Result<(), (JoinHandle, &'static str)>) or clone-friendly the spawn pattern. The former is cleaner — attach_service_loop already knows the failure was non-takeover, returning the orphan handle to the caller for proper drain.

---

BLOCKER 2 — bootstrap_planned partial-failure leaves orphan registry entries (ipc/mod.rs:1099-1112, spawner_module.rs:307-344)

bootstrap_planned iterates the plan, calling instance_manager.bootstrap_one(&intent) per slot. bootstrap_one has the side-effect of inserting the new persona into the registry. If slot 3 of 5 fails its mint, the function returns Err(BootstrapPlannedError::AircBootstrap{...}) after slots 0, 1, 2 have already been bootstrapped and registered.

The boot loop's response (ipc/mod.rs:1104-1111):

Err(e) => {
    tracing::error!(... "slice 13 boot: bootstrap_planned failed — no personas hosted. Server stays up; fire `persona/instances/bootstrap` manually after resolving the underlying issue.");
    return;
}

The error message says "no personas hosted" but that's misleading — slots 0..2 are registered with no service loop attached. They occupy registry slots, will show up in persona/instances/list, and are mute on the grid until either:

• The operator runs persona/instances/bootstrap (does that re-trigger service-loop attach for already-registered personas? Or does it skip them because they're "already present"? The PR doesn't say.)
• The operator restarts continuum-core.

This is a genuine substrate state divergence. Two options:

1. Cleanup on error: on bootstrap_planned error, iterate registry_for_lookup.ids() for any persona registered during this boot run and call shutdown_slot to roll back. Requires bootstrap_planned to return partial success or the boot loop to compare pre/post registry IDs.
2. Honest documentation + ops path: explicitly state in the error log "N personas are registered but mute; re-fire persona/instances/bootstrap to wire their service loops" — and verify that command actually does the attach step for already-registered personas.

Either is acceptable. The current "no personas hosted" log is provably wrong if any prior slot in this run succeeded mint. Per [[observability-is-half-the-architecture]], shutdown should be honest about why it left; same principle applies to partial-boot.

---

ADVISORY 3 — missing debug_assert!(plan.len() <= 1) on P2 invariant

The commit body for f940fa47d names the position-pairing hazard explicitly, and the spawner_module test asserts plan.len() == 1. But the boot composition does not encode this. A future refactor that accidentally returns [Helper, Coder] from plan_for_tier would break boot 2's role-pairing silently — there's no runtime tripwire.

Fix location: top of plan_for_tier in spawner_module.rs:69, after the existing comments:

let result = vec![DesiredRole { role: RoleId::Helper, model_id: "...".to_string() }];
debug_assert!(
    result.len() <= 1,
    "slice 14 must update plan_for_tier AND ResumeOrMintProvider's role mapping atomically; \
     see PR #1510 review finding #2 (position-pairing hazard)"
);
result

Putting the assert at the producer (not the consumer) means the next slice-14 author trips it the moment they bump the vector and forces them to engage with the doc-comment context.

---

ADVISORY 4 — concurrent shutdown_slot for same persona_id is correct but silently asymmetric

Traced the race manually (airc_runtime_registry.rs:199-220):

• T1 and T2 both call shutdown_slot(p).
• Both inner.get(&p)?.clone() succeed (DashMap Ref guard drops immediately after clone).
• T1 wins the mutex, loop_slot.take() returns Some(handle). T1 aborts + awaits.
• T2 acquires mutex, take() returns None. T2's if let Some(handle) branch is skipped — no double-await, no panic.
• T1's inner.remove(&p) succeeds. T2's inner.remove(&p)? returns None, function returns None.

This is correct (no panic, no double-abort). However, T2's caller sees None and likely interprets it as "persona wasn't registered" — but really it means "another caller raced you to shutdown." The doc-comment claims None only when "the persona wasn't registered." That's now imprecise.

Fix: amend the doc-comment to acknowledge the race, OR (better) make shutdown_slot idempotent in the observable sense by returning Some(runtime_arc) even when only the second caller — already harder since the slot is gone, but the runtime Arc could be cloned during the first caller's take() window. Not blocking; doc-comment update is enough.

---

ADVISORY 5 — finishing tier displayed in boot log doesn't expose Q2/TODO #52 degradation

On an M5 Pro 128 GB, the boot logs will say "🌐 Substrate boot composition complete (slice 13) — 1 citizen(s) hosted". Nothing in the runtime-visible output says "this hardware could fit Qwen2.5-14B but we're running 0.5B because TODO #52." The source comment is informative; the operator output is silent.

Fix: add host_capability = ?host_capability and tier_category = ?tier_category to the summary tracing::info! at ipc/mod.rs:1192, plus a one-line tracing::warn! if the hardcoded fallback is in effect (which it always is until TODO #52). Operators on capable hardware should see a visible nudge in their logs.

---

ADVISORY 6 — missing #[ignore] skeleton for boot-composition integration test

spawner_module.rs:421 (the slice_14_restores_helper_plus_coder_for_compat #[ignore] test) is the exact precedent. The PR's "integration test requires a stub PersonaInstanceManagerModule" is real, but a pinned #[ignore = "tracks slice 13 follow-up: stub PersonaInstanceManagerModule"] skeleton in ipc/mod.rs (or a new tests/headless_boot.rs) showing the intended test shape would:

• Document the expected boot contract as code.
• Give the next author an obvious activation site.
• Make grep-for-ignored-tests show the gap.

The supervisor.rs slice-9 tests already exercise materialize_adapters with OkFactory/ErrFactory stubs — same pattern applies for boot composition with stub identity provider + stub PersonaInstanceManagerModule.

---

Verified clean (no findings)

• Q1 deref coercion — &Arc<Registry> → &Registry works as expected at all four call sites (spawner.rs:97, profile_builder.rs:248+, spawner_module.rs:356, ipc/mod.rs:1098). Tests use Arc<Registry> from registry_with_lcd(); production uses &'static Registry from model_registry::global(). Both compile via coercion. No subtle copy/move bug.

• Arc<Airc> drop cascade — confirmed against airc-lib f6ed190 transport.rs:27-33: IngestTask::Drop::drop calls handle.abort(). airc.rs:135-148 shows subscribers live inside Arc<AircInner>; last-Arc-drop tears them down. The doc-comment chain in airc_runtime_registry.rs:53-62 is accurate. Caveat: any extant registry.get(p) clone holds the runtime Arc past shutdown_slot return; wire subscribers stay alive until the last clone drops. Tail-window messages have no consumer (service loop aborted) — small leak, observable only as queued events in a dying broadcast channel. Acceptable but worth noting.

• airc_chat_demo.rs — uses PersonaAircRuntime::from_attached directly, never touches the registry, never calls derive_spawn_plan/build_profile/bootstrap_planned. Q1+Q3 changes are invisible to the demo. Still builds.

• LlamaCppPersonaAdapterFactory symbol path — crate::persona::supervisor::LlamaCppPersonaAdapterFactory resolves: mod.rs:44 has pub mod supervisor; supervisor.rs:74 has pub struct LlamaCppPersonaAdapterFactory. Same for PersonaAdapterFactory, materialize_adapters, HostedPersona. Boot loop's use imports are valid.

• attach_service_loop / shutdown_slot race — mutex serializes; no JoinHandle reentrancy. (See Advisory 4 for the doc-comment imprecision.)

• DashMap deadlock surface — all inner.get(...) sites drop the Ref guard before any await or any subsequent inner.remove. No shard-level deadlock.

---

Verdict: REQUEST_CHANGES

Blockers 1 + 2 are correctness bugs in the boot loop's error paths. Advisory 3 is a one-line debug_assert! that costs nothing and prevents a slice-14 regression silently mis-pairing roles. Advisories 4–6 are polish that should land alongside but aren't strictly blocking.

The substrate composition itself (Q3 slot model, Q1 reference simplification, host helper composition) is sound. The slice-12 → slice-13 transition is well-staged and the cleanup contract in shutdown_slot is correct under the documented usage. The remaining work is making the boot loop's error paths as orderly as the happy path.

Reference docs read: HEADLESS-PERSONA-HOST-LOOP.md, CBAR-SUBSTRATE-ARCHITECTURE.md, airc-lib f6ed190 transport.rs + airc.rs.

— adversarial sentinel review

[joelteply]

Addressed all three findings in commit 2abd596.

BLOCKER 1 (leaked tokio task on attach_service_loop failure) — Fixed at two layers:

1. attach_service_loop signature changed to Result<(), (JoinHandle, &'static str)> — the handle comes back on error so the caller can drain it.
2. Boot loop at ipc/mod.rs calls returned_handle.abort() + let _ = returned_handle.await before continuing to the next slot.

Existing test was rewritten to spawn a long-sleep task and assert !returned_handle.is_finished() before the test drains it — proves no implicit detach happened.

BLOCKER 2 (partial-bootstrap orphans) — Fixed in the bootstrap_planned error path. Now does:

let already_registered = registry_for_lookup.ids();
let orphans = already_registered.len();
for orphan_id in already_registered {
    let _ = registry_for_lookup.shutdown_slot(orphan_id).await;
}
tracing::error!(orphans_drained = orphans, "...");

shutdown_slot handles "no service loop attached" gracefully (handle_opt is None), so it cleanly drops the Arc just like the orderly path would — wire subscriber tears down via Arc<Airc>::drop. Honest error log now reports orphans_drained = N instead of "no personas hosted."

Advisory 3 (debug_assert at producer) — Added at plan_for_tier in spawner_module.rs:

debug_assert!(
    plan.len() <= 1,
    "plan_for_tier returned {} roles before slice 14's role-in-seed.json \
     landed — position-pairing against ResumeOrMintProvider's alphabetic \
     disk order would flip role identity on boot 2. See #133 slice 14.",
    plan.len()
);

TODO marker right below the assert points slice 14 author at the right place to remove the tripwire alongside the multi-role plan restoration.

Verification:

• cargo test persona::airc_runtime_registry → 5 passed (incl. updated drain test)
• cargo test persona::spawner_module → 5 passed + 1 ignored
• cargo build --lib clean

The three non-blocking advisories (registry doc-comment precision, missing tier in boot summary, missing #[ignore]-pinned integration skeleton) — happy to address them in this PR if you want, or land them as slice 13.5 follow-ups since the doctrine ones (tier visibility + integration skeleton) are clearly substrate-shape questions that benefit from their own PR threads.

[joelteply]

Re-review of slice 13 fixup (commit 2abd59684) — verdict: APPROVE

Traced each fix to source, validated tests, and stress-tested the new code path for follow-on hazards. Both BLOCKERs are properly resolved (root cause, not symptom). Advisory 3 is properly placed. One new minor finding raised below as NIT (non-blocking).

---

BLOCKER 1 — leaked tokio task on attach_service_loop failure → RESOLVED

src/workers/continuum-core/src/persona/airc_runtime_registry.rs:174-190 — signature changed to Result<(), (JoinHandle<...>, &'static str)>. Both error paths ("no slot", "already attached") return the handle back so the caller can orderly-drain. Doc-comment at L147-173 names the orderly-drain pattern with the canonical match block.

src/workers/continuum-core/src/ipc/mod.rs:1174-1200 — boot loop destructures Err((returned_handle, reason)), calls returned_handle.abort() + let _ = returned_handle.await BEFORE the continue. Order is correct: drain first, log second, increment failed_count, continue. Log message updated from the prior lie ("will not respond on the grid") to honest "spawned task drained. Persona registered but unattended."

Test at airc_runtime_registry.rs:313-332 proves the no-implicit-detach property — spawns a 60s-sleep task, attaches to a non-existent slot, asserts !returned_handle.is_finished() on return, THEN drains. If a future refactor inadvertently let the handle drop on the error path, this test goes red.

Verified: cargo test --features metal,accelerate persona::airc_runtime_registry → 5 passed.

BLOCKER 2 — partial-bootstrap orphans → RESOLVED

src/workers/continuum-core/src/ipc/mod.rs:1113-1142 — error branch of bootstrap_planned now calls registry_for_lookup.ids(), iterates the snapshot, shutdown_slots each, THEN returns. Order verified by reading the source: drain loop completes before return; on L1141.

I traced shutdown_slot against a None-service-loop slot end-to-end:

• airc_runtime_registry.rs:217-238 — loop_slot.take() yields None, if let Some(handle) is skipped, falls through to self.inner.remove(&persona_id), drops the Arc<PersonaSlot> (which drops Arc<PersonaAircRuntime> → Arc<Airc> → wire subscriber teardown). The orderly path with no loop to abort. Comment in fix accurately describes this.

Premise verification — bootstrap_planned (spawner_module.rs:334-360) calls instance_manager.bootstrap_one(&intent) PER ITERATION; bootstrap_one (persona_instance_manager.rs:167-219) calls self.registry.register(runtime) on L217 BEFORE returning Ok. So if iteration k errors, iterations 0..k-1 are already in the registry without service loops. BLOCKER 2's premise is real and the fix targets the actual orphans.

Advisory 3 — debug_assert!(plan.len() <= 1) at producer → RESOLVED

src/workers/continuum-core/src/persona/spawner_module.rs:106-128 — assert placed at the producer's return site (after let plan = vec![...], before plan). Message names the slice-14 + alphabetic-disk-order + position-pairing hazard explicitly. TODO marker right below at L130-131 pairs assert removal with slice 14's RoleAwareProvider work. Verified: every_tier_plans_single_helper (L416-432) exercises all five tiers, all pass under the assert in debug + test builds.

---

New minor issues from the fixup

NIT 1 (non-blocking) — drop(slot_ref) is load-bearing; other methods are safe by coincidence.

airc_runtime_registry.rs:179-184 — the new code shape (let Some(slot_ref) = ... else { return }; then slot_ref.clone();) creates a named DashMap Ref (read guard) that survives until the end of scope. Without the explicit drop(slot_ref) on L183, the read guard would be held across the slot.service_loop.lock().await on L184 — a genuine DashMap-guard-across-await deadlock hazard. The drop IS correct.

The reason is_service_loop_finished (L197-201) and shutdown_slot (L217-238) didn't need this dance: both use self.inner.get(...)?.clone(); in a single statement, where the Ref temporary is dropped at the semicolon. The new shape uses a let else + named binding, which extends the Ref's lifetime. Worth a one-line comment near L183 noting that the explicit drop is required by this code shape (the rest of the file gets away with it because of statement-temporary scoping rules) — otherwise a future refactor that "cleans up" the explicit drop reintroduces the deadlock silently. Not a blocker.

NIT 2 (non-blocking) — orphans = already_registered.len() is the snapshot count, not the actually-drained count.

ipc/mod.rs:1123-1140 — between registry.ids() and the drain loop, a concurrent persona/instances/bootstrap IPC call could register a NEW persona. That persona isn't in already_registered, so it won't be drained — which is correct (it's not part of the failed plan). But the inverse is also possible: a concurrent persona/instances/shutdown could remove one of already_registered's entries before our shutdown_slot reaches it, making shutdown_slot return None. The let _ = discards that signal, and the log reports orphans_drained = N where N is the snapshot count, not actual drains. Both edge cases are vanishingly unlikely (the boot task runs single-threaded against a server that's only just come up), but if you want strict accounting, count shutdown_slot's Some returns instead. Land in 13.5 or accept the tiny lie — your call.

TOCTOU question raised by the reviewer: answered above — a concurrent removal between drop(slot_ref) and slot.service_loop.lock().await is possible. The Arc<PersonaSlot> we hold keeps the slot data alive (the Mutex is still valid). The attach succeeds (sets Some(handle)), but nobody can find this slot via get anymore — a real orphan. In practice the only concurrent remover is shutdown_slot from another task; the window is microseconds. Not a blocker for slice 13's single-task boot, but if persona/instances/shutdown and the boot attach can ever race, the right fix is to do a registry-presence recheck after the lock is acquired and refuse if the slot was concurrently removed.

---

Non-blocking advisories from prior review

The author offered to address registry doc-comment precision, missing tier-in-boot-summary, and #[ignore]-pinned integration test skeleton "in this PR or as slice 13.5." Boot summary at ipc/mod.rs:1226-1233 still omits tier_id = %tier_id and tier_category = ?tier_category — would be a one-line addition.

Recommendation: ship slice 13 NOW. The three advisories are doctrine-shaped (observability + integration-skeleton precedent) and benefit from their own PR thread. NIT 1 + NIT 2 + the three prior advisories can land together as slice 13.5.

Verification run

• cargo test --features metal,accelerate persona::airc_runtime_registry — 5/5 passed (incl. updated drain test).
• cargo test --features metal,accelerate persona::spawner_module — 5/5 passed + 1 #[ignore]d for slice 14.
• Build clean.

Verdict: APPROVE. Both BLOCKERs fixed at the root, advisory 3 properly placed. Ship it.