Feature: Tool call prediction (speculative read prefetching)

[Fullstop000]

Summary

Predict which read-only tools the LLM will call and execute them before the model finishes generating, overlapping tool I/O with model inference. This cuts multi-tool turn latency by overlapping serial work.

The Problem

In the current agent loop, the LLM generates a response, then tools execute sequentially, then the LLM generates again:

LLM generating... (800ms)
  → read_file("src/auth.rs")     (150ms)
  → grep("password_hash")        (200ms)
  → read_file("src/db.rs")       (120ms)
LLM generating... (600ms)

Total: ~1.9s of which 470ms is pure waiting on filesystem/network I/O.

If the model is going to call read_file("src/auth.rs"), we already know that from the streaming JSON before generation ends. We could have started reading it 500ms ago.

Proposed Design

Scope: Read-Only Only

Only speculate safe, idempotent tools:

• ✅ read_file
• ✅ grep
• ✅ glob
• ✅ list_dir
• ❌ bash (side effects)
• ❌ edit_file (mutates state)
• ❌ create_file (mutates state)

If the prediction is wrong, the only cost is a wasted read — no state corruption.

How It Works

1. Stream parsing: While the LLM streams its response, inspect chunks for the beginning of a tool_use JSON block.
2. Early extraction: As soon as the tool name and arguments are parseable (even partially), check if it is a whitelisted read-only tool.
3. Speculative execution: Fire the tool call immediately in a background task, storing the result in a per-turn cache keyed by (tool_name, arguments_hash).
4. Result reuse: When the agent loop reaches the tool execution phase, check the cache first. If the speculative result is ready (or still in-flight), use it directly.

// Pseudo-code
while let Some(delta) = stream.next().await {
    // Accumulate into partial tool call buffer
    if let Some(predicted) = try_extract_tool_call(&delta) {
        if is_read_only(&predicted.name) {
            let cache_key = (predicted.name.clone(), predicted.args.clone());
            if !speculative_cache.contains(&cache_key) {
                let tool = tools.get(&predicted.name).cloned();
                tokio::spawn(async move {
                    if let Some(t) = tool {
                        let result = t.call(predicted.args).await;
                        speculative_cache.insert(cache_key, result);
                    }
                });
            }
        }
    }
}

Confidence Threshold

Not every partial match should trigger speculation. Start simple:

• Tool name is fully parsed and matches whitelist
• Arguments are at least 50% parseable (basic heuristic)
• Maximum 3 speculative calls per turn (cap waste)

Later: add a lightweight predictor based on prompt + recent history.

Measurement-First Approach

Do not build this until we have baseline data. Add one week of logging:

// In agent::run, log per turn:
tracing::info!(
    turn_duration_ms,
    model_duration_ms,
    tool_duration_ms,
    tool_count,
    tool_names,
    "turn.latency"
);

Build condition:

If P50(tool_duration_ms / turn_duration_ms) > 0.30:
    → Speculative prefetching has clear ROI
Else:
    → Model inference dominates; skip this feature

Benefits

Scenario	Latency Impact
Single-tool turn	~0% (nothing to overlap)
Multi-tool turn (2–3 reads)	~20–35% faster
Deep research (5+ reads)	~30–50% faster

Risks & Mitigations

Risk	Mitigation
Wrong prediction wastes reads	Limit to read-only tools; cap concurrent speculations
Cache grows unbounded	Per-turn cache dropped after each turn
Parse overhead slows streaming	Parse only after detecting "tool_use" substring
Race: speculative result arrives after normal execution	Use tokio::sync::OnceCell pattern; first caller wins

Implementation Sketch

1. Add speculative module in agent/mod.rs
2. Create SpeculativeCache (HashMap + tokio::sync::Mutex)
3. Modify stream consumer in Agent::run to parse deltas and trigger prefetches
4. Modify tool execution loop to check cache before calling
5. Add IGNIS_ENABLE_SPECULATIVE=true feature flag (default off)

Acceptance Criteria

• Baseline latency logging added to agent::run
• Feature only activates when IGNIS_ENABLE_SPECULATIVE=1
• Read-only tool whitelist is explicit and auditable
• Maximum 3 speculative calls per turn
• P50 multi-tool turn latency ↓ 20% (measured against baseline)
• Zero correctness regressions (speculative results are never used for write tools)
• Feature is disabled if baseline tool_time_ratio < 0.30

Open Questions

1. Should we cache speculative results across turns or only within a single turn?
2. How do we handle tool calls with relative paths that depend on the agent's current working directory mid-stream?
3. Should speculative calls run with lower CPU priority or on a dedicated tokio task pool to avoid starving the main agent loop?,