feat(tracing): trace scheduled Runtime processing (#1243)

docs/en/docs/how-to/trace-with-phoenix.md

@@ -68,12 +68,13 @@ Memory extraction runs during the flush, not during capture.
 ## Read the trace
 
 Open <http://localhost:6006>, select the `default` project, and open the most recent trace for
-`powercontext-server`. The flush produces four nested spans in one trace:
+`powercontext-server`. The flush produces five nested spans in one trace:
 
 | Span | Meaning |
 | --- | --- |
 | `HTTP flush_memory` | The inbound HTTP request. `powercontext.request.id` matches the `X-PowerContext-Request-ID` response header. |
 | `powercontext flush_memory` | The application operation, independent of the transport that invoked it. |
+| `memory.flush` | The Runtime stage that processes the Source window. Inference spans nest beneath it when extraction runs. |
 | `invoke_agent memory_extraction` | One PowerContext generation task. The name identifies the purpose, not the model. |
 | `chat <model>` | One request to the model provider, with token usage and latency. |
 
@@ -84,9 +85,11 @@ take the write lock, so they emit no `scope.lock` span:
 | --- | --- |
 | `scope.context` | Resolving the scope's context from the configured provider; near zero for the built-in provider, visible when a provider does I/O here. |
 | `scope.lock` | Waiting for the scope write lock, ending the moment it is acquired. `powercontext.scope.lock.contended` reports whether another operation already held it. |
+| `memory.flush` | One Source-window flush for `flush_memory` or a scheduled activation. |
 | `memory.search` | Memory lookup for `search_memory` or `prepare_context`; embedding and reranking spans, when present, are nested beneath it. |
 | `memory.rerank` | One actual reranker call; model-backed reranking nests `invoke_agent memory_rerank` beneath it. |
 | `experience.search` | Experience recall during `prepare_context`; emitted even when recall is not configured. |
+| `experience.incubation` | One Experience incubation run for a scheduled activation. |
 | `context.build` | The synchronous step that selects and renders the final prepared context from recalled candidates. |
 
 The other PowerContext generation tasks appear under the same convention: `experience_incubation`,
@@ -97,6 +100,23 @@ Spans are exported in batches, so allow a few seconds before refreshing. An MCP
 `MCP mcp.tools.call` in place of the `HTTP` span. Readiness probes are deliberately not traced, so health checks do not
 create single-span traces.
 
+## Scheduled background spans
+
+When a scheduler interval is configured (`schedule_seconds` or `experience_schedule_seconds`), each scheduled activation
+starts its own trace instead of joining an unrelated request trace. The activation is the root span with
+`powercontext.operation.unit` set to `background`:
+
+| Span | Meaning |
+| --- | --- |
+| `scheduled.process_source_window` | One scheduled Source-window activation. Its outcome is `success`, `noop`, `failure`, or `cancelled`. |
+| `scheduled.incubate_experience_candidates` | One scheduled Experience incubation activation, with the same outcome vocabulary. |
+| `memory.flush` | The flush run beneath a Source-window activation; it also appears under `HTTP flush_memory`. |
+| `experience.incubation` | The incubation run beneath an Experience activation. |
+
+Scheduled roots record only bounded counts — `powercontext.background.source_count` and
+`powercontext.background.candidate_count` — and never a `scope_id`, request ID, or Memory content. Inference spans created
+by a scheduled activation are nested beneath its root in the same trace.
+
 ## What is not exported
 
 PowerContext configures inference instrumentation to exclude content. Spans carry model identifiers, token usage,

docs/en/rfcs/0046_observability_foundations.md

@@ -87,6 +87,9 @@ PowerContext distinguishes these units:
 Runtime stage spans use `stage` as their `powercontext.operation.unit` value. They expose internal latency without
 creating another application operation.
 
+Background activations use `background` as their `powercontext.operation.unit` value. Scheduled activations start their
+own trace rather than inheriting an inbound request context.
+
 A direct HTTP call produces one external request and one application operation. An MCP tool call also produces one
 external request and one application operation. Its internal HTTP bridge does not count as a second external request.
 

docs/zh/docs/how-to/trace-with-phoenix.md

@@ -66,12 +66,13 @@ Memory extraction 发生在 flush 阶段，而不是捕获阶段。
 ## 查看 trace
 
 打开 <http://localhost:6006>，选择 `default` project，打开 `powercontext-server` 最新的一条 trace。这次 flush
-在同一条 trace 中产生四层嵌套 span：
+在同一条 trace 中产生五层嵌套 span：
 
 | Span | 含义 |
 | --- | --- |
 | `HTTP flush_memory` | 入站 HTTP 请求。`powercontext.request.id` 与响应头 `X-PowerContext-Request-ID` 一致。 |
 | `powercontext flush_memory` | application 操作，与调用它的 transport 无关。 |
+| `memory.flush` | 实际处理 Source window 的 Runtime stage。extraction 跑起来时，推理 span 嵌套在它下面。 |
 | `invoke_agent memory_extraction` | 一次 PowerContext generation 任务。名字标识用途，不是模型名。 |
 | `chat <model>` | 一次发往模型 provider 的请求，包含 token 用量和耗时。 |
 
@@ -82,9 +83,11 @@ scope 相关操作还会在 application operation 之下添加以下内部 stage
 | --- | --- |
 | `scope.context` | 从配置的 provider 解析该 scope 的 context；内建 provider 下接近零，provider 在此做 I/O 时才可见。 |
 | `scope.lock` | 等待该 scope 的写锁，在获取到锁的瞬间结束。`powercontext.scope.lock.contended` 表示进入时是否已被其他操作持有。 |
+| `memory.flush` | 一次 Source-window flush，出现在 `flush_memory` 或定时激活之下。 |
 | `memory.search` | `search_memory` 或 `prepare_context` 中的 Memory 查询；存在 embedding 或 reranking span 时，它们嵌套在其下。 |
 | `memory.rerank` | 一次实际 reranker 调用；使用模型的 reranking 会在其下嵌套 `invoke_agent memory_rerank`。 |
 | `experience.search` | `prepare_context` 中的 Experience recall；未配置 recall 时也会产生。 |
+| `experience.incubation` | 一次 Experience incubation，出现在定时激活之下。 |
 | `context.build` | 根据召回候选同步选择并渲染最终 prepared context 的步骤。 |
 
 其他 generation 任务遵循同样的命名约定：`experience_incubation`、`experience_generation`、`skill_generation`、
@@ -94,6 +97,22 @@ span 挂在触发它的操作之下。
 span 是批量导出的，刷新前请稍等几秒。MCP 请求会用 `MCP mcp.tools.call` 取代 `HTTP` span。readiness 探活被有意
 排除在 trace 之外，因此健康检查不会产生只含单个 span 的 trace。
 
+## 定时后台 span
+
+配置了定时器间隔（`schedule_seconds` 或 `experience_schedule_seconds`）后，每次定时激活都会开启一条**独立**的
+trace，而不会并入无关的请求 trace。激活本身是根 span，其 `powercontext.operation.unit` 为 `background`：
+
+| Span | 含义 |
+| --- | --- |
+| `scheduled.process_source_window` | 一次定时 Source-window 激活，outcome 为 `success`、`noop`、`failure` 或 `cancelled`。 |
+| `scheduled.incubate_experience_candidates` | 一次定时 Experience incubation 激活，outcome 词表相同。 |
+| `memory.flush` | Source-window 激活下的 flush 执行；它也会出现在 `HTTP flush_memory` 之下。 |
+| `experience.incubation` | Experience 激活下的 incubation 执行。 |
+
+定时根 span 只记录有界的计数——`powercontext.background.source_count` 与
+`powercontext.background.candidate_count`——绝不携带 `scope_id`、request ID 或 Memory 内容。定时激活产生的推理
+span 会作为其根 span 的子节点嵌套在同一条 trace 中。
+
 ## 哪些内容不会被导出
 
 PowerContext 在配置推理 instrumentation 时关闭了内容记录。span 只携带模型标识、token 用量、耗时和错误类别；

docs/zh/rfcs/0046_observability_foundations.md

@@ -84,6 +84,9 @@ PowerContext 区分以下工作单元：
 Runtime stage span 的 `powercontext.operation.unit` 值为 `stage`。它们用于展示内部耗时，但不会产生新的
 application operation。
 
+Background activation 的 `powercontext.operation.unit` 值为 `background`。定时 activation 会开启独立 trace，而不是
+继承入站请求上下文。
+
 直接 HTTP call 会产生一次 external request 和一次 application operation。MCP tool call 同样产生一次
 external request 和一次 application operation。其内部 HTTP bridge 不计为第二次 external request。
 

src/powercontext/builtin/runtime/application.py

@@ -492,7 +492,15 @@ async def incubate(self, /, *, limit: int | None = None) -> ExperienceIncubation
             ),
             self._runtime._locked(self.scope_id),
         ):
-            return await incubator(self.scope_id, window_limit)
+            with self._runtime._stage("experience.incubation", attributes={}) as span:
+                result = await incubator(self.scope_id, window_limit)
+                if span is not None:
+                    span.set_attributes({
+                        "powercontext.experience.incubation.source_count": result.source_count,
+                        "powercontext.experience.incubation.candidate_count": result.candidate_count,
+                    })
+                    span.set_outcome("success" if result.processed else "noop")
+                return result
 
 
 class ExperienceApplication:
@@ -1042,7 +1050,12 @@ async def flush(self, /, *, limit: int | None = None) -> MemoryFlushResult:
         ) as context:
             window_limit = self._runtime.source_window_limit if limit is None else limit
             async with self._runtime._locked(self.scope_id):
-                return await context.triggers.flush(limit=window_limit)
+                with self._runtime._stage("memory.flush", attributes={}) as span:
+                    result = await context.triggers.flush(limit=window_limit)
+                    if span is not None:
+                        span.set_attributes({"powercontext.memory.flush.source_count": result.source_count})
+                        span.set_outcome("success" if result.processed else "noop")
+                    return result
 
     async def cursor(self) -> SourceCursor:
         async with self._runtime._context(self.scope_id) as context:
@@ -1074,29 +1087,39 @@ async def run(self) -> None:
                 if self._runtime._closing or self._runtime._closed:
                     return
                 started_at = perf_counter()
-                try:
-                    result = await self._runtime.memory.for_scope(scope_id).flush()
-                except asyncio.CancelledError:
-                    _log_scheduled_processing(
-                        "cancelled",
-                        operation="process_source_window",
-                        started_at=started_at,
-                    )
-                    raise
-                except Exception as error:
-                    _log_scheduled_processing(
-                        "failure",
-                        operation="process_source_window",
-                        started_at=started_at,
-                        error=error,
-                    )
-                else:
-                    _log_scheduled_processing(
-                        "success" if result.processed else "noop",
-                        operation="process_source_window",
-                        started_at=started_at,
-                        source_count=result.source_count,
-                    )
+                with self._runtime._background(
+                    "scheduled.process_source_window",
+                    operation="process_source_window",
+                ) as span:
+                    try:
+                        result = await self._runtime.memory.for_scope(scope_id).flush()
+                    except asyncio.CancelledError:
+                        _log_scheduled_processing(
+                            "cancelled",
+                            operation="process_source_window",
+                            started_at=started_at,
+                        )
+                        raise
+                    except Exception as error:
+                        _log_scheduled_processing(
+                            "failure",
+                            operation="process_source_window",
+                            started_at=started_at,
+                            error=error,
+                        )
+                        if span is not None:
+                            span.set_outcome("failure")
+                    else:
+                        outcome = "success" if result.processed else "noop"
+                        _log_scheduled_processing(
+                            outcome,
+                            operation="process_source_window",
+                            started_at=started_at,
+                            source_count=result.source_count,
+                        )
+                        if span is not None:
+                            span.set_outcome(outcome)
+                            span.set_attributes({"powercontext.background.source_count": result.source_count})
 
 
 class ScheduledExperienceProcessor:
@@ -1114,30 +1137,43 @@ async def run(self) -> None:
                 if self._runtime._closing or self._runtime._closed:
                     return
                 started_at = perf_counter()
-                try:
-                    result = await self._runtime.experience.for_scope(scope_id).incubate()
-                except asyncio.CancelledError:
-                    _log_scheduled_processing(
-                        "cancelled",
-                        operation="incubate_experience_candidates",
-                        started_at=started_at,
-                    )
-                    raise
-                except Exception as error:
-                    _log_scheduled_processing(
-                        "failure",
-                        operation="incubate_experience_candidates",
-                        started_at=started_at,
-                        error=error,
-                    )
-                else:
-                    _log_scheduled_processing(
-                        "success" if result.processed else "noop",
-                        operation="incubate_experience_candidates",
-                        started_at=started_at,
-                        source_count=result.source_count,
-                        candidate_count=result.candidate_count,
-                    )
+                with self._runtime._background(
+                    "scheduled.incubate_experience_candidates",
+                    operation="incubate_experience_candidates",
+                ) as span:
+                    try:
+                        result = await self._runtime.experience.for_scope(scope_id).incubate()
+                    except asyncio.CancelledError:
+                        _log_scheduled_processing(
+                            "cancelled",
+                            operation="incubate_experience_candidates",
+                            started_at=started_at,
+                        )
+                        raise
+                    except Exception as error:
+                        _log_scheduled_processing(
+                            "failure",
+                            operation="incubate_experience_candidates",
+                            started_at=started_at,
+                            error=error,
+                        )
+                        if span is not None:
+                            span.set_outcome("failure")
+                    else:
+                        outcome = "success" if result.processed else "noop"
+                        _log_scheduled_processing(
+                            outcome,
+                            operation="incubate_experience_candidates",
+                            started_at=started_at,
+                            source_count=result.source_count,
+                            candidate_count=result.candidate_count,
+                        )
+                        if span is not None:
+                            span.set_outcome(outcome)
+                            span.set_attributes({
+                                "powercontext.background.source_count": result.source_count,
+                                "powercontext.background.candidate_count": result.candidate_count,
+                            })
 
 
 def _log_scheduled_processing(
@@ -1451,6 +1487,16 @@ def _stage(
             return nullcontext(None)
         return self._tracing.stage(name, attributes=attributes)
 
+    def _background(
+        self,
+        name: str,
+        *,
+        operation: str,
+    ) -> AbstractContextManager[RuntimeSpan | None]:
+        if self._tracing is None:
+            return nullcontext(None)
+        return self._tracing.background(name, operation=operation, attributes={})
+
     def _review(self, scope_id: str) -> ReviewService:
         if self._review_service is None:
             raise _RuntimeStateError("review")

src/powercontext/builtin/runtime/protocols.py

@@ -32,10 +32,12 @@
 
 
 class RuntimeSpan(Protocol):
-    """Record bounded attributes for one internal Runtime stage."""
+    """Record bounded attributes and a deferred outcome for one internal Runtime stage."""
 
     def set_attributes(self, attributes: Mapping[str, TraceAttribute], /) -> None: ...
 
+    def set_outcome(self, outcome: str, /) -> None: ...
+
 
 class RuntimeTracing(Protocol):
     """Create framework-neutral spans for internal Runtime stages."""
@@ -47,6 +49,14 @@ def stage(
         attributes: Mapping[str, TraceAttribute],
     ) -> AbstractContextManager[RuntimeSpan]: ...
 
+    def background(
+        self,
+        name: str,
+        *,
+        operation: str,
+        attributes: Mapping[str, TraceAttribute],
+    ) -> AbstractContextManager[RuntimeSpan]: ...
+
 
 class PowerContextProvider(Protocol[SourcesT, ArtifactsT, TriggersT]):
     """Resolve an already composed context without transferring lifecycle ownership."""

src/powercontext/server/tracing.py

@@ -138,7 +138,47 @@ def stage(
         except BaseException as error:
             span.finish("failure", error=error)
             raise
-        span.finish("success")
+        span.finish(span.outcome or "success")
+
+    @contextmanager
+    def background(
+        self,
+        name: str,
+        *,
+        operation: str,
+        attributes: Mapping[str, _TraceAttribute],
+    ) -> Iterator[_ActiveSpan]:
+        """Trace one scheduled activation as an independent trace root."""
+
+        isolation_token: Token[Context] | None = None
+        with suppress(Exception):
+            # Keep an empty context attached even if the root span fails to start, so
+            # child stages cannot join an ambient HTTP/MCP trace.
+            isolation_token = otel_context.attach(Context())
+        try:
+            span = self.start_span(
+                name,
+                kind=SpanKind.INTERNAL,
+                context=Context(),
+                attributes={
+                    **attributes,
+                    "powercontext.operation.name": operation,
+                    "powercontext.operation.unit": "background",
+                },
+            )
+            try:
+                yield span
+            except asyncio.CancelledError as error:
+                span.finish("cancelled", error=error)
+                raise
+            except BaseException as error:
+                span.finish("failure", error=error)
+                raise
+            span.finish(span.outcome or "success")
+        finally:
+            if isolation_token is not None:
+                with suppress(Exception):
+                    otel_context.detach(isolation_token)
 
     @contextmanager
     def _suppress_readiness_spans(self) -> Iterator[None]:
@@ -178,6 +218,14 @@ def __init__(self, span: Span | None, token: Token[Context] | None) -> None:
         self.span = span
         self.token = token
         self.finished = False
+        self._outcome: str | None = None
+
+    @property
+    def outcome(self) -> str | None:
+        return self._outcome
+
+    def set_outcome(self, outcome: str) -> None:
+        self._outcome = outcome
 
     @classmethod
     def start(