feat(plugin): add timestamped scope guards

[afourniernv]

Overview

This is one of two independent Relay changes needed for the Switchyard plugin to replay captured routing scopes and marks beneath the managed LLM execution that triggered them.

Timestamped scopes preserve the calls' real timing. Switchyard captures libsy.client_call and libsy.upstream_attempt spans while provider work is happening, then publishes the completed hierarchy through Relay. If Relay timestamps those scopes when they are replayed, the provider work has already finished and the recorded durations collapse toward zero.

This PR exposes historical scope timestamps consistently across Relay's plugin APIs. It does not change Relay core scope behavior.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Why the existing API is insufficient

The ordinary scope methods preserve hierarchy but use the current time. They cannot supply the original captured start and end times.

Two Switchyard-only approaches were evaluated:

• Emitting scopes live is unsafe when spans overlap because they may close outside Relay's required LIFO order.
• Calling the raw scope_push and scope_pop ABI directly would preserve timing, but would duplicate unsafe pointer handling, JSON conversion, status handling, and scope-handle ownership inside Switchyard. The typed SDK should expose functionality already supported by the ABI.

Details

• Native Rust plugin SDK: add PluginRuntime::scope_at(...) and ScopeGuard::close_at(...) as typed wrappers over timestamp fields already present in the native ABI.
• Rust worker SDK: add push_scope_at(...) and pop_scope_at(...).
• Python worker SDK: add an optional timezone-aware timestamp keyword to push_scope(...) and pop_scope(...), matching Relay's existing Python scope convention.
• Worker protocol: add optional signed Unix-microsecond fields to the existing PushScope and PopScope messages.
• Host runtime: validate worker timestamps before opening or consuming a scope handle, then publish the supplied times on the real scope events.
• Leave all ordinary scope methods unchanged; omitted timestamps retain current-time behavior.

This is additive. No existing method, callback, serialized field number, native function table, field layout, or ABI version changes. The worker protocol remains grpc-v1; older protobuf hosts ignore the optional fields. A worker that depends on historical timestamp preservation should require Relay 0.10 or newer.

Relationship and landing order

• This PR and #1167 are independent and non-breaking; either can land first.
• This PR preserves original timing. fix(plugin): isolate async callback scope context #1167 preserves correct parentage when native or worker callbacks overlap.
• The Switchyard consumer should land after both Relay changes are available.

Testing

• Native plugin SDK unit and historical-scope ownership tests.
• Worker protobuf compatibility and presence tests, including explicit epoch and pre-epoch values.
• Rust worker SDK request tests and out-of-range rejection before RPC.
• Python worker SDK ordinary, epoch, pre-epoch, naive-datetime, and invalid-type tests.
• Host runtime round trip proving the final exported start/end events retain the supplied historical timestamps and an invalid pop timestamp does not consume the handle.
• Full worker host, worker integration, Python plugin, formatting, and warnings-as-errors checks.

Where should the reviewer start?

Start with the native methods in crates/plugin/src/lib.rs, then the worker protocol and host translation in:

• crates/worker-proto/proto/nemo/relay/worker/v1/plugin_worker.proto
• crates/core/src/plugin/dynamic/worker.rs
• crates/worker/src/lib.rs
• python/plugin/src/nemo_relay_plugin/_api.py

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to Export existing Switchyard routing spans as nested Relay scopes NVIDIA-NeMo/Switchyard#871.
• Companion callback-context draft: fix(plugin): isolate async callback scope context #1167
• Switchyard consumer branch: https://github.com/afourniernv/Switchyard/tree/codex/relay-nested-routing-spans

Summary by CodeRabbit

• New Features
  • Scope operations can now use caller-specified start and end times, supporting replay with original timestamps in Rust and Python.
  • Python scope timestamps must include a timezone; timestamps are transmitted in Unix microseconds. Calls without a timestamp continue using host time.
• Documentation
  • Updated runtime and protocol guides with timestamp usage, behavior, and compatibility details.
• Tests
  • Added coverage for timestamp transmission, pre-epoch values, invalid inputs, and out-of-range timestamps.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/NeMo-Relay/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 5e9747c5-093d-4350-a351-d8a56961f2f6

📥 Commits

Reviewing files that changed from the base of the PR and between 2e0e433 and c395285.

📒 Files selected for processing (14)

• crates/core/src/plugin/dynamic/worker.rs
• crates/core/tests/unit/dynamic_worker_tests.rs
• crates/plugin/src/lib.rs
• crates/plugin/tests/typed_callbacks.rs
• crates/worker-proto/proto/nemo/relay/worker/v1/plugin_worker.proto
• crates/worker-proto/tests/proto_tests.rs
• crates/worker/src/lib.rs
• crates/worker/tests/unit/timestamp_tests.rs
• crates/worker/tests/worker_sdk_tests.rs
• docs/build-plugins/native/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/grpc-v1-protocol.mdx
• docs/build-plugins/workers/runtime-events-and-scopes.mdx
• python/plugin/src/nemo_relay_plugin/_api.py
• python/tests/plugin/test_worker_sdk.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📜 Recent review details

⏰ Context from checks skipped due to timeout. (3)

• GitHub Check: request / require-nvskills-ci / require-nvskills-ci
• GitHub Check: Detect docs changes
• GitHub Check: Apply PR labels

🧰 Additional context used

📓 Path-based instructions (7)

Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.

⚙️ CodeRabbit configuration file

Files:

• docs/build-plugins/native/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/grpc-v1-protocol.mdx

Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.

⚙️ CodeRabbit configuration file

Files:

• crates/worker/tests/unit/timestamp_tests.rs
• crates/worker/tests/worker_sdk_tests.rs
• crates/worker-proto/tests/proto_tests.rs
• python/tests/plugin/test_worker_sdk.py
• crates/core/tests/unit/dynamic_worker_tests.rs
• crates/plugin/tests/typed_callbacks.rs

Review the Rust runtime for async correctness, scope isolation, middleware ordering, and event lifecycle regressions.

⚙️ CodeRabbit configuration file

Files:

• crates/core/tests/unit/dynamic_worker_tests.rs
• crates/core/src/plugin/dynamic/worker.rs

Source excerpt: In MDX files, top-of-file comments must use JSX comment delimiters: `{/*` to open and `*/}` to close.

📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)

Files:

• docs/build-plugins/native/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/grpc-v1-protocol.mdx

Source excerpt: Verify MDX files use JSX delimiters for top-of-file SPDX comments.

📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)

Files:

• docs/build-plugins/native/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/grpc-v1-protocol.mdx

Source excerpt: Search documentation source for references to the old version and update current-version install commands, package examples, and configuration examples to `` where appropriate: Review matches before changing th...

📄 CodeRabbit inference engine (.agents/skills/prepare-code-freeze/SKILL.md)

Files:

• docs/build-plugins/native/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/grpc-v1-protocol.mdx

Source excerpt: Preserve MDX front matter and the JSX SPDX comment.

📄 CodeRabbit inference engine (.agents/skills/draft-release-notes/SKILL.md)

Files:

• docs/build-plugins/native/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/runtime-events-and-scopes.mdx
• docs/build-plugins/workers/grpc-v1-protocol.mdx

🧠 Learnings (1)

📚 Learning: 2026-08-12T17:13:14.808Z

Learnt from: SandyChapman
Repo: NVIDIA/NeMo-Relay PR: 755
File: python/tests/integrations/langchain_tests/test_callbacks_scope_stack.py:185-185
Timestamp: 2026-08-12T17:13:14.808Z
Learning: In Python files, do not report Ruff UP017 findings unless pyproject.toml enables the UP rule set or the individual file explicitly enables UP017. The repository currently enables Ruff rule sets E, F, W, and I only.

Applied to files:

• python/tests/plugin/test_worker_sdk.py

🔇 Additional comments (14)

crates/plugin/src/lib.rs (1)

21-21: LGTM!

Also applies to: 2098-2127, 2249-2250, 2277-2302, 2621-2632, 2649-2649, 2666-2675, 2684-2684, 2694-2709

docs/build-plugins/native/runtime-events-and-scopes.mdx (1)

82-84: LGTM!

crates/plugin/tests/typed_callbacks.rs (1)

20-20: LGTM!

Also applies to: 448-448, 1478-1478, 1509-1511, 1524-1524, 1546-1548, 3198-3198, 3562-3687

python/plugin/src/nemo_relay_plugin/_api.py (2)

2903-2912: Range-check the timestamp before assigning it to the proto field.

_datetime_to_unix_micros does not check that its result fits in int64. For example, datetime.max with UTC is about 2.5e17 µs and fits. But a datetime with a large positive offset cannot exceed int64 either, so Python cannot overflow here. Python's own datetime range already bounds the result. The host then rejects values outside chrono's range. No change is required.

---

2022-2035: LGTM!

Also applies to: 2064-2074

crates/worker-proto/proto/nemo/relay/worker/v1/plugin_worker.proto (1)

448-448: LGTM!

Also applies to: 462-462

crates/worker-proto/tests/proto_tests.rs (1)

292-323: LGTM!

docs/build-plugins/workers/grpc-v1-protocol.mdx (1)

284-285: LGTM!

Also applies to: 302-303, 317-322

crates/worker/src/lib.rs (1)

1634-1652: LGTM!

crates/worker/tests/unit/timestamp_tests.rs (1)

1-49: LGTM!

crates/worker/tests/worker_sdk_tests.rs (1)

1086-1104: LGTM!

python/tests/plugin/test_worker_sdk.py (1)

2537-2555: LGTM!

Also applies to: 2875-2898

docs/build-plugins/workers/runtime-events-and-scopes.mdx (1)

21-21: LGTM!

Also applies to: 31-31

crates/core/src/plugin/dynamic/worker.rs (1)

3356-3356: LGTM!

Also applies to: 3398-3401, 3417-3417, 3941-3951

---

Walkthrough

The change adds optional historical timestamps to Rust and Python scope APIs and worker protocol requests. The worker runtime converts supplied Unix-microsecond values for scope events. Untimestamped calls continue to omit the timestamp.

Changes

Historical scope timestamps

Layer / File(s)	Summary
Add optional timestamps to scope requests crates/worker-proto/proto/nemo/relay/worker/v1/plugin_worker.proto, crates/worker-proto/tests/proto_tests.rs, docs/build-plugins/workers/grpc-v1-protocol.mdx	Push and pop requests add optional timestamp fields. Protocol tests cover absent and present values. The protocol documentation describes timestamp semantics and compatibility.
Serialize timestamps in worker SDKs crates/worker/src/lib.rs, crates/worker/tests/*, python/plugin/src/nemo_relay_plugin/_api.py, python/tests/plugin/test_worker_sdk.py, docs/build-plugins/workers/runtime-events-and-scopes.mdx	Rust adds push_scope_at and pop_scope_at. Python adds optional timezone-aware timestamps to push_scope and pop_scope. Both APIs convert timestamps to Unix microseconds; tests cover serialization, omission, and invalid values.
Add timestamped native plugin scopes crates/plugin/src/lib.rs, crates/plugin/tests/typed_callbacks.rs, docs/build-plugins/native/runtime-events-and-scopes.mdx	The native plugin API adds scope_at and ScopeGuard::close_at. These methods pass converted timestamps to host callbacks. Tests cover conversion, failed closes, and handle retention.
Convert request timestamps into scope events crates/core/src/plugin/dynamic/worker.rs, crates/core/tests/unit/dynamic_worker_tests.rs	The dynamic worker converts optional request timestamps to UTC datetimes. An invalid pop timestamp returns an error acknowledgment before the worker looks up or removes the scope handle. Tests verify emitted event times and handle retention after a failed pop.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant PluginRuntime
  participant PushScopeRequest
  participant DynamicWorker
  participant ScopeEvents
  PluginRuntime->>PushScopeRequest: Set optional timestamp_unix_micros
  PushScopeRequest->>DynamicWorker: Send scope request
  DynamicWorker->>ScopeEvents: Emit scope event with converted UTC timestamp

Loading

Merge Risk: ⚪ Minimal · up to c3952

The timestamp additions preserve ordinary scope behavior, and no actionable issue remains. This change is mergeable subject to normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 48.21% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 56 functions across 10 files. (4 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title uses the required Conventional Commits format, uses the allowed lowercase type and scope, states the main change clearly, contains 42 characters, and has no trailing period.
Description check	✅ Passed	The description includes the required Overview, Details, reviewer-start, and Related Issues sections. It provides clear rationale, implementation details, testing scope, and uses the required Relates …

Full details: Docstring Coverage

Explanation

Docstring coverage is 48.21% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 56 functions across 10 files. (4 skipped: 4 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{Comment @coderabbitai help to get the list of available commands.}

[willkill07]

Any changes made here also have to be made to:

• nemo-relay-worker
• nemo-relay-plugin (Python)

We need total consistency across all Plugin APIs.

[github-actions]

License Diff

Compared against origin/main.

Lockfile license changes

Lockfile License Changes

Rust

Added

• None

Removed

• None

Updated/Changed

• None

Node

Added

• None

Removed

• None

Updated/Changed

• None

Python

Added

• None

Removed

• None

Updated/Changed

• None

Status output

[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (469 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (424 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (115 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/main into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (469 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (424 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (115 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-1166.docs.buildwithfern.com/nemo/relay/dev

[afourniernv]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Already reviewed the last commit. Use @coderabbitai full review to rerun a review of the entire changeset.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[mnajafian-nv]

LGTM, no p0s or p1s

[mnajafian-nv]

/merge