Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ http = "1"
httpdate = "1"
jsonschema = { version = "0.49.4", default-features = false }
jsonptr = { version = "0.8.1", default-features = false, features = ["std", "json", "resolve"] }
nemo-relay-plugin = "0.10.0"
nemo-relay-plugin = "0.10.1"
parking_lot = "0.12"
prefill-router = { path = "crates/prefill-router", version = "0.3.0" }
rand = "0.10"
Expand Down
9 changes: 5 additions & 4 deletions crates/switchyard-nemo-relay-plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,14 @@ algorithm construction, retry policy, and route validation.

## Install

The plugin uses `nemo-relay-plugin` 0.10.0 and requires NeMo Relay
`>=0.10.0, <1.0.0`. Relay 0.10 changes the native callback layout, so older
hosts cannot load this build. The manifest's native plugin API label stays `1`.
The plugin uses `nemo-relay-plugin` 0.10.1 and requires NeMo Relay
`>=0.10.1, <1.0.0`. Relay 0.10 changed the native callback layout; 0.10.1 adds
the HTTP-error status handling used by this build. The manifest's native plugin
API label stays `1`.

Relay 0.8.x and 0.9.0 can lose upstream error status and details when this native
plugin is enabled, including for models outside its configured routes. This is
a known issue on those older hosts. This build requires Relay 0.10 or later.
a known issue on those older hosts. This build requires Relay 0.10.1 or later.
See the [upstream error compatibility note](../../docs/integrations/nemo_relay.md#upstream-error-compatibility).

### Install a released bundle
Expand Down
2 changes: 1 addition & 1 deletion crates/switchyard-nemo-relay-plugin/relay-plugin.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ id = "nvidia.switchyard"
kind = "rust_dynamic"

[compat]
relay = ">=0.10.0, <1.0.0"
relay = ">=0.10.1, <1.0.0"
native_api = "1"

[defaults]
Expand Down
54 changes: 47 additions & 7 deletions crates/switchyard-nemo-relay-plugin/src/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -302,14 +302,16 @@ impl SwitchyardRuntime {
&metadata,
route,
);
self.route_execution_error_mark(
&mut events,
&error.execution_error_summary(),
None,
outcome_fields,
);
let summary = error.execution_error_summary();
self.route_execution_error_mark(&mut events, &summary, None, outcome_fields);
// Preserve the status without exposing the provider's error body.
let message = "Switchyard route execution failed";
let message = summary
.upstream_status
.and_then(|status| nemo_relay_plugin::http_error_message(status, message).ok())
.unwrap_or_else(|| message.to_owned());
Execution {
result: Err("Switchyard route execution failed".into()),
result: Err(message),
events,
}
}
Expand Down Expand Up @@ -955,6 +957,7 @@ mod tests {
"target": {
"format": format,
"base_url": base_url,
"max_retries": 0,
}
},
"targets": {
Expand Down Expand Up @@ -1168,6 +1171,43 @@ seed = 17
assert!(saw_decision);
}

#[tokio::test]
async fn failed_routes_preserve_upstream_status_without_provider_body() {
for status in [401, 429, 503] {
let server = MockServer::start().await;
let private_body = "private provider error details";
Mock::given(method("POST"))
.and(path("/v1/chat/completions"))
.respond_with(ResponseTemplate::new(status).set_body_string(private_body))
.expect(1)
.mount(&server)
.await;
let runtime = runtime_for_target("openai_chat", &format!("{}/v1", server.uri()));
let request = runtime
.decode_request(
WireFormat::OpenAiChat,
RelayRequest {
headers: Map::new(),
content: json!({
"model": "switchyard/default",
"messages": [{"role": "user", "content": "hello"}],
}),
},
false,
)
.unwrap();
let execution = runtime
.execute_buffered(WireFormat::OpenAiChat, request)
.await;
let error = execution.result.unwrap_err();
assert_eq!(
error,
format!("HTTP {status}: Switchyard route execution failed")
);
assert!(!format!("{error} {:?}", execution.events).contains(private_body));
}
}

// Routing succeeds before answer candidates exhaust, so the error keeps its metadata.
#[tokio::test]
async fn failed_answer_keeps_routing_outcome_in_error_mark() {
Expand Down
22 changes: 16 additions & 6 deletions docs/integrations/nemo_relay.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,21 @@ a configured Switchyard route. For example, if the route is `switchyard/core`
and its target is `azure/openai/gpt-5.5`, requesting the target name directly
still delegates the request to Relay.

This plugin build uses `nemo-relay-plugin` 0.10.0 and requires Relay
`>=0.10.0, <1.0.0`. Older hosts cannot load it because Relay 0.10 changes the
native callback layout. If you use an older plugin build on Relay 0.8.x or
0.9.0, send unmanaged traffic through a separate Relay instance with the plugin
disabled. Upgrading Switchyard alone does not upgrade the Relay host.
This plugin build uses `nemo-relay-plugin` 0.10.1 and requires Relay
`>=0.10.1, <1.0.0`. Relay 0.10 changed the native callback layout; 0.10.1 adds
the HTTP-error status handling used by this build. If you use an older plugin
build on Relay 0.8.x or 0.9.0, send unmanaged traffic through a separate Relay
instance with the plugin disabled. Upgrading Switchyard alone does not upgrade
the Relay host.

For managed routes, known upstream HTTP errors retain their status through
Relay's HTTP-error API. The client receives the generic message
`Switchyard route execution failed`; provider error bodies remain private.
Failures without a known HTTP error status keep Relay's existing error handling.

This preserves the HTTP status for buffered responses. Relay's streaming path
can send HTTP 200 before route execution finishes, so later failures cannot
change that status and retain the existing stream-error behavior.

## Why Use Switchyard with NeMo Relay?

Expand Down Expand Up @@ -171,7 +181,7 @@ and how it

!!! note "Relay compatibility"

The plugin requires `relay = ">=0.10.0, <1.0.0"` and native plugin API `1`.
The plugin requires `relay = ">=0.10.1, <1.0.0"` and native plugin API `1`.
The packaged
[`relay-plugin.toml`](../../crates/switchyard-nemo-relay-plugin/relay-plugin.toml)
is the source of truth.
Expand Down
2 changes: 1 addition & 1 deletion tests/relay_plugin/test_package_bundle.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ def test_materializes_and_archives_supported_formats(self) -> None:
self.assertEqual({path.name for path in output.iterdir()}, expected)
manifest = (output / "relay-plugin.toml").read_text(encoding="utf-8")
self.assertIn(f'artifact = "{library.name}"', manifest)
self.assertIn('relay = ">=0.10.0, <1.0.0"', manifest)
self.assertIn('relay = ">=0.10.1, <1.0.0"', manifest)
self.assertIn(hashlib.sha256(library.read_bytes()).hexdigest(), manifest)
self.assertNotIn("<platform-library-file>", manifest)
self.assertNotIn("<artifact-sha256>", manifest)
Expand Down
Loading