Add launcher type and cluster URL to root execution span#255
Add launcher type and cluster URL to root execution span#255morgan-wowk wants to merge 1 commit into
Conversation
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
morgan-wowk
force-pushed
the
execution-tracing-launcher-type
branch
2 times, most recently
from
57ba606 to
54e0001
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-data
branch
from
3aa355d to
3e85d85
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-type
branch
from
54e0001 to
6f8a30a
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-data
branch
from
3e85d85 to
239a70c
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-type
branch
from
6f8a30a to
a8e9b03
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-data
branch
from
239a70c to
6521dc7
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-type
branch
from
a8e9b03 to
5a111d0
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-data
branch
from
6521dc7 to
374f9cf
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-type
branch
from
5a111d0 to
3baf255
Compare
yuechao-qin
left a comment
yuechao-qin
left a comment
There was a problem hiding this comment.
Doesn't this PR also distinguish between Nebius or not? Can you then filter on
cloud_provider = Nebius && status = <SOMETHING, Maybe Failures>?
|
|
||
|
|
||
| class TestLauncherTypeAttrs: | ||
| def test_root_span_carries_launcher_type( |
There was a problem hiding this comment.
This is not testing for launcher type?
There was a problem hiding this comment.
🤖 Added — test_root_span_carries_launcher_type_and_cluster_url now sets up a ContainerExecution with kubernetes launcher_data and asserts execution.launcher == "kubernetes" and k8s.cluster.url on the root span.
morgan-wowk
force-pushed
the
execution-tracing-launcher-type
branch
from
3baf255 to
d7605f3
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-data
branch
from
374f9cf to
22df75e
Compare
|
🤖 Exactly — with this PR you can filter traces on |
execution.launcher = top-level key from launcher_data (e.g. 'kubernetes', 'kubernetes_job', 'skypilot') distinguishes launcher mechanism. k8s.cluster.url on root span allows GKE vs Nebius cluster identification by URL pattern in oasis-backend's multi-cloud setup.
morgan-wowk
force-pushed
the
execution-tracing-launcher-data
branch
from
22df75e to
b7aa164
Compare
morgan-wowk
force-pushed
the
execution-tracing-launcher-type
branch
from
d7605f3 to
4871348
Compare
| def test_root_span_omits_launcher_without_container_execution( | ||
| self, span_exporter: InMemorySpanExporter | ||
| ) -> None: | ||
| execution = _make_execution(statuses=["QUEUED", "SUCCEEDED"]) | ||
| execution_tracing.emit_execution_trace(execution=execution) | ||
|
|
||
| root = next( | ||
| s for s in span_exporter.get_finished_spans() if s.name == "execution" | ||
| ) | ||
| assert "execution.launcher" not in (root.attributes or {}) | ||
|
|
||
| def test_root_span_no_launcher_attrs_without_container_execution( |
There was a problem hiding this comment.
Is my understanding correct that these K8 attr will only be available when the span has ended (has terminal state)?
What's the difference between
- test_root_span_omits_launcher_without_container_execution
- test_root_span_no_launcher_attrs_without_container_execution
Just testing two different terminal states?
2 participants
Add
execution.launcherandexecution.cloud_providerOTel trace attributes to root execution spanexecution.launcheris derived from the top-level key oflauncher_data(e.g.kubernetes,kubernetes_job,skypilot) and distinguishes the launcher mechanism used for a given execution.k8s.cluster.urlon the root span allows GKE vs Nebius cluster identification by URL pattern in oasis-backend's multi-cloud setup, populated fromcluster_serverinside the launcher's data block.execution.cloud_provideris read from thecloud-pipelines.net/orchestration/cloud_providertask_spec annotation, set at routing time by callers such asMultiLauncherContainerLauncher. This enables traces to be searched by cloud provider (e.g.gke,nebius) without relying on URL pattern matching against cluster hostnames. Launchers with a fixed cloud affinity can also set this annotation directly.The
CLOUD_PROVIDER_ANNOTATION_KEYconstant is defined incommon_annotationsso it can be shared across launchers and the tracing layer without duplication.Screenshots