Enrich error/success status spans with failure context

[morgan-wowk]

Enrich execution trace spans with error details and root span status

FAILED span: Attaches error.message from orchestration_error_message in extra_data and execution.exit_code from the associated ContainerExecution when available.

SYSTEM_ERROR span: Attaches error.message from the system error exception message and exception.stacktrace from the full exception traceback stored in extra_data.

SUCCEEDED span: Attaches execution.exit_code from the associated ContainerExecution when available.

Root span: Calls set_status(ERROR) when the terminal status is FAILED or SYSTEM_ERROR, leaving the root span unmarked for successful executions.

Screenshots

[morgan-wowk]

• Add CPU, memory, and accelerator resource requests to PENDING span #258
• Add pipeline/parent context to root execution span #257
• Add cache hit/miss context to root execution span #256
• Add launcher type and cluster URL to root execution span #255
• Add k8s pod/cluster attributes to PENDING status span #254
• Enrich error/success status spans with failure context #253 👈 (View in Graphite)
• Add retroactive OTel execution lifecycle tracing #252
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[yuechao-qin]

FYI

_ERROR_TERMINAL_STATUSES = frozenset({
    bts.ContainerExecutionStatus.FAILED,
    bts.ContainerExecutionStatus.SYSTEM_ERROR,
})

[morgan-wowk]

🤖 Adopted — switched to frozenset({...}) with the enum members directly. Simpler since ContainerExecutionStatus is a str enum.

[yuechao-qin]

Curious, if you don't set the Status, assuming it's "successful" by default?

[morgan-wowk]

🤖 Correct — OTel default status is UNSET which means "no error". We only explicitly set ERROR for failed/system_error executions.

[yuechao-qin]

Wondering, would

• exception.message
• exception.stacktrace
  Make more sense since that aligns with the SQL keys, or is "error" not from an "exception"?

[morgan-wowk]

🤖 Good call — renamed to exception.message to align with OTel semantic conventions.

[yuechao-qin]

I saw you using walrus operator in the upstream PR and it got me thinking

def _error_attrs(*, execution: bts.ExecutionNode, status: str) -> dict[str, object]:

    """Extra attributes for terminal error/success status spans."""
    extra = execution.extra_data or {}
    attrs: dict[str, object] = {}

    def _set_exit_code() -> None:
        if (ce := execution.container_execution) and ce.exit_code is not None:
            attrs["execution.exit_code"] = ce.exit_code

    if status == bts.ContainerExecutionStatus.FAILED:
        msg = extra.get(bts.EXECUTION_NODE_EXTRA_DATA_ORCHESTRATION_ERROR_MESSAGE_KEY)
        if msg is not None:
            attrs["error.message"] = msg
        _set_exit_code()
    elif status == bts.ContainerExecutionStatus.SYSTEM_ERROR:
        msg = extra.get(
            bts.EXECUTION_NODE_EXTRA_DATA_SYSTEM_ERROR_EXCEPTION_MESSAGE_KEY
        )
        if msg is not None:
            attrs["error.message"] = msg
        tb = extra.get(bts.EXECUTION_NODE_EXTRA_DATA_SYSTEM_ERROR_EXCEPTION_FULL_KEY)
        if tb is not None:
            attrs["exception.stacktrace"] = tb
    elif status == bts.ContainerExecutionStatus.SUCCEEDED:
        _set_exit_code()
        
    return attrs

[morgan-wowk]

🤖 Adopted this — cleaner with the walrus operator and _set_exit_code helper.

[morgan-wowk]

PS: This entire stack is written by AI with supervision. To make your suggestions permanent adoptions, please add to the CLAUDE.md.