draft: Go parametric gen_ai OTLP LLM Obs contract (hermetic schema assertion) by khanayan123 · Pull Request #7314 · DataDog/system-tests

khanayan123 · 2026-07-15T15:34:58Z

What

Reorients this PR from the native dd-trace-go llmobs SDK path to the OpenTelemetry gen_ai OTLP path, per the authoritative schema in Datadog's LLM Obs OTel instrumentation docs. Related dd-trace-go export PRs: #4936 (llmobs/export), #4994 (otlp/export).

Previously this wired /llm_observability/* into the native llmobs SDK and enabled the tests/parametric/test_llm_observability/ suite (reading LLM Obs span-events off the test agent). That native enablement is reverted. Instead:

/llm_observability/trace keeps the same SpanRequest-tree input, but each node is mapped to gen_ai.* attributes and emitted as OTel spans.
Spans go through the app's existing ddotel bridge provider and are exported as OTLP/JSON by dd-trace-go's own OTLP trace exporter (OTEL_TRACES_EXPORTER=otlp + OTEL_EXPORTER_OTLP_TRACES_*) — the same transport test_otlp_trace_metrics.py uses, so no new Go module (compiles against existing parametric/go.mod deps).
The export carries dd-otlp-source=llmobs + dd-ml-app (via OTEL_EXPORTER_OTLP_TRACES_HEADERS).

Scope of assertion (hermetic — "just assert the schema")

A new pytest drives the SpanRequest tree and asserts the captured OTLP payload carries the correct gen_ai.* attributes (operation.name, request.model, provider.name, conversation.id, usage.*_tokens, input/output.messages, _dd.ml_obs.metadata) + the dd-otlp-source=llmobs / dd-ml-app headers + service.name. It reads raw /v1/traces off the ddapm-test-agent OTLP receiver. No backend read-back, no claim about Datadog's internal gen_ai→LLM-Obs mapping.

gen_ai mapping (per node)

kind → gen_ai.operation.name: llm→chat, embedding→embeddings, tool→execute_tool, agent→invoke_agent; workflow/task/retrieval omit it (backend default = workflow).
model_name→gen_ai.request.model; model_provider→gen_ai.provider.name; session_id→gen_ai.conversation.id.
annotation input/output → gen_ai.input.messages / gen_ai.output.messages (JSON); metadata → _dd.ml_obs.metadata (JSON); token metrics → gen_ai.usage.*_tokens.

Both transports (direct intake otlp.<site> vs. via a collector) use the same schema; HTTP/JSON is pinned so the captured body is JSON-decodable.

Files

utils/build/docker/golang/parametric/llmobs.go — rewritten: SpanRequest tree → gen_ai OTel spans (native llmobs/dataset SDK + dataset endpoints dropped).
utils/build/docker/golang/parametric/main.go — keep only /llm_observability/trace.
manifests/golang.yml — revert native suite to incomplete_test_app; gate the new gen_ai OTLP test at v2.11.0-dev.
tests/parametric/test_llm_observability/test_llm_observability_gen_ai_otlp.py — new hermetic test.

Verified locally: go build/go vet/gofmt clean, ruff clean, manifest validate.py passes. The hermetic wire behavior is verified by CI (I can't run the Docker harness locally).

Open questions (the load-bearing ones)

Does dd-trace-go's OTLP trace export preserve arbitrary gen_ai.* attributes verbatim? (convertSpanAttributes emits every meta entry with the key intact, so it should — the new test proves it.) If not, an explicit otlptracehttp exporter would be needed in llmobs.go.
Does tracer.Flush() drain the OTLP trace exporter (vs. only the native writer)? The test polls the receiver with a deadline as a backstop.
Does the exporter honor OTEL_EXPORTER_OTLP_TRACES_HEADERS (so dd-otlp-source lands on /v1/traces) and emit http/json?
Token metrics arrive as gen_ai.usage.* — int vs double representation depends on whether the bridge routes them via DD meta vs metrics; the assertion accepts either.
Dataset coverage (/dataset/*) is dropped — it depended on the native SDK. Confirm that's acceptable for the gen_ai OTLP scope.

🤖 Generated with Claude Code

Implement /llm_observability/{trace,dataset/create,dataset/delete} in the Go parametric test app using dd-trace-go's public llmobs SDK, and enable tests/parametric/test_llm_observability for golang (was incomplete_test_app). Core span-tree (all seven kinds + APM spans), IO annotation, tags/metadata/ metrics, export-span context, flush, and dataset create are implemented. Prompt annotation and cost_tags are deferred pending a dd-trace-go release with WithAnnotatedPrompt/WithAnnotatedCostTagKeys; annotation_context, annotate-after raising, and dataset delete-by-id need upstream SDK additions. See the PR description for the full list of TODO(draft) items. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-actions · 2026-07-15T15:35:30Z

CODEOWNERS have been resolved as:

tests/parametric/test_llm_observability/test_llm_observability_gen_ai_otlp.py  @DataDog/ml-observability @DataDog/system-tests-core
utils/build/docker/golang/parametric/llmobs.go                          @DataDog/dd-trace-go-guild @DataDog/system-tests-core
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
utils/build/docker/golang/parametric/main.go                            @DataDog/dd-trace-go-guild @DataDog/system-tests-core

datadog-official · 2026-07-15T16:07:17Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 4e40884 | Docs | Datadog PR Page | Give us feedback!}

The newly-enabled golang LLM Obs parametric suite passes 200/215. The 15 failures are all expected gaps, now declared missing_feature so the suite is green with the gaps explicitly recorded: - Test_Prompts: needs llmobs.WithAnnotatedPrompt (unreleased) - Test_CostTags: needs llmobs.WithAnnotatedCostTagKeys (unreleased) - Test_Enablement::test_ml_app: dd-trace-go emits no span when ml_app is unset, vs dd-trace-py defaulting the ml_app tag to the service name - Test_Dataset::test_dataset_create_delete: no public delete-dataset-by-id API Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The missing_feature entries were inserted out of order, failing the manifest key-order check (lint / lint and Test the test :: test_formats). Ran utils/manifest/format.py to sort them (CostTags, Enablement, Prompts, then the _dne Dataset entry); validate.py now passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

khanayan123 · 2026-07-15T17:21:39Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7dd153ddb9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-15T17:30:51Z

+
+	// APM (plain tracer) span branch.
+	if node.SDK == "tracer" {
+		span := tracer.StartSpan(node.Name)


Preserve parent context for tracer spans

When a request nests an APM tracer span under an LLMObs span, this starts the APM span without the incoming ctx, so it becomes a new root and the requested mixed tree loses the LLMObs parent/trace relationship; only grandchildren are placed under this new APM root via ContextWithSpan. Use the passed context when starting tracer spans so llmobs -> tracer -> llmobs structures from the shared request model keep the intended parentage.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-15T17:30:51Z

+		DatasetID:     ds.ID(),
+		Name:          ds.Name(),
+		Description:   req.Description,
+		ProjectName:   optStr(req.ProjectName),


Return configured project name for datasets

When the dataset create request omits project_name, this responds with null even though the LLMObs dataset tests configure DD_LLMOBS_PROJECT_NAME via the llmobs_project_name fixture and assert the response project name is test-project. That means test_dataset_create_delete will still fail at the create-response assertion once the delete API gap is removed; fall back to the configured project name rather than only echoing the request field.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-15T17:30:51Z

-  tests/parametric/test_llm_observability/: incomplete_test_app
+  tests/parametric/test_llm_observability/: v2.11.0-dev
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_CostTags: missing_feature (cost tags need llmobs.WithAnnotatedCostTagKeys, not in a released dd-trace-go yet)
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_Enablement::test_ml_app: missing_feature (Go llmobs does not default ml_app to the service name when unset)


Avoid skipping supported ml_app cases

This manifest entry disables the entire parametrized test_ml_app method for Go; manifest matching strips the [param] suffix, so it skips the explicit "overridden-test-ml-app" case as well as the unset/default cases called out in the reason. That hides coverage for the normal configured-ML-app path, which this handler is supposed to support via DD_LLMOBS_ML_APP, so split or narrow the failing default cases instead of marking the whole method missing.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-15T17:30:51Z

+  tests/parametric/test_llm_observability/: v2.11.0-dev
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_CostTags: missing_feature (cost tags need llmobs.WithAnnotatedCostTagKeys, not in a released dd-trace-go yet)
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_Enablement::test_ml_app: missing_feature (Go llmobs does not default ml_app to the service name when unset)
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_Prompts: missing_feature (prompt annotation needs llmobs.WithAnnotatedPrompt, not in a released dd-trace-go yet)


Stop gating released prompt support

The suite is enabled only for v2.11.0-dev, but this unconditional class-level missing_feature still skips all prompt contract tests even though the Go LLMObs SDK already exposes prompt annotation in released v2.x APIs. As a result, prompt payload regressions in Go will never be exercised despite the target version having the public surface needed to implement them; either wire WithAnnotatedPrompt here or gate only versions before that API exists.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-15T17:30:51Z

+	}
+
+	for _, child := range node.Children {
+		s.createLLMObsTrace(childCtx, child)


Bubble exported contexts from child spans

When export_span is requested on a descendant rather than this span, this loop discards the child return value, so /llm_observability/trace responds with {} even though the shared Python and Node handlers bubble the first exported child context back to the caller. Capture the first non-nil child result so nested export-span contract cases can assert on the returned span_id/trace_id.

Useful? React with 👍 / 👎.

…ect name) - tracer spans now start from the incoming context (StartSpanFromContext) so a tracer span nested under an LLMObs span keeps parentage (llmobs->tracer->llmobs) - bubble the first exported child span-context up through all branches so /llm_observability/trace returns span_id/trace_id when export_span is on a descendant (matches the Python/Node handlers) - dataset create falls back to DD_LLMOBS_PROJECT_NAME for the response project name when the request omits it Kept (with sharpened reasons, not code changes): Test_Prompts / Test_CostTags gates (WithAnnotatedPrompt/WithAnnotatedCostTagKeys are unreleased — absent in v2.9.1 — and using them would break the golang prod image build); test_ml_app whole-method gate (per-param gating is unsupported; shimming a service-name default client-side would hide the real Go/py divergence). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

khanayan123 · 2026-07-15T17:40:05Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9926cbd883

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-15T17:51:41Z

-  tests/parametric/test_llm_observability/: incomplete_test_app
+  tests/parametric/test_llm_observability/: v2.11.0-dev
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_CostTags: missing_feature (cost tags need llmobs.WithAnnotatedCostTagKeys, not in a released dd-trace-go yet)
+  tests/parametric/test_llm_observability/test_llm_observability.py::Test_Enablement::test_ml_app: missing_feature (only the unset/default params fail; Go llmobs emits no span when ml_app is unset, vs dd-trace-py defaulting to the service; per-param gating is unsupported and the configured-ml_app path is covered elsewhere)


Keep the passing ml_app case enabled

This manifest entry suppresses every parametrization of Test_Enablement::test_ml_app, including the llmobs_ml_app="overridden-test-ml-app" case that the reason says passes. I checked tests/parametric/test_llm_observability and this test is the only contract assertion that an explicitly configured ML app is emitted for LLMObs spans, so Go will stop exercising that behavior entirely; if only the empty/None cases are missing, split or otherwise keep the configured case enabled instead of marking the whole method missing.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-15T17:51:41Z

+	for _, child := range node.Children {
+		if e := s.createLLMObsTrace(childCtx, child); e != nil && exported == nil {
+			exported = e


Do not bubble child exports from normal spans

For a normal span request where only a child has export_span set, this bubbles that child's context into the HTTP response. The existing Python and Node handlers only bubble child exports inside annotation_context; for ordinary span children they call create_trace(child) and ignore its return value, so the response remains {} unless the current node itself exports. This makes Go return a different response shape for the same contract input and can break tests or callers that distinguish parent export from child export.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-15T17:51:41Z

+		for _, a := range node.Annotations {
+			applyLLMObsAnnotation(span, a)


Return an error for annotate_after requests

When the request sets annotate_after=true, this applies the annotations after finishing the span but still lets the handler return 200 because dd-trace-go only logs/no-ops on finished-span annotation. The shared Python/Node handlers intentionally let this path raise after the span is closed, and the client supports raise_on_error=False specifically so tests can assert the non-2xx response; Go will report success for the same contract input unless this branch explicitly returns an error or is gated.

Useful? React with 👍 / 👎.

Reorient from the native dd-trace-go llmobs SDK path to the OpenTelemetry gen_ai OTLP path per Datadog's LLM Obs OTel instrumentation schema. - llmobs.go rewritten: the /llm_observability/trace SpanRequest tree is mapped to gen_ai.* attributes and emitted as OTel spans via the existing ddotel bridge; dd-trace-go's OTLP trace exporter (OTEL_TRACES_EXPORTER=otlp) ships them, carrying dd-otlp-source=llmobs. No new Go module. - main.go: keep only /llm_observability/trace (drop the native dataset routes). - manifests/golang.yml: revert the native test_llm_observability enablement to incomplete_test_app; gate the new hermetic gen_ai OTLP test at v2.11.0-dev. - new test asserts the captured OTLP /v1/traces payload carries the correct gen_ai.* attributes + dd-otlp-source=llmobs header; no backend read-back. Local: go build/vet/gofmt clean, ruff clean, manifest validate passes. The OTLP wire behavior is verified by CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The new test_llm_observability_gen_ai_otlp.py lives in the shared parametric dir, so it runs for every language that implements the native LLM Obs suite (python/nodejs/java). Only the Go parametric app was reoriented to emit gen_ai OTLP; the others still emit the native llmobs envelope, so the gen_ai assertions fail for them. Mark the test incomplete_test_app for python/nodejs/ java (Ruby/dotnet/php/cpp already mark the suite irrelevant). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This was referenced Jul 15, 2026

feat(llmobs/export): offline LLM Obs span & evaluation export DataDog/dd-trace-go#4936

Draft

feat(otlp/export): offline OTLP trace/metric/log export DataDog/dd-trace-go#4994

Draft

khanayan123 and others added 2 commits July 15, 2026 12:07

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

khanayan123 force-pushed the ayan.khan/parametric-go-llmobs branch from 828a9c2 to 9926cbd Compare July 15, 2026 17:39

chatgpt-codex-connector Bot reviewed Jul 15, 2026

View reviewed changes

khanayan123 changed the title ~~draft: wire Go parametric app into the LLM Obs contract suite~~ draft: Go parametric gen_ai OTLP LLM Obs contract (hermetic schema assertion) Jul 15, 2026

		for _, a := range node.Annotations {
		applyLLMObsAnnotation(span, a)

Uh oh!

Conversation

khanayan123 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Scope of assertion (hermetic — "just assert the schema")

gen_ai mapping (per node)

Files

Open questions (the load-bearing ones)

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Jul 15, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khanayan123 commented Jul 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

khanayan123 commented Jul 15, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

khanayan123 commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 15, 2026 •

edited

Loading

datadog-official Bot commented Jul 15, 2026 •

edited by datadog-prod-us1-3 Bot

Loading