test(openai): enforce experimental embedding metric conformance by tomatotomata · Pull Request #327 · open-telemetry/opentelemetry-python-genai

tomatotomata · 2026-07-28T15:30:46Z

Summary

run the OpenAI embedding conformance scenario explicitly in latest experimental semantic-convention mode
add a regression test that requires gen_ai.provider.name on both embedding metrics

Fixes #38

Validation

python -m pytest instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_embedding_invocation_unit.py -q � 11 passed
Ruff and git diff --check pass
The live Weaver conformance scenario is configured but cannot run locally because the Weaver binary is not installed; CI conformance will execute it.

opentelemetry-pr-dashboard · 2026-07-28T15:32:38Z

Pull request dashboard status

Waiting on reviewers · refreshed 2026-08-04 14:21 UTC

Review the latest changes.

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

lmolkova · 2026-07-28T18:48:07Z

            logger_provider=logger_provider,
            meter_provider=meter_provider,
            content_capture="SPAN_ONLY",
+            extra_env={


this should not be necessary in this repo - we should have removed this flag here. If there are some leftovers that require it, could you please send a PR to switch to latest experimental by default and unconditionally ?

The conformance scenario no longer passes an experimental-mode flag; commit c220d9d removed that plumbing so it now exercises the repository default unconditionally.

Copilot

Pull request overview

This PR strengthens the OpenAI embeddings instrumentation test suite to prevent metric semantic-convention regressions, and records the fix in the package changelog fragment.

Changes:

Adds a unit regression test asserting gen_ai.provider.name is present on the two emitted embedding metrics.
Adds a towncrier fragment documenting the conformance/metric attribute guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_embedding_invocation_unit.py	Adds a regression test around embedding metric attributes for `gen_ai.provider.name`.
instrumentation/opentelemetry-instrumentation-genai-openai/.changelog/38.fixed	Documents the user-visible fix/regression guard.

tomatotomata · 2026-07-29T13:41:52Z

+    metrics = metric_reader.get_metrics_data()
+    metric_names = {
+        metric.name
+        for resource_metric in metrics.resource_metrics
+        for scope_metric in resource_metric.scope_metrics
+        for metric in scope_metric.metrics
+    }
+    assert {
+        "gen_ai.client.operation.duration",
+        "gen_ai.client.token.usage",
+    } <= metric_names
+
+    for resource_metric in metrics.resource_metrics:
+        for scope_metric in resource_metric.scope_metrics:
+            for metric in scope_metric.metrics:
+                if metric.name in {
+                    "gen_ai.client.operation.duration",
+                    "gen_ai.client.token.usage",
+                }:
+                    for point in metric.data.data_points:
+                        assert (
+                            point.attributes["gen_ai.provider.name"]
+                            == "openai"
+                        )
+


Implemented in 8895d2f: the regression now lives in the existing embedding integration test, selects the duration and token metrics explicitly, requires data points, uses GenAIAttributes constants/enums, asserts gen_ai.provider.name == OPENAI, and verifies the deprecated gen_ai.system attribute is absent.

lmolkova · 2026-07-29T04:16:14Z

        invocation.stop()


+def test_embedding_metrics_use_provider_name_attribute(handler, metric_reader):


this is already covered in

opentelemetry-python-genai/util/opentelemetry-util-genai/tests/test_handler_metrics.py

Line 266 in e92e604

duration_point.attributes[GenAI.GEN_AI_PROVIDER_NAME], "embed-prov"

if the goal is to test all openai instrumentation e2e consider adding missing attributes in existing tests https://github.com/ahmadalguydi/opentelemetry-python-genai/blob/77dbb757b64025a90db5841eb6e79e8513c19be9/instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_embeddings.py

Moved the regression out of the unit test and into test_embeddings_token_metrics in test_embeddings.py, so it covers the end-to-end OpenAI instrumentation path. The test now checks both emitted metrics and their data points, including the provider attribute and absence of the deprecated system key.

Run embedding conformance with the latest semantic conventions and assert both emitted metrics carry gen_ai.provider.name. Fixes open-telemetry#38

Remove the scenario-local stability opt-in so the conformance test follows the repository-wide experimental semantic convention default.

Move the provider-name regression into the existing embedding integration test. Check both expected metrics explicitly, require data points, and reject the deprecated provider attribute.

lmolkova · 2026-07-29T14:41:51Z

+                point.attributes[GenAIAttributes.GEN_AI_PROVIDER_NAME]
+                == GenAIAttributes.GenAiProviderNameValues.OPENAI.value
+            )
+            assert GenAIAttributes.GEN_AI_SYSTEM not in point.attributes


there is no need to test it. the original issue was fixed and conformance test would flag usage of deprecated attributes

Suggested change

assert GenAIAttributes.GEN_AI_SYSTEM not in point.attributes

Removed the redundant assertions in 488c735. The existing conformance coverage remains the guard for deprecated attributes. The affected test_embeddings_token_metrics test passes locally, along with Ruff, formatting, and git diff --check.

lmolkova

this PR is empty

tomatotomata · 2026-08-02T19:13:24Z

I checked the current base comparison before replying. The implementation commits are already present on main, so this PR now contains only the changelog fragment. I was thinking the cleanest path is to treat it as superseded rather than add another no-op commit. If the fragment is still wanted separately, I can keep it here; otherwise I’ll close this stale remainder.

opentelemetry-pr-dashboard Bot mentioned this pull request Jul 28, 2026

Pull Request Dashboard #171

Open

tomatotomata marked this pull request as ready for review July 28, 2026 15:37

tomatotomata requested a review from a team as a code owner July 28, 2026 15:37

lmolkova reviewed Jul 28, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 29, 2026 04:06

Copilot started reviewing on behalf of tomatotomata July 29, 2026 04:07 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

lmolkova reviewed Jul 29, 2026

View reviewed changes

tomatotomata force-pushed the codex/openai-embedding-metrics-conformance-38 branch from 77dbb75 to 24c9894 Compare July 29, 2026 04:23

test(openai): enforce experimental embedding metric conformance

a238901

Run embedding conformance with the latest semantic conventions and assert both emitted metrics carry gen_ai.provider.name. Fixes open-telemetry#38

tomatotomata force-pushed the codex/openai-embedding-metrics-conformance-38 branch from 24c9894 to b7e9ec9 Compare July 29, 2026 04:59

tomatotomata added 2 commits July 29, 2026 08:00

fix(test): use repository semantic convention defaults

c220d9d

Remove the scenario-local stability opt-in so the conformance test follows the repository-wide experimental semantic convention default.

test(openai): strengthen embedding metric assertions

8895d2f

Move the provider-name regression into the existing embedding integration test. Check both expected metrics explicitly, require data points, and reject the deprecated provider attribute.

tomatotomata force-pushed the codex/openai-embedding-metrics-conformance-38 branch from b7e9ec9 to 8895d2f Compare July 29, 2026 05:00

github-project-automation Bot added this to GenAI Semantic Conventions and Instrumentation libraries Jul 29, 2026

lmolkova reviewed Jul 29, 2026

View reviewed changes

test(openai): remove redundant embedding metric assertions

488c735

lmolkova requested changes Aug 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

test(openai): enforce experimental embedding metric conformance - #327

test(openai): enforce experimental embedding metric conformance#327
tomatotomata wants to merge 4 commits into
open-telemetry:mainfrom
tomatotomata:codex/openai-embedding-metrics-conformance-38

tomatotomata commented Jul 28, 2026 •

edited

Loading

Uh oh!

opentelemetry-pr-dashboard Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

lmolkova Jul 28, 2026

Uh oh!

tomatotomata Jul 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

tomatotomata Jul 29, 2026

Uh oh!

lmolkova Jul 29, 2026

Uh oh!

tomatotomata Jul 29, 2026

Uh oh!

lmolkova Jul 29, 2026

Uh oh!

tomatotomata Jul 30, 2026

Uh oh!

lmolkova left a comment

Uh oh!

tomatotomata commented Aug 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		invocation.stop()


		def test_embedding_metrics_use_provider_name_attribute(handler, metric_reader):

Uh oh!

Conversation

tomatotomata commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

opentelemetry-pr-dashboard Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

lmolkova Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

tomatotomata Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

tomatotomata Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

tomatotomata Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

tomatotomata Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova left a comment

Choose a reason for hiding this comment

Uh oh!

tomatotomata commented Aug 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tomatotomata commented Jul 28, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Jul 28, 2026 •

edited

Loading