[opentelemetry-instrumentation-genai-smolagents] Add chat instrumentation by alexander-akhmetov · Pull Request #352 · open-telemetry/opentelemetry-python-genai

alexander-akhmetov · 2026-08-03T21:23:08Z

Split from #340. Agent runs (invoke_agent) and tool calls (execute_tool) will follow in separate PRs.

In smolagents, every LLM call goes through a Model class: OpenAIModel, LiteLLMModel, InferenceClientModel, AmazonBedrockModel, etc.

Each of these classes defines its own generate method, and so does the Model base class. The instrumentation wraps generate wherever it is defined. A call to a wrapped generate emits one chat span, an operation-duration metric and a token-usage metric, all through the public opentelemetry-util-genai API.

Known gaps, also listed in the package README.rst:

If your own Model subclass overrides generate, the override hides the wrapped method and the call produces no chat span. A subclass that inherits generate is fine.
A streamed chat span reports no gen_ai.response.id and no gen_ai.response.model, because a smolagents stream delta does not have them.

Part of #141

Type of change

New feature (non-breaking change which adds functionality)

How has this been tested?

uv run tox -e py310-test-instrumentation-genai-smolagents-oldest
uv run tox -e py314-test-instrumentation-genai-smolagents-latest
uv run tox -e py314-test-instrumentation-genai-smolagents-conformance

Checklist

See CONTRIBUTING.md
for the style guide, changelog guidance, and more.

Followed the style guidelines of this project
Changelog updated if the change requires an entry
Unit tests added
Documentation updated

opentelemetry-pr-dashboard · 2026-08-03T21:24:01Z

Pull request dashboard status

Waiting on the author · refreshed 2026-08-05 05:11 UTC

Respond to 9 review items (e.g. link a commit, explain why not, ask a follow-up):

Inline threads: 1, 2, 3, 4, 5, 6, 7, 8, 9

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Copilot

Pull request overview

Adds chat instrumentation to the opentelemetry-instrumentation-genai-smolagents package by wrapping smolagents model generate() methods and emitting GenAI semconv spans/metrics via opentelemetry-util-genai, plus accompanying unit + conformance tests.

Changes:

Patch smolagents model classes that define generate() to emit chat spans (and related metrics), with provider/endpoint resolution and message/tool mapping.
Add VCR-backed unit tests and Weaver conformance scenarios for chat, tool-calling, multimodal input, and reasoning output.
Wire smolagents conformance tox env + mark metrics support in package docs/metadata.

Reviewed changes

Copilot reviewed 23 out of 24 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tox.ini	Adds a smolagents conformance env and runner command.
instrumentation/README.md	Marks smolagents as supporting metrics.
instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/init.py	Implements wrapping/unwrapping of `Model.generate` on defining classes; updates module docs.
instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/patch.py	Wrapt wrapper implementing `chat` span creation + attribute extraction via util-genai.
instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/_messages.py	Converts smolagents message/tool shapes into util-genai typed models.
instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/provider.py	Resolves `gen_ai.provider.name` and `(server.address, server.port)` from model instances.
instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/package.py	Declares `_supports_metrics = True`.
instrumentation/opentelemetry-instrumentation-genai-smolagents/README.rst	Documents chat instrumentation, known gaps, and conformance location.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/conftest.py	Adds shared fixtures, VCR config/scrubbing, and content-capture variants.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_instrumentor.py	Adds lifecycle, rollback, and completion-hook precedence tests.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_models.py	Adds extensive VCR-backed tests for providers, parameters, messages, tools, errors, and local runtimes.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_utils.py	Adds shared test helpers and stubs.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/test_conformance.py	Adds conformance runner for scenario modules.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/conformance/_helpers.py	Adds helper functions for conformance validations.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/conformance/inference.py	Adds conformance scenarios for basic chat + tool-calling chat.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/conformance/multimodal.py	Adds conformance scenarios for image input and reasoning output (with expected violation).
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/cassettes/*.yaml	Adds recorded VCR cassettes for OpenAI + Anthropic(LiteLLM) scenarios.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/requirements.oldest.txt	Pins test-only model backends needed for oldest test matrix.
instrumentation/opentelemetry-instrumentation-genai-smolagents/tests/requirements.latest.txt	Installs smolagents with backend extras used by VCR tests.
instrumentation/opentelemetry-instrumentation-genai-smolagents/.changelog/352.added	Adds towncrier fragment for the new chat instrumentation feature.

Wrap every smolagents model class that defines its own `generate`, and emit a chat span through the public opentelemetry-util-genai API. The span records the request parameters, provider and endpoint, token usage, finish reason, tool definitions, and the input and output messages. Agent runs and tool calls stay uninstrumented. They follow in separate PRs. Tests run against smolagents 1.24.0 (the declared floor) and against the latest release, plus Weaver conformance scenarios for chat, tool-calling, image input, and reasoning output. Known gaps: - Model.generate_stream is not instrumented, so a call made with stream_outputs=True produces no chat span, no metrics, and no token usage. - A user-defined Model subclass that overrides generate shadows the patched base method and produces no chat span.

Copilot

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.

Suppressed comments (3)

instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/init.py:26

The usage example instantiates InferenceClientModel() with no arguments, but the tests in this repo construct it with model_id and token. The example is likely not runnable as-is; please include the required constructor args and use the messages= kwarg for clarity.

    model = InferenceClientModel()
    model.generate([{"role": "user", "content": "How many seconds are in a week?"}])

instrumentation/opentelemetry-instrumentation-genai-smolagents/README.rst:56

The usage snippet uses InferenceClientModel() with no constructor args, but this repo’s tests instantiate it with model_id and token. As written, the README example is likely not runnable; update it to pass the required args and call generate(messages=...).

    model = InferenceClientModel()
    model.generate([{"role": "user", "content": "How many seconds are in a week?"}])

instrumentation/opentelemetry-instrumentation-genai-smolagents/src/opentelemetry/instrumentation/genai/smolagents/patch.py:377

Streamed tool calls are emitted in dict.values() iteration order, which depends on the order deltas arrive. Since tool calls are indexed, the output should be ordered by index to keep telemetry deterministic and consistent with provider ordering.

        parts.extend(
            ToolCallRequest(
                name=tool_call.name,
                id=tool_call.id,
                arguments=tool_call.arguments or None,

DylanRussell · 2026-08-04T16:24:48Z

+
+def resolve_provider(instance: Any) -> str:
+    """Return the ``gen_ai.provider.name`` value for a smolagents model instance."""
+    class_names = _class_names(instance)


_class_names is just called in 1 place it seems ? If thats the case I'd just inline the code

DylanRussell · 2026-08-04T16:30:46Z

                - completion_hook: CompletionHook instance
        """
-        TelemetryHandler(
+        import smolagents  # pylint: disable=import-outside-toplevel  # noqa: PLC0415


why is this outside the toplevel?

DylanRussell · 2026-08-04T18:10:46Z

+    """
+    from smolagents.models import (  # noqa: PLC0415  # pylint: disable=import-outside-toplevel
+        Model,
+    )


same question.. we seem to always call this function so why not put this at the top ?

DylanRussell · 2026-08-04T18:21:31Z

+        wrapped_generate_classes: list[type] = []
+        self._wrapped_generate_classes = wrapped_generate_classes
+        wrapped_generate_stream_classes: list[type] = []
+        self._wrapped_generate_stream_classes = wrapped_generate_stream_classes


why not just do self._wrapped_generate_classes = [] and self._wrapped_generate_classes.append()

DylanRussell · 2026-08-04T18:36:50Z

+    return None
+
+
+def _parts_from_content(content: Any) -> list[Any]:


list[Any] seems to broad.. looks like it should be list[Text, ImagePart] ?

DylanRussell · 2026-08-04T18:39:29Z

+            if element.get("type") == "text" and (text := element.get("text")):
+                parts.append(Text(content=text))
+                continue
+            image_part = _image_part_from_element(element)


could be reduced to if image_part := _image_part_from_element..

DylanRussell · 2026-08-04T18:53:59Z

+    ) -> Any:
+        invocation = _start_inference(handler, wrapped, instance, args, kwargs)
+
+        try:


do we want to generate telemetry if the stream fails to be initialized completely ??

I sort of think we don't want to emit telemetry in this case (it's just a user failing to use this library correctly and we should just skip telemetry in this case... just let the exception get raised..)

That's what I did for the google genai instrumentation anyway, happy to hear other opinion..

DylanRussell · 2026-08-04T18:55:24Z

+        args: tuple[Any, ...],
+        kwargs: dict[str, Any],
+    ) -> Any:
+        invocation = _start_inference(handler, wrapped, instance, args, kwargs)


i would use the context manager here.. with _start_inference as invocation... that way all the finalization and error capturing stuff is done for you automatically when you exit the with block..

DylanRussell · 2026-08-04T19:06:40Z

+        if output.finish_reason:
+            invocation.finish_reasons = [output.finish_reason]
+        if handler.should_capture_content():
+            invocation.output_messages = [output]


we shouldn't generate output messages at all if capture content is not enabled

opentelemetry-pr-dashboard Bot mentioned this pull request Aug 3, 2026

Pull Request Dashboard #171

Open

alexander-akhmetov force-pushed the alexander-akhmetov/smolagents-chat branch from 84a5276 to 1c65b72 Compare August 3, 2026 21:25

alexander-akhmetov marked this pull request as ready for review August 3, 2026 21:51

alexander-akhmetov requested a review from a team as a code owner August 3, 2026 21:51

Copilot AI review requested due to automatic review settings August 3, 2026 21:51

Copilot started reviewing on behalf of alexander-akhmetov August 3, 2026 21:51 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

alexander-akhmetov force-pushed the alexander-akhmetov/smolagents-chat branch from 1c65b72 to cb5bd69 Compare August 3, 2026 23:21

github-project-automation Bot added this to GenAI Semantic Conventions and Instrumentation libraries Aug 3, 2026

alexander-akhmetov requested a review from Copilot August 3, 2026 23:24

Copilot started reviewing on behalf of alexander-akhmetov August 3, 2026 23:25 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

DylanRussell reviewed Aug 4, 2026

View reviewed changes

Merge branch 'main' into alexander-akhmetov/smolagents-chat

a7c344b

		return None


		def _parts_from_content(content: Any) -> list[Any]:

Uh oh!

Conversation

alexander-akhmetov commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

How has this been tested?

Checklist

Uh oh!

opentelemetry-pr-dashboard Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexander-akhmetov commented Aug 3, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Aug 3, 2026 •

edited

Loading