Add opentelemetry-instrumentation-genai-smolagents instrumentation package

instrumentation/README.md

@@ -9,6 +9,6 @@
 | [opentelemetry-instrumentation-genai-openai](./opentelemetry-instrumentation-genai-openai) | openai >= 1.26.0 | Yes | development
 | [opentelemetry-instrumentation-genai-openai-agents](./opentelemetry-instrumentation-genai-openai-agents) | openai-agents >= 0.3.3 | No | development
 | [opentelemetry-instrumentation-genai-qwen-agent](./opentelemetry-instrumentation-genai-qwen-agent) | qwen-agent >= 0.0.20 | No | development
-| [opentelemetry-instrumentation-genai-smolagents](./opentelemetry-instrumentation-genai-smolagents) | smolagents >= 1.24.0 | No | development
+| [opentelemetry-instrumentation-genai-smolagents](./opentelemetry-instrumentation-genai-smolagents) | smolagents >= 1.24.0 | Yes | development
 | [opentelemetry-instrumentation-genai-weaviate-client](./opentelemetry-instrumentation-genai-weaviate-client) | weaviate-client >= 3.0.0,<5.0.0 | No | development
 | [opentelemetry-instrumentation-google-genai](./opentelemetry-instrumentation-google-genai) | google-genai >= 1.32.0, <3 | No | development

instrumentation/opentelemetry-instrumentation-genai-smolagents/.changelog/340.added

@@ -0,0 +1 @@
+Add smolagents instrumentation package (``opentelemetry-instrumentation-genai-smolagents``).

instrumentation/opentelemetry-instrumentation-genai-smolagents/README.rst

@@ -7,7 +7,53 @@ OpenTelemetry smolagents Instrumentation
    :target: https://pypi.org/project/opentelemetry-instrumentation-genai-smolagents/
 
 This library provides OpenTelemetry instrumentation for `smolagents
-<https://github.com/huggingface/smolagents>`_.
+<https://github.com/huggingface/smolagents>`_. It emits GenAI semantic-convention
+telemetry (``invoke_agent``, ``chat``, and ``execute_tool`` spans plus the
+associated metrics) through ``opentelemetry-util-genai``.
+
+Token usage is reported on the ``chat`` span of the model call that produced it,
+not summed onto the ``invoke_agent`` span.
+
+A streamed model call, whether it comes from ``stream_outputs=True`` on an agent
+or from calling ``Model.generate_stream`` directly, gets a ``chat`` span that
+stays open until the caller drains the deltas. The span carries
+``gen_ai.request.stream``, and the call also records the
+``gen_ai.client.operation.time_to_first_chunk`` and
+``gen_ai.client.operation.time_per_output_chunk`` metrics.
+
+Known gaps:
+
+* There is no span for an individual agent step, so ``chat`` and
+  ``execute_tool`` spans nest directly under ``invoke_agent``. The GenAI
+  semantic conventions define no operation for one iteration of a
+  reason-and-act loop. ``invoke_workflow`` is the closest name, but the
+  OpenAI Agents and LangChain instrumentations both use ``invoke_workflow``
+  for the outermost orchestration, above ``invoke_agent``. A dedicated span is
+  proposed in
+  `semantic-conventions-genai#81
+  <https://github.com/open-telemetry/semantic-conventions-genai/issues/81>`_.
+* The instrumentation patches ``generate`` and ``generate_stream`` on the model
+  classes that smolagents ships. A subclass that inherits either method is
+  instrumented. A subclass that overrides one is not: the override shadows the
+  patched method, so the call produces no ``chat`` span.
+* A streamed ``chat`` span reports no ``gen_ai.response.id`` and no
+  ``gen_ai.response.model``, because a smolagents stream delta carries neither.
+  The span reports ``gen_ai.response.finish_reasons`` only when the model
+  requested tool calls, which is the one stop reason the deltas make visible.
+* The agent's step budget (``max_steps``) is not recorded. The GenAI semantic
+  conventions define no attribute for it, and a run that uses the budget up
+  already reports ``gen_ai.response.finish_reasons`` as ``length`` on the
+  ``invoke_agent`` span.
+* ``gen_ai.tool.call.id`` is omitted when a single step calls the *same* tool
+  more than once. ``Tool.__call__`` receives only the argument values, so this
+  instrumentation matches a call to its id by tool name, and two calls to one
+  tool in the same step have no unambiguous match. A ``CodeAgent`` has no tool
+  call ids at all: its model writes code rather than tool calls.
+
+A ``CodeAgent`` with ``executor_type="local"`` runs generated code through
+smolagents' ``local_python_executor.timeout()``, which this instrumentation
+wraps so tool spans started in its worker thread stay nested under the agent
+span.
 
 Installation
 ------------
@@ -24,10 +70,13 @@ Usage
     from opentelemetry.instrumentation.genai.smolagents import (
         SmolagentsInstrumentor,
     )
+    from smolagents import CodeAgent, InferenceClientModel
 
-    # Instrument smolagents
     SmolagentsInstrumentor().instrument()
 
+    agent = CodeAgent(tools=[], model=InferenceClientModel())
+    agent.run("How many seconds are in a week?")
+
 Configuration
 -------------
 
@@ -71,6 +120,16 @@ environment variable:
 
     SmolagentsInstrumentor().instrument(completion_hook=my_hook)
 
+See ``examples/manual/`` for a runnable example with the SDK configured in
+code (including ``custom_hook.py`` for the completion hook) and
+``examples/zero-code/`` for the ``opentelemetry-instrument`` variant.
+
+Conformance
+-----------
+
+The scenarios that check this package against the GenAI semantic conventions
+live under ``tests/conformance/``.
+
 References
 ----------
 

instrumentation/opentelemetry-instrumentation-genai-smolagents/examples/manual/.env

@@ -0,0 +1,24 @@
+# Update this with your real Hugging Face token
+HF_TOKEN=hf_YOUR_TOKEN
+
+# Uncomment to run a different model through Hugging Face Inference Providers
+# CHAT_MODEL=meta-llama/Llama-3.3-70B-Instruct
+
+# Uncomment and change to your OTLP endpoint
+# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+# OTEL_EXPORTER_OTLP_PROTOCOL=grpc
+
+OTEL_SERVICE_NAME=opentelemetry-python-smolagents
+
+# Remove to hide prompt and completion content
+# Possible values (case insensitive):
+# - `span_only` - record content on span attributes
+# - `event_only` - record content on event attributes
+# - `span_and_event` - record content on both span and event attributes
+# - everything else - don't record content on any signal
+OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
+
+# Uncomment to upload prompts and responses to an fsspec-compatible
+# destination instead of or in addition to recording them inline on spans/events.
+# OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload
+# OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH=/path/to/prompts

instrumentation/opentelemetry-instrumentation-genai-smolagents/examples/manual/README.rst

@@ -0,0 +1,58 @@
+OpenTelemetry smolagents Instrumentation Example
+================================================
+
+This is an example of how to instrument smolagents agent runs when configuring
+the OpenTelemetry SDK and instrumentations manually.
+
+When `main.py <main.py>`_ is run, it exports traces, metrics, and logs to an
+OTLP compatible endpoint. The agent run produces an ``invoke_agent`` span with
+``chat`` and ``execute_tool`` spans nested under it, so you can see which model
+calls and tools each step of the run made.
+
+Note: `.env <.env>`_ file configures additional environment variables:
+
+- ``OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only`` configures the smolagents instrumentation to capture prompt and completion contents on *span* attributes.
+- ``OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK`` (commented out) - uncomment along with ``OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH`` to upload prompts and completions to an ``fsspec``-compatible destination instead of recording them inline. Also uncomment the ``opentelemetry-util-genai[upload]`` line in `requirements.txt <requirements.txt>`_ and reinstall.
+
+Setup
+-----
+
+Minimally, update the `.env <.env>`_ file with your ``HF_TOKEN``. An OTLP
+compatible endpoint should be listening for traces, metrics, and logs on
+http://localhost:4317. If not, update ``OTEL_EXPORTER_OTLP_ENDPOINT`` as well.
+
+Next, set up a virtual environment like this:
+
+::
+
+    python3 -m venv .venv
+    source .venv/bin/activate
+    pip install "python-dotenv[cli]"
+    pip install -r requirements.txt
+
+Run
+---
+
+Run the example like this:
+
+::
+
+    dotenv run -- python main.py
+
+You should see the agent's answer while traces, metrics, and logs export to
+your configured observability tool.
+
+Custom completion hook
+----------------------
+
+`custom_hook.py <custom_hook.py>`_ is a variant of ``main.py`` that passes a
+custom ``CompletionHook`` implementation programmatically via
+``SmolagentsInstrumentor().instrument(completion_hook=...)``. The example hook
+prints prompts and completions to stdout; real hooks typically forward content
+to external storage and record reference URIs on the span/log_record.
+
+Run it the same way:
+
+::
+
+    dotenv run -- python custom_hook.py

instrumentation/opentelemetry-instrumentation-genai-smolagents/examples/manual/custom_hook.py

@@ -0,0 +1,94 @@
+# Copyright The OpenTelemetry Authors
+# SPDX-License-Identifier: Apache-2.0
+
+# pylint: skip-file
+"""Instrument smolagents with a custom CompletionHook that prints prompts and
+completions to stdout.
+
+Run with: dotenv run -- python custom_hook.py
+"""
+
+import os
+
+from smolagents import CodeAgent, InferenceClientModel
+
+from opentelemetry import _logs, metrics, trace
+from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
+    OTLPLogExporter,
+)
+from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
+    OTLPMetricExporter,
+)
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
+    OTLPSpanExporter,
+)
+from opentelemetry.instrumentation.genai.smolagents import (
+    SmolagentsInstrumentor,
+)
+from opentelemetry.sdk._logs import LoggerProvider
+from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
+from opentelemetry.sdk.metrics import MeterProvider
+from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.util.genai.completion_hook import CompletionHook
+from opentelemetry.util.genai.types import (
+    InputMessage,
+    MessagePart,
+    OutputMessage,
+    ToolDefinition,
+)
+
+
+class PrintCompletionHook(CompletionHook):
+    """Minimal CompletionHook that prints inputs/outputs to stdout.
+
+    Real hooks typically forward content to external storage (object store,
+    database, etc.) and record reference URIs on the span/log_record.
+    """
+
+    def on_completion(
+        self,
+        *,
+        inputs: list[InputMessage],
+        outputs: list[OutputMessage],
+        system_instruction: list[MessagePart],
+        tool_definitions: list[ToolDefinition] | None = None,
+        span=None,
+        log_record=None,
+    ) -> None:
+        print(f"[hook] inputs: {inputs}")
+        print(f"[hook] outputs: {outputs}")
+
+
+trace.set_tracer_provider(TracerProvider())
+trace.get_tracer_provider().add_span_processor(
+    BatchSpanProcessor(OTLPSpanExporter())
+)
+
+_logs.set_logger_provider(LoggerProvider())
+_logs.get_logger_provider().add_log_record_processor(
+    BatchLogRecordProcessor(OTLPLogExporter())
+)
+
+metrics.set_meter_provider(
+    MeterProvider(
+        metric_readers=[
+            PeriodicExportingMetricReader(OTLPMetricExporter()),
+        ]
+    )
+)
+
+SmolagentsInstrumentor().instrument(completion_hook=PrintCompletionHook())
+
+
+def main():
+    agent = CodeAgent(tools=[], model=InferenceClientModel())
+    result = agent.run(
+        os.getenv("SMOLAGENTS_TASK", "How many seconds are in a week?")
+    )
+    print(result)
+
+
+if __name__ == "__main__":
+    main()

instrumentation/opentelemetry-instrumentation-genai-smolagents/examples/manual/main.py

@@ -0,0 +1,72 @@
+# Copyright The OpenTelemetry Authors
+# SPDX-License-Identifier: Apache-2.0
+
+# pylint: skip-file
+import os
+
+from smolagents import CodeAgent, InferenceClientModel
+
+# NOTE: OpenTelemetry Python Logs API is in beta
+from opentelemetry import _logs, metrics, trace
+from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
+    OTLPLogExporter,
+)
+from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
+    OTLPMetricExporter,
+)
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
+    OTLPSpanExporter,
+)
+from opentelemetry.instrumentation.genai.smolagents import (
+    SmolagentsInstrumentor,
+)
+from opentelemetry.sdk._logs import LoggerProvider
+from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
+from opentelemetry.sdk.metrics import MeterProvider
+from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+
+# configure tracing
+trace.set_tracer_provider(TracerProvider())
+trace.get_tracer_provider().add_span_processor(
+    BatchSpanProcessor(OTLPSpanExporter())
+)
+
+# configure logging
+_logs.set_logger_provider(LoggerProvider())
+_logs.get_logger_provider().add_log_record_processor(
+    BatchLogRecordProcessor(OTLPLogExporter())
+)
+
+# configure metrics
+metrics.set_meter_provider(
+    MeterProvider(
+        metric_readers=[
+            PeriodicExportingMetricReader(
+                OTLPMetricExporter(),
+            ),
+        ]
+    )
+)
+
+# instrument smolagents
+SmolagentsInstrumentor().instrument()
+
+
+def main():
+    agent = CodeAgent(
+        tools=[],
+        model=InferenceClientModel(
+            model_id=os.getenv("CHAT_MODEL", "Qwen/Qwen2.5-Coder-32B-Instruct")
+        ),
+    )
+    print(
+        agent.run(
+            os.getenv("SMOLAGENTS_TASK", "How many seconds are in a week?")
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

instrumentation/opentelemetry-instrumentation-genai-smolagents/examples/manual/requirements.txt

@@ -0,0 +1,9 @@
+smolagents>=1.24.0
+
+opentelemetry-sdk~=1.43.0
+opentelemetry-exporter-otlp-proto-grpc~=1.43.0
+opentelemetry-instrumentation-genai-smolagents~=1.1b0
+
+# Uncomment to enable the upload completion hook (pulls in fsspec).
+# Required when OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload.
+# opentelemetry-util-genai[upload]

instrumentation/opentelemetry-instrumentation-genai-smolagents/examples/zero-code/.env

@@ -0,0 +1,27 @@
+# Update this with your real Hugging Face token
+HF_TOKEN=hf_YOUR_TOKEN
+
+# Uncomment to run a different model through Hugging Face Inference Providers
+# CHAT_MODEL=meta-llama/Llama-3.3-70B-Instruct
+
+# Uncomment and change to your OTLP endpoint
+# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
+# OTEL_EXPORTER_OTLP_PROTOCOL=grpc
+
+OTEL_SERVICE_NAME=opentelemetry-python-smolagents
+
+# Remove to hide prompt and completion content
+# Possible values (case insensitive):
+# - `span_only` - record content on span attributes
+# - `event_only` - record content on event attributes
+# - `span_and_event` - record content on both span and event attributes
+# - everything else - don't record content on any signal
+OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
+
+# Uncomment to upload prompts and responses to an fsspec-compatible
+# destination instead of or in addition to recording them inline on spans/events.
+# OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload
+# OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH=/path/to/prompts
+
+# Uncomment if your OTLP endpoint doesn't support logs
+# OTEL_LOGS_EXPORTER=console