[Instrumentation] Contribute loongsuite GenAI instrumentations to OTel Python GenAI

[123liuziming]

Summary

We would like to contribute a set of GenAI instrumentation packages from alibaba/loongsuite-python to this repository. These instrumentations follow the OpenTelemetry GenAI Semantic Conventions and have been validated through the genai-otel-conformance test suite.

Motivation

There is a growing ecosystem of GenAI agent frameworks and LLM libraries that lack official OTel instrumentation. We have developed and tested instrumentations for 26 additional libraries, covering:

• Inference Spans (chat completions, text generation)
• Agent Spans (create agent, invoke agent client/internal)
• Embeddings Spans
• Execute Tool Spans
• Client Operation Duration Metric (gen_ai.client.operation.duration)
• Client Token Usage Metric (gen_ai.client.token.usage)

Conformance Test Results

All proposed instrumentations have passed the GenAI OTel Conformance Tests. The full conformance dashboard showing coverage across Required, Conditionally Required, Recommended, and Opt-In attributes for each signal type is attached below:

dashboard-full.pdf

Proposed Instrumentations

Below is the full list of instrumentations available for contribution, grouped by priority:

Priority 1: Popular Open-Source Frameworks (High Community Value)

Instrumentation	Target Package	Description
loongsuite-instrumentation-crewai	crewai >= 0.80.0	Multi-agent orchestration framework
loongsuite-instrumentation-langgraph	langgraph >= 0.2	LangChain agent orchestration
loongsuite-instrumentation-litellm	litellm >= 1.0.0	Universal LLM proxy/gateway
loongsuite-instrumentation-google-adk	google-adk >= 0.1.0	Google Agent Development Kit
loongsuite-instrumentation-dashscope	dashscope >= 1.0.0	Alibaba Cloud model serving platform
loongsuite-instrumentation-qwen-agent	qwen-agent >= 0.0.20	Qwen agent framework
loongsuite-instrumentation-mcp	mcp >= 1.3.0	Model Context Protocol
loongsuite-instrumentation-mem0	mem0ai >= 1.0.0, < 2.0.0	Memory layer for AI apps
loongsuite-instrumentation-dify	dify	LLMOps platform
loongsuite-instrumentation-agno	agno >= 2.0.0, < 3	Agent framework
loongsuite-instrumentation-agentscope	agentscope >= 1.0.0	Multi-agent platform

Priority 2: Agent Evaluation & Benchmark Frameworks

Instrumentation	Target Package	Description
loongsuite-instrumentation-openhands	openhands	AI software engineering agent
loongsuite-instrumentation-minisweagent	mini-swe-agent >= 2.2.0	Minimal SWE agent
loongsuite-instrumentation-bfclv4	bfcl-eval >= 4.0.0	Function calling evaluation
loongsuite-instrumentation-claw-eval	claw-eval >= 0.1.0	Agent evaluation framework
loongsuite-instrumentation-algotune	algotune	Algorithm tuning benchmark
loongsuite-instrumentation-webarena	webarena >= 0.0.1	Web agent benchmark
loongsuite-instrumentation-widesearch	widesearch >= 0.1.0	Search agent benchmark
loongsuite-instrumentation-wildtool	openai >= 1.0.0	Tool-use evaluation

Priority 3: Domain-Specific Agent Instrumentations

Instrumentation	Target Package	Description
loongsuite-instrumentation-claude-agent-sdk	claude-agent-sdk >= 0.1.0	Anthropic Claude agent SDK
loongsuite-instrumentation-hermes-agent	openai >= 1.0.0	Hermes agent framework
loongsuite-instrumentation-qwenpaw	qwenpaw >= 1.1.0	QwenPaw personal assistant
loongsuite-instrumentation-slop-code	slop-code-bench >= 0.1	Code agent benchmark
loongsuite-instrumentation-terminus2	terminal-bench >= 0.1.0	Terminal agent benchmark
loongsuite-instrumentation-vita	vita >= 0.0.1	Vita agent
loongsuite-instrumentation-langchain	langchain_core >= 0.1.0	Enhanced LangChain instrumentation

Semantic Convention Coverage

Each instrumentation emits telemetry aligned with the GenAI semantic conventions. Example coverage for Inference Spans:

• Required: gen_ai.operation.name, gen_ai.system
• Conditionally Required: error.type, gen_ai.request.model, server.address, server.port
• Recommended: gen_ai.response.finish_reasons, gen_ai.response.id, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens
• Opt-In: gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions

Contribution Plan

We propose to contribute these instrumentations in phases:

1. Phase 1: Submit Priority 1 instrumentations (popular open-source frameworks) as individual PRs
2. Phase 2: Submit Priority 2 instrumentations (evaluation/benchmark frameworks)
3. Phase 3: Submit Priority 3 instrumentations (domain-specific agents)

Each PR will include:

• Instrumentation source code (following existing package structure)
• Unit tests
• Updated conformance test data in genai-otel-conformance

Source Repository

• Instrumentation code: https://github.com/alibaba/loongsuite-python/tree/main/instrumentation-loongsuite
• Conformance tests: https://github.com/trask/genai-otel-conformance

Questions for Maintainers

1. Is there a preference for batching PRs (e.g., one PR per instrumentation, or grouping related ones)?
2. For libraries that already have instrumentations in this repo (e.g., langchain, claude-agent-sdk), should we propose enhancements to existing packages or submit as alternatives?
3. Are there any specific requirements for evaluation/benchmark-focused instrumentations vs. production-library instrumentations?

[eternalcuriouslearner]

Howdy @123liuziming Thanks a lot for this contribution. We are currently working on porting openinference's instrumentation. Having said that priority 1 list seems solid except for langgraph as we need to figure out how to port that. Answers to you questions:

1. Is there a preference for batching PRs (e.g., one PR per instrumentation, or grouping related ones)? I am probably leaning towards creating base structures(adding required typechecker, gh actions etc) as first pr followed by small prs which can help make review easier(Imo this helps reduce the cognitive load). For example for agent framework one pr for create-agent, invoke-agent etc. Lmk if this works for you or not.
2. For libraries that already have instrumentations in this repo (e.g., langchain, claude-agent-sdk), should we propose enhancements to existing packages or submit as alternatives? Currently langchain and claude-agent-sdk are being evaluated from openinference side. So please give us sometime to get back on this. But I am leaning towards using the current repo's because it will make implementation faster and easier.
3. Are there any specific requirements for evaluation/benchmark-focused instrumentations vs. production-library instrumentations? Not sure on this we need to discuss this in the SIG.

Open Questions:

1. Steps to accept this donation. Probably @lmolkova and @trask can shed more light on the process.
2. I am not sure about agent evaluations and benchmarks as I haven't seen semantic conventions related to these. If you can help elaborate more on this it will be really helpful. Please mention this in the SIG for getting more traction around this.

Lmk if this helps and if you have any more questions.

[123liuziming]

Howdy @123liuziming Thanks a lot for this contribution. We are currently working on porting openinference's instrumentation. Having said that priority 1 list seems solid except for langgraph as we need to figure out how to port that. Answers to you questions:

1. Is there a preference for batching PRs (e.g., one PR per instrumentation, or grouping related ones)? I am probably leaning towards creating base structures(adding required typechecker, gh actions etc) as first pr followed by small prs which can help make review easier(Imo this helps reduce the cognitive load). For example for agent framework one pr for create-agent, invoke-agent etc. Lmk if this works for you or not.
2. For libraries that already have instrumentations in this repo (e.g., langchain, claude-agent-sdk), should we propose enhancements to existing packages or submit as alternatives? Currently langchain and claude-agent-sdk are being evaluated from openinference side. So please give us sometime to get back on this. But I am leaning towards using the current repo's because it will make implementation faster and easier.
3. Are there any specific requirements for evaluation/benchmark-focused instrumentations vs. production-library instrumentations? Not sure on this we need to discuss this in the SIG.

Open Questions:

1. Steps to accept this donation. Probably @lmolkova and @trask can shed more light on the process.
2. I am not sure about agent evaluations and benchmarks as I haven't seen semantic conventions related to these. If you can help elaborate more on this it will be really helpful. Please mention this in the SIG for getting more traction around this.

Lmk if this helps and if you have any more questions.

As for existing instrumentation in the community, there may be no immediate need to contribute them to upstream. We may prioritize contributing agent frameworks—such as qwen-agent and qwenpaw—that are not yet covered by the community. Support for benchmark-type frameworks will depend on whether there is a corresponding need within the community. As for the contribution process, we will submit a separate PR for each piece of instrumentation.