Skip to content

Wire OTLP Trace Metrics Writers into Metrics Aggregators#11680

Merged
mhlidd merged 13 commits into
mhlidd/otlp_trace_metrics_setupfrom
mhlidd/otlp_trace_metrics_factory_wiring
Jul 17, 2026
Merged

Wire OTLP Trace Metrics Writers into Metrics Aggregators#11680
mhlidd merged 13 commits into
mhlidd/otlp_trace_metrics_setupfrom
mhlidd/otlp_trace_metrics_factory_wiring

Conversation

@mhlidd

@mhlidd mhlidd commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

This PR wires the OTLP Trace Metrics logic and writers into the existing Metrics Aggregators used for Client Side Stats.

This involves:

  • Adding a new OTLP export variant for ConflatingMetricsAggregator which uses the OtlpStatsMetricWriter introduced in Define OTLP Trace Metrics Histogram and Writers #11656
  • Making native CSS metrics mutually exclusive w/ OTLP Trace Metrics
  • Dropping sampled traces when OTLP Trace Metrics is active
  • Adding the Datadog-Client-Computed-Stats header when OTLP Trace Metrics are active to prevent dual writing in the Agent
  • Adding package-private getters in ConflatingMetricsAggregator for testing purposes

Motivation

Additional Notes

Contributor Checklist

  • Format the title according to the contribution guidelines
  • Assign the type: and (comp: or inst:) labels in addition to any other useful labels
  • Avoid using close, fix, or any linking keywords when referencing an issue
    Use solves instead, and assign the PR milestone to the issue
  • Update the CODEOWNERS file on source file addition, migration, or deletion
  • Update public documentation with any new configuration flags or behaviors
  • Add your completed PR to the merge queue by commenting /merge. You can also:
    • Customize the commit message associated with the merge with /merge --commit-message "..."
    • Remove your PR from the merge queue with /merge -c
    • Skip all merge queue checks with /merge -f --reason "reason"; please use this judiciously, as some checks do not run at the PR-level (note: the PR still needs to be mergeable, this will only skip the pre-merge build)
    • Get more information in this doc

Jira ticket: [PROJ-IDENT]

@mhlidd
mhlidd changed the base branch from master to mhlidd/update_hostname_metric_resource_attributes June 18, 2026 21:59
@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.96 s 14.03 s [-1.2%; +0.1%] (no difference)
startup:insecure-bank:tracing:Agent 13.00 s 12.97 s [-0.4%; +0.9%] (no difference)
startup:petclinic:appsec:Agent 16.78 s 16.76 s [-1.1%; +1.2%] (no difference)
startup:petclinic:iast:Agent 16.86 s 16.88 s [-0.9%; +0.7%] (no difference)
startup:petclinic:profiling:Agent 16.58 s 16.72 s [-1.8%; +0.2%] (no difference)
startup:petclinic:sca:Agent 16.85 s 16.70 s [+0.0%; +1.8%] (maybe worse)
startup:petclinic:tracing:Agent 15.51 s 16.11 s [-7.8%; +0.4%] (no difference)

Commit: 0e5a05fb · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@mhlidd mhlidd changed the title Mhlidd/otlp trace metrics factory wiring Wire OTLP Trace Metrics Writers into Metrics Aggregators Jul 1, 2026
@mhlidd mhlidd added comp: metrics Metrics type: feature Enhancements and improvements labels Jul 1, 2026
@mhlidd
mhlidd marked this pull request as ready for review July 1, 2026 19:52
@mhlidd
mhlidd requested a review from a team as a code owner July 1, 2026 19:52
@mhlidd
mhlidd requested review from PerfectSlayer and mcculls July 1, 2026 19:52

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9be5584c2a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

return maps
}

def "Datadog-Client-Computed-Stats header set when either stats pipeline is enabled (otlpSpanMetrics=#otlpSpanMetrics, nativeMetrics=#nativeMetrics)"() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move the new stats-header coverage to JUnit 5

The root AGENTS.md says, "Always use JUnit 5" and "Do not write new Groovy / Spock tests"; this adds a new Spock feature method to DDAgentApiTest.groovy. Please move this coverage to a JUnit 5 test (or migrate the existing file) instead of adding new Groovy/Spock test code.

Useful? React with 👍 / 👎.

Comment thread dd-trace-core/src/main/java/datadog/trace/common/writer/WriterFactory.java Outdated
@mhlidd
mhlidd force-pushed the mhlidd/update_hostname_metric_resource_attributes branch from eefcc42 to a00f34b Compare July 2, 2026 21:11
@mhlidd
mhlidd force-pushed the mhlidd/otlp_trace_metrics_factory_wiring branch from 9be5584 to 3d2a088 Compare July 3, 2026 00:57
@mhlidd
mhlidd force-pushed the mhlidd/update_hostname_metric_resource_attributes branch from a00f34b to 484cafa Compare July 16, 2026 15:19
@mhlidd
mhlidd force-pushed the mhlidd/otlp_trace_metrics_factory_wiring branch from 3d2a088 to 8cd214f Compare July 16, 2026 17:24
Comment thread dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/DDAgentApi.java Outdated
Comment thread dd-trace-core/src/main/java/datadog/trace/common/writer/DDAgentWriter.java Outdated

static byte[] buildResourceMessage(Config config, boolean includeDatadogResourceAttributes) {
static byte[] buildResourceMessage(
Config config, boolean includeDatadogResourceAttributes, boolean includeStatsComputed) {

@mcculls mcculls Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of continually adding boolean flags which could become fragile (i.e. remembering the order of the flags) we could consider providing a Map<String, String> of additional attributes.

Then you could create a static initializer block under TRACE_RESOURCE_MESSAGE

For example, something like:

  Map<String, String> extraAttributes = new HashMap<>();
  if (Config.get().isTracesSpanMetricsEnabled()) {
    extraAttributes.put(STATS_COMPUTED_KEY, "true");
  }
  TRACE_RESOURCE_MESSAGE =
      buildResourceMessage(Config.get(), false, extraAttributes);

The buildResourceMessage method would then iterate over the extraAttributes entries, building them into the final resource payload.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! I replaced the static block w/ a helper function that get's called instead. Let me know if you feel strongly about using the static initializer block instead.

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments/improvements to be addressed - after that this can be merged into the parent PR for a final review of the overall picture.

@mhlidd
mhlidd requested review from a team as code owners July 17, 2026 17:56
@mhlidd
mhlidd requested review from mtoffl01 and removed request for a team July 17, 2026 17:56
Base automatically changed from mhlidd/update_hostname_metric_resource_attributes to mhlidd/otlp_trace_metrics_setup July 17, 2026 18:30

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The XOR logic in MetricsAggregatorFactory correctly gates OTLP vs native stats, the DroppingPolicy lambda captures the OTLP flag at construction time to always-drop p0 traces when active, and DDAgentApi's Datadog-Client-Computed-Stats header OR-combines both pipelines — matching the existing Config.get() call pattern already present for isApmTracingEnabled(). No behavioral regressions found across 10+ adversarial scenarios covering gRPC fallback extraction, service-name per-point logic, resource attribute selection, and factory selection XOR. Tests could not be executed due to a Java 25 toolchain requirement (only Java 8–21 available in the sandbox), so the analysis is static.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 0e5a05f · What is Autotest? · Any feedback? Reach out in #autotest

@mhlidd
mhlidd force-pushed the mhlidd/otlp_trace_metrics_factory_wiring branch from 0e5a05f to e70afbc Compare July 17, 2026 18:55
@mhlidd
mhlidd merged commit f89c28a into mhlidd/otlp_trace_metrics_setup Jul 17, 2026
47 of 173 checks passed
@mhlidd
mhlidd deleted the mhlidd/otlp_trace_metrics_factory_wiring branch July 17, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: metrics Metrics type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants