chore(antithesis): non-utf8 bytes in datagrams represent ~1% of generation by blt · Pull Request #2217 · DataDog/saluki

blt · 2026-07-27T22:59:58Z

Summary

ADP and Datadog Agent deal with non-utf8 bytes differently. The way that
intake accepts non-utf8 bytes varies depending on where in the payload they
appear and for which protocol version. Previously I created datagrams that were
too non-utf8 heavy, meaning most payloads could be rejected by intake. This
commit changes the generator to emit non-utf8 bytes into about 1% of datagrams.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

How did you test this PR?

References

blt · 2026-07-27T23:00:11Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

pr-commenter · 2026-07-27T23:07:57Z

Binary Size Analysis (Agent Data Plane)

Baseline: 7f0ba13 · Comparison: 0f55ac4 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.60 MiB (baseline) vs 41.60 MiB (comparison)
Size Change: +0 B (+0.00%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`anon.44a0faa3240fa409baa379409d152aec.1.llvm.17202424994989785089`	-130 B	1
`anon.44a0faa3240fa409baa379409d152aec.1.llvm.1357467956718404416`	+129 B	1
`anon.44a0faa3240fa409baa379409d152aec.4.llvm.17202424994989785089`	-114 B	1
`anon.44a0faa3240fa409baa379409d152aec.4.llvm.1357467956718404416`	+113 B	1
`anon.44a0faa3240fa409baa379409d152aec.3.llvm.17202424994989785089`	-109 B	1
`anon.44a0faa3240fa409baa379409d152aec.3.llvm.1357467956718404416`	+108 B	1
`anon.44a0faa3240fa409baa379409d152aec.0.llvm.17202424994989785089`	-97 B	1
`anon.44a0faa3240fa409baa379409d152aec.0.llvm.1357467956718404416`	+96 B	1
`anon.44a0faa3240fa409baa379409d152aec.2.llvm.17202424994989785089`	-95 B	1
`anon.44a0faa3240fa409baa379409d152aec.2.llvm.1357467956718404416`	+94 B	1
`[Unmapped]`	+5 B	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW]    +129  [NEW]     +40    anon.44a0faa3240fa409baa379409d152aec.1.llvm.1357467956718404416
  [NEW]    +113  [NEW]     +24    anon.44a0faa3240fa409baa379409d152aec.4.llvm.1357467956718404416
  [NEW]    +108  [NEW]     +19    anon.44a0faa3240fa409baa379409d152aec.3.llvm.1357467956718404416
  [NEW]     +96  [NEW]      +7    anon.44a0faa3240fa409baa379409d152aec.0.llvm.1357467956718404416
  [NEW]     +94  [NEW]      +5    anon.44a0faa3240fa409baa379409d152aec.2.llvm.1357467956718404416
   +10%      +5  [ = ]       0    [Unmapped]
  [DEL]     -95  [DEL]      -5    anon.44a0faa3240fa409baa379409d152aec.2.llvm.17202424994989785089
  [DEL]     -97  [DEL]      -7    anon.44a0faa3240fa409baa379409d152aec.0.llvm.17202424994989785089
  [DEL]    -109  [DEL]     -19    anon.44a0faa3240fa409baa379409d152aec.3.llvm.17202424994989785089
  [DEL]    -114  [DEL]     -24    anon.44a0faa3240fa409baa379409d152aec.4.llvm.17202424994989785089
  [DEL]    -130  [DEL]     -40    anon.44a0faa3240fa409baa379409d152aec.1.llvm.17202424994989785089
  [ = ]       0  [ = ]       0    TOTAL

datadog-prod-us1-6

More details

The new injection is bounded to one byte per datagram, preserves the existing size and forwarding invariants, and excludes event and service-check lines from metric-name mutation. No diff-only behavioral regression was identified; runtime confirmation was blocked by unavailable Rust dependencies in the sandbox.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 43cf97d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

pr-commenter · 2026-07-27T23:27:32Z

Regression Detector (Agent Data Plane)

Run ID: 3e52bfc8-e047-46eb-b816-d091ac5c30ed
Baseline: 7f0ba132 · Comparison: 0f55ac4b · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_medium	memory	⚪ +0.33	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.08	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.07	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.04	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.14	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 143 MiB ≤ 145 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 49.3 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 77.8 MiB ≤ 80 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 202 MiB ≤ 205 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 31.8 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 43cf97d287

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 793eced78f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da6321ea36

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c01556e0e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d9441efaee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f670417223

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f6682ebdc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d1cf982d9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The essential property we want to determine is whether ADP-on/ADP-off emits payloads to intake API that the intake API rejects. This is especially of interest for inputs that we know are ultimately rejected by intake API -- non-utf8 bytes in the wrong spot -- but are _not_ rejected by the SUT. The old mechanism had a feral/clean 'vibe' which served for a while but was confusing to debug. Is feral malformed? Is feral well-formed but wild? Anyway I got tired of it. There's now a predicate which defines whether a payload is well-formed or not -- that is, accepted by ADP-off Datadog Agent -- independent of whether intake API ultimately accepts the payloads that ingress inspires. Later I will build a generator that only emits malformed ingress but that is only hinted at in this work.

tobz · 2026-08-03T18:46:50Z

 /// The per-kind caps a timeline's shared context pool fills to before it recurs existing contexts.
 /// `first_sample_config` samples this beside `datadog.yaml` so cardinality varies per timeline. Each
 /// cap is drawn against the budget the earlier draws left, so a kind's cardinality still varies at
 /// random while the three together stay under [`MAX_CONTEXTS_TOTAL`].


Same deal here as the other PR: we should fix this, so things pass... and I'll look at exclude these crates out-of-band.

tobz

Woops, did not mean to approve yet. 😅

misclick

This was referenced Jul 27, 2026

chore(antithesis): Intro Intake V3 #2214

Merged

chore(antithesis): Alter dogstatsd generation to is_malformed regime #2215

Open

This was referenced Jul 27, 2026

chore(antithesis): Bound contexts in runs #2216

Open

chore(antithesis): Add provider_kind in sample config #2218

Open

dd-octo-sts Bot added the area/test All things testing: unit/integration, correctness, SMP regression, etc. label Jul 27, 2026

This comment has been minimized.

Sign in to view

blt changed the title ~~chore(antithesis): is_malformed based load generation~~ chore(antithesis): non-utf8 bytes payloads represent 1% of generation Jul 27, 2026

blt marked this pull request as ready for review July 27, 2026 23:24

blt requested review from a team as code owners July 27, 2026 23:24

datadog-prod-us1-6 Bot reviewed Jul 27, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs Outdated

blt mentioned this pull request Jul 28, 2026

chore(antithesis): Frechet Distance and intake rework #2228

Open

4 tasks

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch from 34de63f to a827899 Compare July 29, 2026 17:15

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from 43cf97d to 86e0959 Compare July 29, 2026 17:15

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch from a827899 to e45d94e Compare July 29, 2026 20:37

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from 86e0959 to 793eced Compare July 29, 2026 20:37

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/payload/dogstatsd.rs Outdated

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch from e45d94e to cb1f9df Compare July 29, 2026 20:50

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch 2 times, most recently from 2a19b2e to 74d04b8 Compare July 29, 2026 21:18

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch from cb1f9df to 639f6b7 Compare July 29, 2026 21:18

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from 74d04b8 to fdea3d5 Compare July 29, 2026 22:42

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch 2 times, most recently from 39f3b2a to a8467bf Compare July 29, 2026 22:55

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from fdea3d5 to 52ca146 Compare July 29, 2026 22:55

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch from a8467bf to 75b7b70 Compare July 29, 2026 23:00

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch 2 times, most recently from f3e05f8 to da6321e Compare July 30, 2026 15:24

chatgpt-codex-connector Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/contexts/event.rs Outdated

Comment thread test/antithesis/harness/src/payload/dogstatsd.rs Outdated

Comment thread test/antithesis/harness/src/payload/dogstatsd.rs Outdated

Comment thread test/antithesis/harness/src/contexts/metric.rs Outdated

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from da6321e to c01556e Compare July 30, 2026 15:57

chatgpt-codex-connector Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from c01556e to d9441ef Compare July 30, 2026 20:40

chatgpt-codex-connector Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/payload/dogstatsd.rs Outdated

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from d9441ef to f670417 Compare July 30, 2026 21:05

chatgpt-codex-connector Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/payload/dogstatsd.rs Outdated

Comment thread test/antithesis/harness/src/payload/dogstatsd.rs Outdated

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from f670417 to 9f6682e Compare July 30, 2026 21:19

chatgpt-codex-connector Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread test/antithesis/intake/src/context_pool.rs Outdated

blt changed the title ~~chore(antithesis): non-utf8 bytes payloads represent 1% of generation~~ chore(antithesis): non-utf8 bytes in datagrams represent ~1% of generation Jul 30, 2026

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch 6 times, most recently from 7fc724f to 8d1cf98 Compare July 31, 2026 18:26

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch from 966965a to 2c757ed Compare July 31, 2026 18:26

chatgpt-codex-connector Bot reviewed Jul 31, 2026

View reviewed changes

Comment thread test/antithesis/harness/src/payload/dogstatsd.rs

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from 8d1cf98 to 5f2c894 Compare July 31, 2026 22:51

blt force-pushed the blt/chore_antithesis_bound_contexts_in_runs branch 2 times, most recently from 3906c44 to 085e47a Compare August 3, 2026 18:35

blt force-pushed the blt/chore_antithesis_is_malformed_based_load_generation branch from 5f2c894 to 0f55ac4 Compare August 3, 2026 18:35

tobz previously approved these changes Aug 3, 2026

View reviewed changes

tobz reviewed Aug 3, 2026

View reviewed changes

webern approved these changes Aug 5, 2026

View reviewed changes

Uh oh!

Conversation

blt commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

How did you test this PR?

References

Uh oh!

blt commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

pr-commenter Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

datadog-prod-us1-6 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pr-commenter Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

tobz Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

tobz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

blt commented Jul 27, 2026 •

edited

Loading

blt commented Jul 27, 2026 •

edited

Loading

pr-commenter Bot commented Jul 27, 2026 •

edited

Loading

pr-commenter Bot commented Jul 27, 2026 •

edited

Loading