[TimPansino]

Background:

At newrelic we recently received bug reports of interoperability issues with our APM agent and other tracing vendors when combined with botocore and other tracing vendors.

W3C Trace Context headers are frequently shared between tracing vendors, and our agent inserts these headers after the request had already been signed by SigV4.

When paired with another tracing provider such as OpenTelemetry, it's possible these headers are injected before request signing as well. When these headers are included in the signature, it unfortunately conflicts with their designed purpose of being mutated throughout the request lifecycle to store up to date tracing information.

The issue is not limited to APM agents running on the same host, as these headers can also be mutated by any intermediary hops such as load balancers or proxies. If any of these run a W3C compliant tracing product, it's possible the headers will be mutated and break the signature.

The following are the W3C Trace Context headers that are designed to be mutated throughout the request lifecycle, with some examples as to why they may be mutated.

• tracestate headers will be mutated as tracing vendors add their own proprietary tracing information into this header's value. It can store multiple vendors' information blobs, so this header is typically added to throughout the request lifecycle.
• traceparent headers can be mutated to set or unset the sampled flag, or update trace IDs.
• baggage headers can contain any key-value pairs desired, and can be mutated at any time separately from and regardless of distributed tracing status.

Description of changes:

• Add W3C Trace Context headers to list of hop-by-hop headers that should not be signed (SIGNED_HEADERS_BLACKLIST), due to potential mutation by tracing libraries on the same host, or running on a proxy or load balancer.
• Add tests for these headers mirroring tests for existing headers.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[TimPansino]

Let me know if you'd like me to add a change record or open an issue on the repo as well.

[TimPansino]

Could I get a review on this from someone? @jonathan343 maybe? This should be a pretty easy change to review.

[alexgromero]

Hi @TimPansino, thanks for the writeup!

We're inclined to take this. Could you add two things?

1. baggage as well, with tests in the same style as the others. It's a separate spec from Trace Context so I can see why it wasn't in scope, but it can also be modified during propagation and is commonly injected alongside the other two.
2. A changelog entry, which you can generate by running scripts/new-change.

Also out of curiosity, are you seeing this with a particular service, or is it general to any signed request? Not a blocker either way, just useful context on our end.

Thanks again for the contribution!

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.50%. Comparing base (ee748e4) to head (365c246).
⚠️ Report is 620 commits behind head on develop.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3805      +/-   ##
===========================================
- Coverage    92.59%   92.50%   -0.10%     
===========================================
  Files           68       68              
  Lines        15634    15866     +232     
===========================================
+ Hits         14477    14677     +200     
- Misses        1157     1189      +32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TimPansino]

Hi @TimPansino, thanks for the writeup!

We're inclined to take this. Could you add two things?

1. baggage as well, with tests in the same style as the others. It's a separate spec from Trace Context so I can see why it wasn't in scope, but it can also be modified during propagation and is commonly injected alongside the other two.
2. A changelog entry, which you can generate by running scripts/new-change.

Also out of curiosity, are you seeing this with a particular service, or is it general to any signed request? Not a blocker either way, just useful context on our end.

Thanks again for the contribution!

I've updated the PR with the addition of baggage as a header, and added a new changelog entry using the script.

We received reports of issues connecting to any service at all when transport was configured to use SigV4 signing. It was only an issue if the user was using multiple sources of W3C headers, such as multiple tracing products. Unfortunately that included our hybrid New Relic Proprietary Agent + OpenTelemetry Instrumentation product offering.

The original report was actually due to the use of bedrock-agentcore and the new trace context header parameters introduced in botocore==1.43.52. The user was manually configuring their own W3C headers, which were later mutated by New Relic. Users not setting any W3C headers in the first place had no issues, as the headers were only generated after SigV4 signing occurred and were therefore excluded from the signature altogether.

Relevant Changelog Entry from 1.43.52:

* api-change:``bedrock-agentcore``: Add W3C trace context headers (traceparent, tracestate, baggage) and X-Amzn-Trace-Id to InvokeHarness request for end-to-end observability propagation. Add toolResultMetadata to the streaming content block delta for MCP tool result meta delivery without oversized SSE frames.

[alexgromero]

Thanks for the updates and the extra context! LGTM