fix(soak): repair OTel v0.151.0 soak failures (collector Telemetry factory, go/nodejs trace validation, Go CI) - #1138
Merged
Conversation
PR #1135 bumped Go from 1.24.0 to 1.26.2 in go.mod files (adot/collector/lambdacomponents, opentelemetry-lambda submodule). The GitHub Actions workflows still referenced Go ~1.24.3 via actions/setup-go, causing build/runtime failures in the soak tests because the installed Go toolchain was too old for the go directive. Update all 8 workflow files to use go-version: '~1.26.2' to match the project's go.mod requirements.
…0.151.0 The OTel Collector v0.151.0 submodule bump introduced two nodejs-only breakages in the soak/integration tests: 1. Terraform "Unsupported argument": upstream nodejs/sample-apps/aws-sdk/deploy/wrapper was refactored to build layer ARNs internally from published layer versions and no longer accepts collector_layer_arn / sdk_layer_arn. Rework the integration test to instantiate terraform-aws-modules/lambda/aws + api-gateway-proxy directly and attach the locally-built layers, mirroring the python test. Go and Java deploy modules still accept the ARN inputs and are unchanged. 2. TraceValidator "data model not matched": the upstream nodejs sample app switched its AWS call from S3 ListBuckets to STS GetCallerIdentity. Update the expected trace template accordingly. Java/Go samples still use S3. sim: https://taskei.amazon.dev/tasks/PROMET-11975
…v0.151.0
The collector v0.151.0 bump surfaced a runtime INIT/invoke failure across soak jobs:
- java-agent: FUNCTION_ERROR_INIT_FAILURE during provisioned-concurrency warm-up
(surfaces as the aws_lambda_provisioned_concurrency_config waiter erroring)
- go/python/java wrappers: sample app returns HTTP 500, so no trace is produced
and TraceValidator reports "data model not matched"
Both paths share the collector extension. The ADOT config override retained
`service.telemetry.metrics.address: localhost:8888`, a field deprecated in
collector v0.123.0 and dropped by upstream in the v0.151.0 default config.
Upstream opentelemetry-collector issue #3911 documents this field causing port
conflicts and collector crashes, which would crash the Lambda extension and
produce the observed INIT/invoke failures.
Remove the telemetry block from the default ADOT collector config and the
java-agent integration test's inline config to realign with upstream v0.151.0.
NOT YET VERIFIED: soak cannot be run locally (deploys real Lambdas) and the
failing run's CloudWatch logs were already torn down. A soak run is required
to confirm this resolves the INIT failures.
sim: https://taskei.amazon.dev/tasks/PROMET-11975
…151.0
v0.151.0 made otelcol.Factories.Telemetry a required field. The ADOT
lambdacomponents builder (adot/collector/lambdacomponents/default.go),
which patch-upstream.sh copies over the submodule, did not set it, so the
collector extension failed at INIT ("otelcol.Factories.Telemetry must not
be nil") -> Extension.InitError on every invoke -> instrumented Lambdas
returned HTTP 500 with no trace in soak (go/python/java-wrapper) and
java-agent FUNCTION_ERROR_INIT_FAILURE.
Set Telemetry: otelconftelemetry.NewFactory(), mirroring upstream
lambdacomponents/custom.go, and add the collector/service require via
go mod tidy.
Ref: V2153606376
…ming The rebuilt go sample app (opentelemetry-lambda submodule @ v0.151.0) emits new span names via updated otelaws/otelhttp instrumentation, which flow to X-Ray inferred-segment names: - S3 AWS SDK call: inferred segment S3 -> S3.ListBuckets - HTTP client call: inferred segment api.github.com -> HTTP GET The inferred S3 segment no longer carries origin/aws.operation, so those fields are dropped. Regenerated from the real trace in soak run 29127081461.
The nodejs aws-sdk sample app returned a caller-identity JSON payload, so the soak trace validator could not extract a trace ID (trace_id=null) and never fetched the trace, failing with 'data model not matched'. Mirror the go/python sample apps: still call STS GetCallerIdentity (to emit the STS segment) but return process.env._X_AMZN_TRACE_ID as the response body. Placed under adot/ so patch-upstream.sh (cp -rf adot/* opentelemetry-lambda/) overrides the submodule sample app at build time.
The soak TraceValidator matches strictly by flattened index key ([N].name), and orders actual segments by start_time. In the v0.151.0 trace the inferred S3.ListBuckets segment sorts to index [3] (between the instrumented segment and the Lambda::Function segment), not [4]. The prior fix renamed correctly but left S3 at [4], colliding stored [3]=lambda-go.* with actual [3]=S3.ListBuckets. Reorder: S3.ListBuckets at [3], lambda-go.* at [4].
Temporary [ADOT-DEBUG] diagnostics to localize why AWS SDK calls are not traced in wrapper mode (PROMET-11975). Logs: (1) static block = SPI class loaded, (2) constructor = captured GlobalOpenTelemetry + TracerProvider class, (3) beforeExecution = interceptor actually invoked per SDK call. Distinguishes 'never registered' vs 'noop OTel' vs 'not attached' vs 'span dropped'. Revert before release.
Interceptor is confirmed registered/attached with a real tracer, yet no S3 span lands in the Lambda X-Ray trace. Add active SpanContext dump (valid/ sampled/traceId) at beforeExecution to test the context-propagation hypothesis (orphaned root span). Revert before release.
…on [TEMP]" This reverts commit 2e85c32.
…tor [TEMP]" This reverts commit ade4b80.
priyankaDhingra
changed the title
priyankaDhingra
marked this pull request as ready for review
July 14, 2026 19:30
alexperez52
approved these changes
Jul 14, 2026
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The OTel Collector
v0.138.0 → v0.151.0+ Go1.24 → 1.26bump (CVE remediation, ships ADOT Collector v0.48.0) broke the Lambda soak tests. This PR repairs the failures that are caused by the bump itself and updates the soak expectations to the v0.151.0 trace/span conventions.Changes
Collector: set the required
Telemetryfactory (root cause of the INIT failures)OTel Collector v0.151.0 made
otelcol.Factories.Telemetrya required field. The ADOT overrideadot/collector/lambdacomponents/default.godid not set it, so the collector extension failed at INIT (otelcol.Factories.Telemetry must not be nil→Extension.InitError). This caused every instrumented job to fail — wrappers returned HTTP 500 (no trace) and java-agent hitFUNCTION_ERROR_INIT_FAILURE. Now mirrors upstreamcustom.go(Telemetry: otelconftelemetry.NewFactory()).Collector: remove deprecated
service.telemetry.metrics.addressDropped for v0.151.0 (deprecated since collector v0.123.0), aligning with upstream.
CI: bump Go
1.24.3 → 1.26.2across the workflows.go-wrapper expected trace: update for v0.151.0 span/segment conventions
The rebuilt sample app emits new inferred X-Ray segment names via the updated instrumentation:
S3→S3.ListBucketsandapi.github.com→HTTP GET. Also corrected the segment order — the validator matches by flattened index and orders actual segments by start_time, which places the inferredS3.ListBucketsat index[3].nodejs sample app: return
_X_AMZN_TRACE_IDThe nodejs aws-sdk sample app returned a caller-identity JSON payload, so the validator could not extract a trace ID (
trace_id=null) and never fetched the trace. It now returns the trace header (mirroring the go/python sample apps) while still making the instrumented STS call. Delivered as anadot/override thatpatch-upstream.shapplies to the submodule.Validation
Soak runs (
-t 1800, us-east-1) confirm the fixes —validate trace samplepasses for:Known issue — java-wrapper (tracked separately, NOT fixed here)
java-wrapperstill failsvalidate trace sample, but the root cause is an upstreamopentelemetry-lambdaregression, not this bump. Live-log diagnosis showed the AWS SDK auto-instrumentation works correctly (global interceptor registered, realTracerProvider,beforeExecutionfires, span createdvalid=true sampled=true), but the wrapper's OTel trace is a fresh root trace whose trace ID does not match the Lambda's X-Ray trace ID — so the SDK/handler spans land in a separate trace and are absent from the invocation's X-Ray trace that the validator inspects.This is a trace-context-linking defect in the Java Lambda wrapper under v0.151.0 (the javaagent path is unaffected). It is being tracked and escalated upstream separately, and is intentionally out of scope for this PR so the CVE remediation is not blocked. The
java-wrappersoak expectation is left unchanged (the check is legitimately catching a real trace-continuity regression and should not be weakened).Notes
[ADOT-DEBUG]diagnostic commits were used to root-cause the java-wrapper issue and have been reverted in this branch.