Avoid retrying untracked Karate scenarios

[daniel-mohedano]

What Does This Do

Prevents retry handling for Karate setup scenarios and scenarios executed through a feature call. These internal scenarios do not create test spans.

It also guards standalone Karate 2 scenario execution, where no FeatureRuntime is available.

Motivation

Retry advice was still applied to scenarios that the tracing listener intentionally ignores. Since no test was started, finishing the scenario did not advance the execution policy. This could cause internal scenarios to be retried incorrectly.

For standalone Karate 2 scenarios, accessing the missing FeatureRuntime caused an NPE.

Additional Notes

Includes regression coverage for Karate 1 and Karate 2 setup and called scenarios, plus standalone Karate 2 execution.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

[cit-pr-commenter-54b7da]

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	56.00	55.43	$\color{red}{\blacktriangle}$ +0.57	55.43	$\color{red}{\blacktriangle}$ +0.57	56/262
agentEvpProxy	56.76	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	87.29	87.80	$\color{green}{\blacktriangledown}$ -0.51	87.80	$\color{green}{\blacktriangledown}$ -0.51	33/137
agentless	81.12	81.05	$\color{red}{\blacktriangle}$ +0.07	81.05	$\color{red}{\blacktriangle}$ +0.07	30/135
agentlessCodeCoverage	96.48	97.04	$\color{green}{\blacktriangledown}$ -0.56	97.04	$\color{green}{\blacktriangledown}$ -0.56	30/133
agentlessLineCoverage	112.98	111.62	$\color{red}{\blacktriangle}$ +1.36	111.62	$\color{red}{\blacktriangle}$ +1.36	29/132

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.77	36.42	$\color{red}{\blacktriangle}$ +0.35	36.42	$\color{red}{\blacktriangle}$ +0.35	30/132
agentless	36.28	36.42	$\color{green}{\blacktriangledown}$ -0.14	36.42	$\color{green}{\blacktriangledown}$ -0.14	29/131
agentlessCodeCoverage	43.92	44.48	$\color{green}{\blacktriangledown}$ -0.56	44.48	$\color{green}{\blacktriangledown}$ -0.56	29/131
agentlessLineCoverage	75.80	73.34	$\color{red}{\blacktriangle}$ +2.46	74.82	$\color{red}{\blacktriangle}$ +0.98	29/131

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	20.78	21.65	$\color{green}{\blacktriangledown}$ -0.87	21.65	$\color{green}{\blacktriangledown}$ -0.87	40/143
agentless	17.73	18.45	$\color{green}{\blacktriangledown}$ -0.72	18.82	$\color{green}{\blacktriangledown}$ -1.09	35/136
agentlessCodeCoverage	19.11	19.99	$\color{green}{\blacktriangledown}$ -0.88	19.99	$\color{green}{\blacktriangledown}$ -0.88	33/133
agentlessLineCoverage	28.97	29.23	$\color{green}{\blacktriangledown}$ -0.26	29.82	$\color{green}{\blacktriangledown}$ -0.85	31/131

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.18	9.54	$\color{green}{\blacktriangledown}$ -0.36	10.13	$\color{green}{\blacktriangledown}$ -0.95	26/124
agentless	13.31	9.73	$\color{red}{\blacktriangle}$ +3.58	10.33	$\color{red}{\blacktriangle}$ +2.98	26/124
agentlessCodeCoverage	14.24	15.72	$\color{green}{\blacktriangledown}$ -1.48	17.03	$\color{green}{\blacktriangledown}$ -2.79	25/122

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.71	9.54	$\color{red}{\blacktriangle}$ +0.17	9.54	$\color{red}{\blacktriangle}$ +0.17	28/69

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.96	13.13	$\color{green}{\blacktriangledown}$ -1.17	13.13	$\color{green}{\blacktriangledown}$ -1.17	32/128
agentless	11.44	12.12	$\color{green}{\blacktriangledown}$ -0.68	12.12	$\color{green}{\blacktriangledown}$ -0.68	30/125
agentlessCodeCoverage	14.60	15.11	$\color{green}{\blacktriangledown}$ -0.51	15.11	$\color{green}{\blacktriangledown}$ -0.51	30/125
agentlessLineCoverage	17.80	18.82	$\color{green}{\blacktriangledown}$ -1.02	19.20	$\color{green}{\blacktriangledown}$ -1.40	30/126

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	91.43	93.23	$\color{green}{\blacktriangledown}$ -1.80	93.23	$\color{green}{\blacktriangledown}$ -1.80	35/143
agentless	88.72	89.58	$\color{green}{\blacktriangledown}$ -0.86	89.58	$\color{green}{\blacktriangledown}$ -0.86	37/142
agentlessCodeCoverage	96.27	99.00	$\color{green}{\blacktriangledown}$ -2.73	99.00	$\color{green}{\blacktriangledown}$ -2.73	33/137
agentlessLineCoverage	98.58	99.00	$\color{green}{\blacktriangledown}$ -0.42	99.00	$\color{green}{\blacktriangledown}$ -0.42	31/133

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.94 s	13.94 s	[-0.6%; +0.6%] (no difference)
startup:insecure-bank:tracing:Agent	12.82 s	12.98 s	[-1.8%; -0.7%] (maybe better)
startup:petclinic:appsec:Agent	16.14 s	16.70 s	[-7.5%; +0.8%] (no difference)
startup:petclinic:iast:Agent	16.91 s	16.88 s	[-0.7%; +1.1%] (no difference)
startup:petclinic:profiling:Agent	16.15 s	16.76 s	[-7.9%; +0.7%] (no difference)
startup:petclinic:sca:Agent	16.87 s	16.82 s	[-0.8%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.09 s	16.11 s	[-1.2%; +1.0%] (no difference)

Commit: 42b93688 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[cit-pr-commenter-54b7da]

Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	18.83	19.59	$\color{green}{\blacktriangledown}$ -0.76	19.20	$\color{green}{\blacktriangledown}$ -0.37	30/137
agentless	19.22	19.20	$\color{red}{\blacktriangle}$ +0.02	19.20	$\color{red}{\blacktriangle}$ +0.02	30/134
agentlessCodeCoverage	22.41	22.54	$\color{green}{\blacktriangledown}$ -0.13	22.54	$\color{green}{\blacktriangledown}$ -0.13	30/133
agentlessLineCoverage	45.19	43.60	$\color{red}{\blacktriangle}$ +1.59	44.48	$\color{red}{\blacktriangle}$ +0.71	29/138

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	15.46	16.36	$\color{green}{\blacktriangledown}$ -0.90	16.36	$\color{green}{\blacktriangledown}$ -0.90	31/127
agentless	9.44	9.73	$\color{green}{\blacktriangledown}$ -0.29	9.73	$\color{green}{\blacktriangledown}$ -0.29	31/128
agentlessCodeCoverage	12.83	13.40	$\color{green}{\blacktriangledown}$ -0.57	13.40	$\color{green}{\blacktriangledown}$ -0.57	31/128
agentlessLineCoverage	33.06	32.30	$\color{red}{\blacktriangle}$ +0.76	32.95	$\color{red}{\blacktriangle}$ +0.11	29/126

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[cit-pr-commenter-54b7da]

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	39.82	8.13	$\color{red}{\blacktriangle}$ +31.69	12.87	$\color{red}{\blacktriangle}$ +26.95	34/135
agentless	39.10	11.42	$\color{red}{\blacktriangle}$ +27.68	15.72	$\color{red}{\blacktriangle}$ +23.38	34/134
agentlessCodeCoverage	66.99	82.69	$\color{green}{\blacktriangledown}$ -15.70	84.36	$\color{green}{\blacktriangledown}$ -17.37	34/134
agentlessLineCoverage	136.30	130.99	$\color{red}{\blacktriangle}$ +5.31	136.34	$\color{green}{\blacktriangledown}$ -0.04	31/131

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

[datadog-datadog-prod-us1]

More details

The fix correctly addresses two bugs: (1) in both karate-1.0 and karate-2.0, called/setup scenarios were incorrectly processed by the RetryAdvice, which could trigger spurious EFD retries since their ExecutionContext was created but no test span was started; (2) in karate-2.0, a standalone ScenarioRuntime (no FeatureRuntime) caused an NPE when skipTracking called getFeatureRuntime().getCaller(). The CallDepthThreadLocalMap invariants are preserved: skipped scenarios exit before incrementCallDepth, leaving the counter clean for the outer tracked scenario.

Was this helpful? React 👍 or 👎

_{📊 Validated against 6 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 42b9368 · What is Autotest? · Any feedback? Reach out in #autotest}