test(ffe): cover billed source precedence by leoromanovsky · Pull Request #7332 · DataDog/system-tests

leoromanovsky · 2026-07-17T05:02:32Z

Motivation

Historically, a server-side tracer customer activated the Datadog OpenFeature provider with DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true. That input implicitly placed the customer on the Agent's FFE_FLAGS Remote Configuration path; there was no independent source-selection decision. Those users are an existing Remote Configuration cohort, not unconfigured new customers.

The new model introduces CDN-backed agentless delivery as the default for customers with no legacy setting, while preserving the existing cohort during a migration window. The distinction is subtle but essential: legacy true must still select Remote Configuration, legacy false must still disable Feature Flags, and an absent legacy value may select agentless. An explicit DD_FEATURE_FLAGS_CONFIGURATION_SOURCE overrides the legacy value, while DD_FEATURE_FLAGS_ENABLED=false overrides every source as the global kill switch.

Both Remote Configuration and CDN requests are billing surfaces. The test contract must therefore prove not only which source wins, but when traffic begins: explicit or grandfathered RC can subscribe eagerly, whereas default agentless must remain silent until application code crosses the provider-access boundary. Without cross-SDK tests for both counters, a precedence change could silently move existing customers to CDN or generate phantom billed traffic for tracer users who never use Feature Flags.

flowchart TD
  Start["Tracer or agent starts"] --> Enabled{"DD_FEATURE_FLAGS_ENABLED"}
  Enabled -->|"false"| Silent["No provider<br/>No RC<br/>No CDN"]
  Enabled -->|"true or unset"| Source{"Resolve explicit source, then legacy state"}
  Source -->|"explicit or grandfathered RC"| RC["Observe FFE_FLAGS capability and product"]
  Source -->|"agentless or new default"| Boundary["Assert zero startup requests"]
  Source -->|"legacy false, reserved offline, or invalid"| Silent
  Boundary --> StartFFE["POST /ffe/start"]
  StartFFE --> CDN["Observe mock CDN request"]
  RC --> Assert["Assert exactly one billed delivery path"]
  CDN --> Assert
  Silent --> Assert

Changes

flowchart LR
  Cases["19-case source-selection matrix"] --> Env["Stable, source, and legacy inputs"]
  Env --> SDK["Node.js or Java parametric implementation"]
  SDK --> Boundary["/ffe/start provider-access boundary"]
  Boundary --> Mock["Mock CDN request counter"]
  SDK --> Agent["Agent RC capability and product state"]
  Slow["Slow-response concurrency case"] --> Mock
  Generic["Generic /ffe/start parity"] --> SDK
  Dynamic["Generic RC capability test"] --> ExplicitRC["Explicit remote_config opt-in"]
  ExplicitRC --> Agent

Extend the mocked agentless backend with observable configuration requests.
Cover the canonical DD_FEATURE_FLAGS_ENABLED kill switch, default agentless activation, explicit billed-source opt-ins, the legacy migration window, and absent/empty/whitespace source equivalence.
Exercise explicit-source-over-legacy precedence and fail-closed behavior for invalid input and the reserved, currently unsupported offline value.
Prove that agentless polling serializes requests when a response lasts longer than the polling interval.
Make the Java and Node.js parametric controllers access the real provider for configuration-source tests while retaining the ready no-op provider for unrelated endpoint-parity tests.
Make the generic dynamic-configuration suite explicitly select remote_config when it validates the FFE_FLAGS Remote Configuration capability.

Decisions

DD_FEATURE_FLAGS_ENABLED=false is the first precedence rule and must suppress both CDN polling and the Feature Flags Remote Configuration subscription.
DD_FEATURE_FLAGS_ENABLED=true, or leaving it unset, does not itself generate traffic; it permits source resolution.
An explicit DD_FEATURE_FLAGS_CONFIGURATION_SOURCE wins over the legacy variable.
With no explicit source, legacy true selects Remote Configuration, legacy false disables Feature Flags, and an absent legacy variable selects default agentless delivery.
Default or explicit agentless delivery must remain silent until /ffe/start crosses the application provider-access boundary.
Explicit or grandfathered Remote Configuration must never fall back to CDN.
agentless and remote_config are the only supported sources today.
offline is reserved for future startup-provided UFC bytes. Until implemented, selecting it must fail closed with no CDN or RC traffic.
Assertions observe both billing surfaces: mock CDN requests and Agent Remote Configuration capabilities/products.

Validation

System-tests head: 2499246536723f242911097124aa251e078ddc5a.

The same 19 source-selection cases and one slow-poller concurrency case were executed against the exact stacked Node.js and Java implementation heads.

Node.js implementation

Validated local tracer head: f62b5ec43d35284b2d0508a00209437ee911a01e (committed locally; its separate Node.js PR push is pending).

binaries/nodejs-load-from-local mounted ../dd-trace-js-node-configuration-contract, whose checked-out head was the commit above.

PYTEST_XDIST_AUTO_NUM_WORKERS=12 TEST_LIBRARY=nodejs ./run.sh PARAMETRIC --skip-parametric-build \
  -F tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Selection \
  -F tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Poller_Concurrency::test_delayed_no_overlap \
  tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Selection \
  tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Poller_Concurrency::test_delayed_no_overlap

Result: 20 passed in 25.81s.

The new reserved-offline case was also run independently: 1 passed in 11.87s.

Java implementation

Validated tracer head: 7647879607f741c5961a450932c6a323689580a8.

The Java parametric image reported java@1.65.0-SNAPSHOT+7647879607, proving that it loaded the agent built from the commit above.

PYTEST_XDIST_AUTO_NUM_WORKERS=12 TEST_LIBRARY=java ./run.sh PARAMETRIC --skip-parametric-build \
  -F tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Selection \
  -F tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Poller_Concurrency::test_delayed_no_overlap \
  tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Selection \
  tests/parametric/test_ffe/test_configuration_sources.py::Test_Feature_Flag_Configuration_Source_Poller_Concurrency::test_delayed_no_overlap

Result: 20 passed in 35.23s.

The new reserved-offline case was also run independently: 1 passed in 13.08s.

Behavior proven by both SDK matrices

Case	Proven behavior
Explicit `remote_config`	Loads and acknowledges UFC, evaluates the expected value, advertises the `FFE_FLAGS` capability/product, and sends zero CDN requests.
Global `false` with explicit RC	Suppresses the RC capability/product and sends zero CDN requests.
RC with no payload	Crosses the provider-access boundary without falling back to CDN.
Absent, empty, or whitespace source with no legacy value	Sends zero Feature Flags traffic and advertises no RC capability at startup; after `/ffe/start`, evaluates the expected value and sends the authenticated request to the expected CDN path.
Stable `true` with no source	Preserves lazy default-agentless behavior instead of selecting RC.
Absent, empty, or whitespace source plus legacy `true`	Preserves Remote Configuration and sends zero CDN requests.
Global `false` with legacy `true`	The stable kill switch wins and suppresses both delivery paths.
Absent, empty, or whitespace source plus legacy `false`	Disables both delivery paths.
Explicit agentless plus legacy `true`	Explicit source wins; CDN starts only after provider access and RC remains off.
Explicit RC plus legacy `false`	Explicit source wins; RC is used and CDN remains off.
Global `false` with default agentless	Provider access cannot start CDN polling and RC remains off.
Invalid source	Fails closed with both delivery paths silent.
Reserved `offline` source	Remains unsupported and fails closed with both delivery paths silent, independently from an arbitrary invalid value.
Slow agentless response	A 1.5-second response against a 1-second poll interval produces repeated poll opportunities while `max_in_flight == 1`, proving requests do not overlap.

github-actions · 2026-07-17T05:02:56Z

CODEOWNERS have been resolved as:

tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/test_the_test/test_mock_ffe_agentless_backend.py                  @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/build/docker/java/parametric/src/main/java/com/datadoghq/trace/controller/FeatureFlagEvaluatorController.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/nodejs/parametric/server.js                          @DataDog/dd-trace-js @DataDog/system-tests-core
utils/docker_fixtures/_mock_ffe_agentless_backend.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

datadog-datadog-prod-us1-2 · 2026-07-17T13:06:03Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 2499246 | Docs | Datadog PR Page | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 965e96e071

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

typotter

overall. lgtm.
+1 to the bot's comments around coverage gaps.

sameerank

Agreeing with the codex comments, but looks good to me otherwise!

pr-shepherd-prod-7cf4a7 · 2026-07-20T23:04:25Z

PR Shepherd is now watching this PR: fixing basic CI failures, rebasing when it falls behind, and re-queueing after transient merge-queue failures. To disable it, add the pr-shepherd:ignore label.

If you have any questions, reach the team in #ai-devx-flow.

leoromanovsky · 2026-07-20T23:28:37Z

/remove

gh-worker-devflow-routing-ef8351 · 2026-07-20T23:28:39Z

View all feedbacks in Devflow UI.

2026-07-20 23:28:39 UTC ℹ️ Start processing command /remove

2026-07-20 23:28:42 UTC ℹ️ Devflow: /remove

leoromanovsky · 2026-07-20T23:35:35Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-20T23:35:39Z

View all feedbacks in Devflow UI.

2026-07-20 23:35:38 UTC ℹ️ Start processing command /merge

2026-07-20 23:35:48 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-07-20 23:59:11 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 2h (p90).

2026-07-21 00:11:10 UTC ℹ️ MergeQueue: This merge request was merged

leoromanovsky mentioned this pull request Jul 17, 2026

feat(openfeature): preserve configuration source semantics DataDog/dd-trace-js#9413

Closed

leoromanovsky changed the title ~~test(ffe): cover billed configuration source opt-ins~~ test(ffe): cover default agentless configuration delivery Jul 17, 2026

leoromanovsky changed the title ~~test(ffe): cover default agentless configuration delivery~~ test(ffe): cover billed source precedence Jul 18, 2026

test(ffe): cover billed source precedence

ca12c9a

leoromanovsky force-pushed the leo.romanovsky/ffl-2731-nodejs-configuration-contract branch from 993b2b1 to ca12c9a Compare July 20, 2026 17:11

test(ffe): explain configuration source assertions

965e96e

leoromanovsky commented Jul 20, 2026

View reviewed changes

Comment thread tests/parametric/test_ffe/test_configuration_sources.py

leoromanovsky marked this pull request as ready for review July 20, 2026 19:19

leoromanovsky requested review from a team as code owners July 20, 2026 19:19

leoromanovsky requested review from anna-git, claponcet, jandro996, mhlidd, pavlokhrebto, sameerank and typotter and removed request for a team July 20, 2026 19:19

chatgpt-codex-connector Bot reviewed Jul 20, 2026

View reviewed changes

typotter approved these changes Jul 20, 2026

View reviewed changes

Comment thread tests/parametric/test_ffe/test_configuration_sources.py

Comment thread tests/parametric/test_ffe/test_configuration_sources.py

Comment thread tests/parametric/test_ffe/test_configuration_sources.py

leoromanovsky added 3 commits July 20, 2026 15:00

test(ffe): verify kill switch overrides legacy opt-in

9538e95

test(ffe): cover explicit stable enable default

19c6d6b

test(ffe): exercise RC provider access without payload

65e8223

leoromanovsky added 2 commits July 20, 2026 15:10

test(ffe): observe past startup poll boundary

95966c8

test(ffe): exercise slow poll overlap boundary

f12f54f

sameerank approved these changes Jul 20, 2026

View reviewed changes

Comment thread tests/parametric/test_dynamic_configuration.py

Comment thread tests/parametric/test_ffe/test_configuration_sources.py

test(ffe): reserve offline fail-closed behavior

6cdb5b2

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: waiting label Jul 20, 2026

pr-shepherd-prod-7cf4a7 Bot added the pr-shepherd:in_progress label Jul 20, 2026

leoromanovsky mentioned this pull request Jul 20, 2026

Preserve Feature Flagging configuration source semantics DataDog/dd-trace-java#11992

Draft

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: removed and removed mergequeue-status: waiting labels Jul 20, 2026

test(ffe): cover blank configuration sources

2499246

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: removed mergequeue-status: waiting mergequeue-status: queued labels Jul 20, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 5464978 into main Jul 21, 2026
441 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Jul 21, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the leo.romanovsky/ffl-2731-nodejs-configuration-contract branch July 21, 2026 00:11

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Jul 21, 2026

anna-git removed their request for review July 21, 2026 09:48

Uh oh!

Conversation

leoromanovsky commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Validation

Node.js implementation

Java implementation

Behavior proven by both SDK matrices

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 17, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

typotter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sameerank left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pr-shepherd-prod-7cf4a7 Bot commented Jul 20, 2026

Uh oh!

leoromanovsky commented Jul 20, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leoromanovsky commented Jul 20, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leoromanovsky commented Jul 17, 2026 •

edited

Loading

github-actions Bot commented Jul 17, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 17, 2026 •

edited by datadog-official Bot

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 20, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 20, 2026 •

edited

Loading