test(ffe): add agentless side-effect routing contracts

[leoromanovsky]

Motivation

Agentless configuration delivery must not imply Agent-backed side effects. In the target serverless architecture, the SDK prefers a local Datadog serverless sidecar for exposure events, aggregate flag-evaluation events, and evaluation metrics; when no sidecar is available, it falls back to authenticated direct intake.

The previous version of this PR still asserted side effects through interfaces.agent. That preserved an Agent dependency in the very scenarios intended to prove no-Agent operation. This revision makes the selected telemetry route observable and fails on both missing delivery and delivery through the wrong route.

Stack Position

flowchart LR
    JAVA1["dd-trace-java #11892<br/>default CDN source"]
    ST1["system-tests #7298<br/>no-Agent configuration harness"]
    ST2["system-tests #7299<br/>sidecar/direct routing contracts"]
    STM0["system-tests #7300<br/>enable Java configuration manifest"]
    JAVAS["Java side-effect implementation<br/>sidecar preferred"]
    JAVAD["Java side-effect implementation<br/>direct fallback"]
    STMS["Small manifest PRs<br/>enable Java side-effect rows"]

    JAVA1 --> STM0
    ST1 --> STM0
    ST1 --> ST2
    ST2 --> JAVAS
    ST2 --> JAVAD
    JAVAS --> STMS
    JAVAD --> STMS

    classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111;
    class ST2 current;

Loading

Supported Architectures

Neither route contains a standalone Datadog Agent.

flowchart TB
    CDN1["Mocked CDN / UFC source"] --> SDK1["Server SDK"]
    SDK1 -->|"EVP: exposure + aggregate evaluation"| SIDECAR["Datadog serverless sidecar"]
    SDK1 -->|"OTLP: evaluation metrics"| SIDECAR
    SIDECAR --> SIDEINTAKE["Mocked Datadog intake<br/>interfaces.ffe_sidecar"]
    SDK1 -.->|assert no direct traffic| DIRECTTRIP["interfaces.ffe_direct<br/>must remain empty"]

    CDN2["Mocked CDN / UFC source"] --> SDK2["Server SDK"]
    SDK2 -->|"authenticated EVP + OTLP"| DIRECTINTAKE["Mocked Datadog intake<br/>interfaces.ffe_direct"]
    SDK2 -.->|assert no sidecar traffic| SIDETRIP["interfaces.ffe_sidecar<br/>must remain empty"]

    subgraph Preferred["Sidecar available: preferred route"]
        SDK1
        SIDECAR
        SIDEINTAKE
        DIRECTTRIP
    end

    subgraph Fallback["No sidecar: direct fallback"]
        SDK2
        DIRECTINTAKE
        SIDETRIP
    end

Loading

Changes

• Add dormant no-Agent scenarios for serverless-sidecar-preferred and direct-fallback telemetry.
• Run datadog/serverless-init as the sidecar conduit and capture sidecar and direct intake on separate proxy ports/interfaces.
• Normalize Datadog-series and direct OTLP metric payloads behind one route-aware telemetry interface.
• Rework the agentless exposure, aggregate evaluation EVP, and evaluation metric contracts so none imports or reads interfaces.agent.
• Assert both successful delivery through the selected route and absence of duplicate delivery through the other route, including replayed runs.
• Synchronize periodic metric export while the weblog is still alive instead of relying on teardown timing.
• Add test-the-test coverage for topology, authentication, scenario selection, replay routing, and OTLP normalization.
• Keep every new SDK manifest row marked missing_feature.

Decisions

• Configuration source and side-effect transport are orthogonal: UFC remains CDN-backed while telemetry independently chooses sidecar or direct delivery.
• auto means prefer the local serverless sidecar and use authenticated direct intake only when the sidecar is unavailable.
• Route provenance is part of the contract. Backend receipt alone is insufficient if the signal took the wrong path or was duplicated.
• The direct route must carry the configured Datadog API key; the sidecar owns backend authentication on the preferred route.
• This infrastructure PR stays green independently of unfinished SDK work by leaving manifests disabled.

Validation

• ./format.sh --check: pass, including mypy, Ruff, YAML, manifest parsers, shellcheck, and Node checks.
• Focused TEST_THE_TEST suite: 48 passed.
• Both new scenarios collect all three side-effect files: exposure EVP, aggregate flag-evaluation EVP, and evaluation metrics.
• Provisional integrated Java head ff586da612, forced past the dormant manifests:
  • Sidecar route: 3 passed live and 3 passed in replay; captures include /api/v2/exposures, /api/v2/flagevaluation, and /api/v2/series, with no direct-route capture.
  • Direct-fallback route: evaluation metrics pass through authenticated OTLP direct intake; exposure and aggregate flag-evaluation EVP produce 2 expected failures because Java does not implement their direct fallback yet.
• Full local TEST_THE_TEST: 336 passed, 1 xfailed. Its only 2 failures are the pre-existing JUnit golden comparison embedding this worktree's absolute path instead of a checkout named system-tests.

Next Steps

1. Implement Java's direct fallback for exposure and aggregate flag-evaluation EVP against these contracts.
2. Enable the corresponding Java manifest rows in small follow-up PRs once each route passes.
3. Keep test(ffe): enable Java agentless configuration #7300 limited to enabling the already-proven Java configuration-source contract.

[github-actions]

CODEOWNERS have been resolved as:

tests/ffe/test_agentless_exposures.py                                   @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/test_agentless_flag_eval_evp.py                               @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/test_agentless_flag_eval_metrics.py                           @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/utils/evaluation.py                                           @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/ffe/utils/telemetry.py                                            @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/test_the_test/test_feature_flag_telemetry.py                      @DataDog/system-tests-core
utils/interfaces/_feature_flag_telemetry.py                             @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
mirror_images.yaml                                                      @DataDog/system-tests-core
tests/test_the_test/scenarios.json                                      @DataDog/system-tests-core
tests/test_the_test/test_compute_libraries_and_scenarios.py             @DataDog/system-tests-core
tests/test_the_test/test_group_rules.py                                 @DataDog/system-tests-core
tests/test_the_test/test_mock_ffe_agentless_backend.py                  @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/interfaces/__init__.py                                            @DataDog/system-tests-core
utils/proxy/core.py                                                     @DataDog/system-tests-core
utils/proxy/ports.py                                                    @DataDog/system-tests-core

[datadog-datadog-us1-prod]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a4c87ca | Docs | Datadog PR Page | Give us feedback!}