test(ffe): add agentless end-to-end harness#7298
Conversation
|
|
|
leoromanovsky
requested review from
ZStriker19,
claponcet,
jandro996,
manuel-alvarez-alvarez and
ncybul
and removed request for
a team
leoromanovsky
requested review from
dubloom,
pavlokhrebto,
sameerank and
typotter
and removed request for
a team and
pavlokhrebto
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9af665a8a9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eb57dc60db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "attributes": {}, | ||
| }, | ||
| ) | ||
| self.backend_status = scenarios.feature_flagging_and_experimentation_agentless.mock_backend_status() |
There was a problem hiding this comment.
Fresh evidence after the prior thread: the current code again calls mock_backend_status() from setup_default_agentless_source. That helper performs a /status request and MockFFEAgentlessBackendServer.status() raises on non-2xx responses, so a transient mock-control failure aborts the scenario as a setup error before test_default_agentless_source can report it as an ordinary test failure; keep setup to triggering /ffe and fetch/assert the backend status in the test method.
Useful? React with 👍 / 👎.
| default_libs_with_dev, | ||
| 3600, | ||
| "false", | ||
| "DEFAULT,FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS", |
There was a problem hiding this comment.
When this test file changes, this new expectation makes the CI selector emit FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS, and compute-workflow-parameters schedules it as end-to-end work; however I checked .github/workflows/run-end-to-end.yml:320-322 and the workflow only has the existing FEATURE_FLAGGING_AND_EXPERIMENTATION run step, with no AGENTLESS step found. Those jobs can build weblogs and then skip every scenario run step, so PRs changing or enabling this test get green end-to-end jobs without running the selected scenario; either keep the dormant scenario out of the selector or add the matching run step.
Useful? React with 👍 / 👎.
cbeauchesne
left a comment
cbeauchesne
left a comment
There was a problem hiding this comment.
Great job, congrats. I've few request changes and question. There are also two missing bits :
- add rules in
utils/scripts/libraries_and_scenarios_rules.ymlfor this new scenario - add the scenario in the CI
| ) | ||
| from utils._context.weblog_infrastructure import EndToEndWeblogInfra | ||
| from utils.docker_fixtures._core import extra_hosts_for_environment | ||
| from utils.docker_fixtures._mock_ffe_agentless_backend import ( |
There was a problem hiding this comment.
There is a mechanism in our CI that control which scenario is executed based on changes. As now, utils.docker_fixtures triggers a change only on INTEGRATION_FRAMEWORKS and PARAMETRIC scenarios.
As _mock_ffe_agentless_backend will be a utilitity of both scenario, we must move this module in an dedicated module.
I also feel that it should be generic, as I expect FFE won't be the only feature to use it.
WDYT about utils.mocked_backend ? For the CI orchestrator, it happens here : utils/scripts/libraries_and_scenarios_rules.yml
| super().start_interfaces_watchdog( | ||
| [interfaces.library, interfaces.agent] | ||
| [interfaces.library] | ||
| + agent_interfaces |
There was a problem hiding this comment.
The type of agent_interfaces is list | None, if I'm not wrong, you can't add None to a list here. The fox is just above, by setting [] if include_agent==False
| ) | ||
|
|
||
| self.weblog_infra.stop() | ||
| self.weblog_infra.stop(flush=self._flush_weblog_on_stop) |
There was a problem hiding this comment.
Why do you need to disable the flush here ?
|
|
||
|
|
||
| @scenarios.test_the_test | ||
| @features.not_reported |
There was a problem hiding this comment.
You don't need to add @features.not_reported here, same goes for other test_the_test files
| client_drop_p0s: bool | None = None, | ||
| iast_enabled: bool = True, | ||
| include_agent: bool = True, | ||
| include_default_scenario_groups: bool = True, |
There was a problem hiding this comment.
IIUC, it's a temporary situation? If yes, could you rather force the groups in FeatureFlaggingAgentlessEndToEndScenario.__init__, rather adding an argument to the constructor? (it's already too big).
| include_agent=False, | ||
| include_default_scenario_groups=False, | ||
| flush_weblog_on_stop=False, | ||
| library_interface_timeout=0, | ||
| scenario_groups=[scenario_groups.ffe], | ||
| use_proxy_for_agent=False, | ||
| use_proxy_for_weblog=False, |
There was a problem hiding this comment.
If ever some of those value will be constant for the majority of FeatureFlaggingAgentlessEndToEndScenario, it's better to set the directly inside FeatureFlaggingAgentlessEndToEndScenario constructor, or inside FeatureFlaggingAgentlessEndToEndScenario construtor prototype as optional argument with defaults.
It keeps this file shorter and easier to maintain.
|
|
||
| def configure(self, config: pytest.Config) -> None: | ||
| try: | ||
| if not self.replay: |
There was a problem hiding this comment.
I'm not sure it's usefull, we never use xdist with end-to-end scenario.
2 participants
Motivation
Server SDK Feature Flags are moving to authenticated CDN delivery as the default configuration path, with Agent Remote Configuration remaining an explicit alternative. The existing FFE end-to-end scenario can exercise only the Agent/RC lifecycle: it starts an Agent and weblog, then supplies UFC after the application is running.
That cannot prove the agentless startup contract. An agentless SDK may request UFC while the application is starting, so the test backend must already exist before the weblog starts. The scenario must also run without an Agent; otherwise a passing test cannot rule out an accidental Agent dependency.
This PR adds that missing, dormant test foundation. It starts a deterministic CDN-style UFC backend before a no-Agent weblog, injects the endpoint and credentials, and verifies authenticated configuration fetch plus evaluation. No SDK manifest is enabled here, so ordinary SDK CI behavior does not change until a later activation PR.
flowchart LR subgraph BEFORE["Existing end-to-end coverage"] B1["Start Agent and weblog"] --> B2["SDK starts"] B3["RC fixture"] --> B4["Agent Remote Configuration"] B4 --> B2 B2 --> B5["Evaluate flag"] end subgraph THIS["Agentless harness added here"] A1["1. Start mock CDN backend"] --> A2["2. Start weblog and SDK<br/>without an Agent"] A2 --> A3["3. Fetch UFC with API key"] A3 --> A4["4. Evaluate flag"] end GAP["Missing on main:<br/>pre-start UFC service<br/>and no-Agent lifecycle"] -.-> A1 classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class A1,A2,A3,A4 current;Stack Position
This is the inactive foundation of the stack. Later PRs add the telemetry topology and then activate individual Java capabilities only when their implementation is green.
flowchart TD ST1["#7298 · this PR<br/>no-Agent CDN configuration harness<br/>all manifests disabled"] ST2["#7299 · next stacked PR<br/>side-effect transport contracts<br/>sidecar preferred, direct fallback"] STM["#7300 and small follow-ups<br/>enable Java capabilities one at a time"] J1["dd-trace-java #11892<br/>agentless configuration source"] J2["dd-trace-java side-effects work<br/>exposures · evaluation EVP · OTLP"] ST1 --> ST2 --> STM J1 --> STM J2 --> STM classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class ST1 current;Architecture Boundary
Configuration source and telemetry transport are independent. The Agent belongs only to the explicit Remote Configuration branch. Default agentless configuration has no Agent dependency. Post-evaluation side effects are a stacked follow-up: prefer the Datadog serverless sidecar when available and fall back to direct authenticated intake otherwise.
flowchart LR subgraph CONFIG["Configuration delivery"] RC["RC fixture"] --> RCA["Agent<br/>RC only"] RCA --> RCS["SDK remote_config source"] CDN["Datadog CDN<br/>or controlled mock"] -->|"authenticated HTTP"| AGS["SDK agentless source<br/>default"] RCS --> EVAL["Same UFC / OpenFeature evaluator"] AGS --> EVAL end subgraph FOLLOWUP["Telemetry transport · stacked follow-up"] EVAL --> FX["Exposure EVP · evaluation EVP · OTLP metrics"] FX --> PICK{"Serverless sidecar available?"} PICK -->|"yes"| SIDECAR["Datadog serverless sidecar"] PICK -->|"no"| DIRECT["Direct authenticated intake"] SIDECAR --> INTAKE["Datadog backend"] DIRECT --> INTAKE end classDef current fill:#fcbf49,stroke:#8a5a00,stroke-width:3px,color:#111; class CDN,AGS,EVAL current;Changes
ETag/If-None-Match, and metadata-only status inspection.FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESSscenario that starts the backend before the weblog.all,end_to_end, andtracer_releaseselections.Decisions
Validation
./format.sh --check— passes.java@1.65.0-SNAPSHOT+f9e70e67c4— 1 passed in 2.11s.Next Steps
interfaces.agent.