WIF-48: credential providers for process and orchestrator forwarders - #55564
Draft
wynbennett wants to merge 20 commits into
Draft
Conversation
Contributor
Go Package Import DifferencesBaseline: c35c49a
|
This comment has been minimized.
This comment has been minimized.
Contributor
Files inventory check summaryFile checks results against ancestor c35c49a9: Results for datadog-agent_7.84.0~devel.git.610.cc8a251.pipeline.134054635-1_amd64.deb:Detected file changes:
|
Contributor
Static quality checks❌ Please find below the results from static quality gates Error
Gate failure full details
Static quality gate failures prevent this PR from merging! Successful checksInfo
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: d725d90 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_memory | memory utilization | +1.53 | [+1.32, +1.74] | 1 | Logs |
| ➖ | quality_gate_private_action_runner | memory utilization | +1.13 | [+1.00, +1.25] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | +0.87 | [+0.82, +0.92] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_cpu | % cpu utilization | +0.85 | [+0.60, +1.09] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.78 | [+0.74, +0.81] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_metrics_logs | memory utilization | +0.74 | [+0.52, +0.96] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | +0.39 | [+0.35, +0.42] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | +0.38 | [+0.30, +0.45] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | +0.27 | [+0.23, +0.30] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | -4.50 | [-5.35, -3.65] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 173.79MiB ≤ 179MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 751.95KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 528.68MiB ≤ 537MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.15MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 19 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 212.75MiB ≤ 228MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 263.38MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 378.02 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 19 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 422.97MiB ≤ 455MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.94GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_private_action_runner | memory_usage | 10/10 | 73.21MiB ≤ 75MiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 27.07 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 329.01MiB ≤ 355MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 59.93 ≤ 200 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 304.19MiB ≤ 335MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 22.78 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 310.65MiB ≤ 345MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 28, 2026 02:24
868eeac to
269325b
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 28, 2026 04:35
da969c8 to
515bae2
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 28, 2026 04:36
269325b to
9e13c36
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 28, 2026 04:41
515bae2 to
20afbc1
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 28, 2026 04:42
9e13c36 to
f82030d
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 28, 2026 05:00
20afbc1 to
62c1e41
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 28, 2026 14:19
f82030d to
0bd0502
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 28, 2026 14:44
62c1e41 to
b83b932
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 28, 2026 14:48
0bd0502 to
1dd3932
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 28, 2026 16:01
b83b932 to
fe7b9a7
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
2 times, most recently
from
August 28, 2026 16:26
9bf5043 to
b42a7ee
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 28, 2026 20:36
cc7a3c6 to
240cc6c
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 28, 2026 20:36
7878e8c to
cc14a5c
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 28, 2026 21:08
240cc6c to
9b9a9e2
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 28, 2026 21:09
cc14a5c to
cc8a251
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 31, 2026 14:35
9b9a9e2 to
9d1ca50
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 31, 2026 14:36
cc8a251 to
d8884dc
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 31, 2026 15:16
9d1ca50 to
afb8e42
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 31, 2026 15:17
d8884dc to
5cafb80
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 31, 2026 15:37
afb8e42 to
44acba4
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 31, 2026 15:38
5cafb80 to
26875e1
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 31, 2026 15:41
44acba4 to
82c87aa
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 31, 2026 15:41
26875e1 to
bafe6a0
Compare
…logs agent This PR wires the credential Provider interface from delegatedauth into the three data-plane consumers: Forwarder: - Forwarder takes credentials from providers, buffering until they resolve - Resolver discovers credential providers for each endpoint - Transactions carry credential context - OTel sync forwarder accepts a delegated auth component - HTTP transactions serializer hardened for credential backpressure Trace writers: - Trace writers take credentials from providers, holding payloads until resolve - Sender resolves credentials before forwarding - Pipeline stats skips delegated-auth endpoints (no API key) - Trace agent impl and config wiring for credential providers Logs agent: - Logs endpoints take their credential from a provider - Wire delegated auth into the logs agent end to end - Stop logs TCP from leaking the directive on the wire - Harden Authorize, fix serverless build - Logs library HTTP destination supports credential providers
Fix two findings from the Codex/Autotest review: 1. (P2) Delay completion when requeuing credential waits For a non-retryable transaction with ErrCredentialNotReady, the CompletionHandler was called before the requeue, and again when the retried transaction finished. Now the credential-not-ready check runs before the completion handler, so completion is suppressed until the transaction actually finishes. 2. (P2) Release payloads dropped while awaiting credentials When a delegated-auth sender's queue was full, the payload was abandoned without returning it to ppool or recording an eventTypeDropped event. Now the drop bookkeeping runs and the payload is returned to the pool, reducing GC pressure and fixing telemetry underreporting.
The NewOTelSyncForwarder signature gained a delegatedauth.Component parameter in this PR, but two callers in serializerexporter were not updated: - serializer.go: fx.Provide now accepts delegatedauth.Component and passes it to NewOTelSyncForwarder - exporter_test.go: passes nil for the delegated auth component - BUILD.bazel: added dep on delegatedauth/def
The NewOTelSyncForwarder signature gained a delegatedauth.Component parameter, but the call in commonAgentFxOptions was not updated. The fx.Provide now accepts delegatedauth.Component (still the noop at this point) and passes it through. PR3 replaces the noop with the real component.
Add tests to each consumer's credential provider test file verifying that an ENC[...] key resolved by the secrets backend behaves as a normal static key — the provider path does not interfere with it. Forwarder: TestResolvedEncKeyUnaffectedByProvider Trace writer: TestAuthorizeStampsResolvedEncKeyWhenThereIsNoProvider Logs: TestAuthorizeStampsResolvedEncKey
Rebase onto foundation which added Refresh() to the Provider interface. Update stubProvider implementations in resolver, logs, and trace writer tests to satisfy the updated interface. WIF-48
…ests Replace the per-package stubProvider copies in resolver, logs, and trace writer tests with the shared StubProvider from comp/core/delegatedauth/mock. WIF-48
_test.go files are only visible within their own package. Move StubProvider to a regular .go file so consumer test packages (resolver, logs, trace writer, trace API) can import it from delegatedauth/mock. WIF-48
- Resolver: CredentialProvider alias points to credential.Provider - Logs: CredentialProvider alias, CredentialProviderLookup → credential.Lookup, isDelaDirective → credential.IsDirective - Trace config: remove redeclared CredentialProvider interface, use credential.Provider alias; CredentialProviderFn → credential.Lookup - Trace writer: apiKeyManager.Authorize → credential.StampAuth, remove headerAPIKey const - Fix gofmt and test reference to removed headerAPIKey const - Update go.mod replace directives for pkg/credential WIF-48
…oint.Authorize
- Remove the isDelaDirective one-liner wrapper; call credential.IsDirective
directly at all 4 call sites in endpoints.go and in the test
- Rewrite Endpoint.Authorize to use credential.StampAuth instead of the
inline h.Set("DD-API-KEY", ...) pattern, keeping the credentialDirective
guard as a logs-specific pre-check
WIF-48
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
August 31, 2026 15:49
82c87aa to
735c243
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 31, 2026 15:49
bafe6a0 to
3f2166e
Compare
Contributor
BenchmarksBenchmark execution time: 2026-08-31 19:31:18 Comparing candidate commit 030bf67 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.
|
Gazelle requires this because credential_provider_test.go imports pkg/credential for credential.IsDirective. WIF-48
…logs agent This PR wires the credential Provider interface from delegatedauth into the three data-plane consumers: Forwarder: - Forwarder takes credentials from providers, buffering until they resolve - Resolver discovers credential providers for each endpoint - Transactions carry credential context - OTel sync forwarder accepts a delegated auth component - HTTP transactions serializer hardened for credential backpressure Trace writers: - Trace writers take credentials from providers, holding payloads until resolve - Sender resolves credentials before forwarding - Pipeline stats skips delegated-auth endpoints (no API key) - Trace agent impl and config wiring for credential providers Logs agent: - Logs endpoints take their credential from a provider - Wire delegated auth into the logs agent end to end - Stop logs TCP from leaking the directive on the wire - Harden Authorize, fix serverless build - Logs library HTTP destination supports credential providers
…rders Process and orchestrator forwarders create their own resolvers via NewSingleDomainResolvers, which don't go through the main forwarder's provider-wired resolver. This PR threads delegatedauth.Component into both forwarders and calls ProvidersFor + SetCredentialProviders on their resolvers, matching the pattern from PR #55480. Process: comp/process/forwarders/impl/forwarders.go Orchestrator: comp/forwarder/orchestrator/impl/forwarder_orchestrator.go
Verify that normal API keys and ENC[...] keys resolved by the secrets backend flow through createParams without interference from the credential provider path.
…d StubProvider The rebase conflict resolution took the pre-refactor versions of several files, losing the credential imports and shared StubProvider usage. Restore: - resolver/domain_resolver.go: import credential, alias to credential.Provider - logs/endpoints.go: import credential, use StampAuth, remove isDelaDirective wrapper - logs/credential_provider_test.go: use delegatedauthmock.StubProvider - trace/config/config.go: import credential, alias to credential.Provider - trace/writer/sender.go: import credential, use StampAuth, remove headerAPIKey - trace/writer/credential_provider_test.go: use delegatedauthmock.StubProvider - resolver/credential_provider_test.go: use delegatedauthmock.StubProvider - Update BUILD.bazel via gazelle WIF-48
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-process-orch
branch
from
August 31, 2026 18:27
3f2166e to
030bf67
Compare
wynbennett
force-pushed
the
wyn.bennett/wif-credential-provider-consumers
branch
from
September 2, 2026 15:30
b59288f to
6939870
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wire credential providers into the process and orchestrator forwarders, which create their own resolvers separate from the main forwarder.
Both forwarders use
NewSingleDomainResolversto build their resolvers, so they bypass the main forwarder's provider-wired resolver. This PR threadsdelegatedauth.Componentinto both and callsProvidersFor+SetCredentialProviderson their resolvers, matching the pattern established in PR #55480.Wiring
Files (4)
comp/process/forwarders/impl/forwarders.go— acceptdelegatedauth.Component, wire providers into resolverscomp/process/forwarders/impl/BUILD.bazel— adddelegatedauth/defandconfig/utilsdepscomp/forwarder/orchestrator/impl/forwarder_orchestrator.go— same pattern for orchestratorcomp/forwarder/orchestrator/impl/BUILD.bazel— adddelegatedauth/defandconfig/utilsdepsArchitecture doc
See Delegated Authentication in the Agent: Architecture and Path Forward for the full design, including the previous approach vs. the new Provider interface, locking, and future work.
Stacked PRs
WIF-48