fix: gracefully handle missing secrets in policy monitor

[ycombinator]

What is the problem this PR solves?

Fleet Server crash-loops when any secret referenced in an agent policy's secret_references cannot be found in the Fleet secrets store. A single missing secret — from a deleted or orphaned integration credential — causes the entire policy monitor to abort with:

Error: failed to get secret values: unexpected status 404 from fleet secret read: ...
Fleet Server failed

Fleet Server then restarts, re-reads the same stale policy document from the policy monitor (the checkpoint hasn't advanced), encounters the same 404, and crashes again — indefinitely. Every agent on that fleet-server instance goes offline for the duration.

This is a disproportionate blast radius: one deleted integration credential (unrelated to fleet-server's own connectivity) can take the entire Fleet Server offline.

The root causes that produce missing secrets are tracked in Kibana (elastic/kibana#282911 and elastic/kibana#282912) and in fleet-server (#7533, #7534). This PR is a defense-in-depth fix: even after those root causes are resolved, fleet-server should not treat a single missing package-policy credential as a reason to crash-loop and take all agents offline.

How does this PR solve the problem?

Two changes:

1. bulk/secret.go — ExtendedAPI.Read: Returns a typed ErrSecretNotFound sentinel when the Fleet secrets API responds with 404. This lets callers distinguish "this secret document doesn't exist" from network/auth failures (403, 500, etc.), which should still be fatal.

2. bulk/engine.go — ReadSecrets: When ReadSecret returns ErrSecretNotFound, logs a warning with the secret ID and skips it (continues the loop) rather than aborting. The returned map contains only the secrets that were successfully fetched.

With this change, a missing secret causes the policy to load without that secret value — the raw $co.elastic.secret{id} placeholder remains unsubstituted in the affected integration's config. That integration's credentials won't resolve (it will fail to authenticate), but fleet-server itself stays up and continues serving all other agents.

Callers that need hard failure for missing secrets (e.g. output API key resolution, where an empty key would silently break agent→ES connectivity) can check whether the expected ID is present in the returned map — as the second commit in #7416 already does.

How to test this PR locally

1. Start a local stack with fleet-server
2. Create an integration that uses secrets (e.g. any OAuth-based integration)
3. Directly delete the secret document from Elasticsearch: DELETE /_fleet/secret/<id>
4. Restart fleet-server and observe that it:
   • Logs WARN secret not found; policy will load without it secret_id=<id>
   • Continues to start successfully
   • Serves other agents normally
5. Verify (via agent diagnostics) that the affected integration's credential field contains the raw $co.elastic.secret{id} placeholder rather than the secret value

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer. (Each instance evaluates ReadSecrets independently per policy load; skipping is per-request with no shared state.)
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected. (The change reduces work on failure — skipping a secret is cheaper than returning an error — so scale impact is neutral or positive.)
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc. (No new load introduced; 404s are now fast-path skips.)

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes Add graceful handling for missing secrets in policy monitor #7536
• Root cause (how secrets become dangling in the first place):
  • [Fleet] Orphaned .fleet-policies documents survive agent policy deletion and can crash-loop fleet-server kibana#282911 — agentPolicyService.delete() can leave orphaned .fleet-policies documents when the cleanup call after package-policy deletion fails silently
  • [Fleet] Secret cleanup only checks package policy saved objects, can delete a secret still referenced by a compiled policy kibana#282912 — deleteSecretsIfNotReferenced only checks live package-policy saved objects, not .fleet-policies, so secrets are deleted even when an orphaned policy document still references them
• Mitigation for the agent-update-failure source of dangling secrets: Retain output secret when agent update fails #7533, Reconcile orphaned output secrets after agent update failures #7534
• Context: the >= 400 check in ExtendedAPI.Read was introduced in Store output API key secrets in .fleet-secrets instead of .fleet-agents #7416. That PR's own second commit described the original tolerant design — ReadSecrets was not expected to fail on a missing secret; hard failure for the output API key was added at the caller level. This PR restores that layered design by making 404 non-fatal in ReadSecrets while preserving fatal behavior for other 4xx/5xx responses.

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[Copilot]

Pull request overview

This PR adds a defense-in-depth safeguard to prevent Fleet Server from crash-looping when an agent policy references a secret that no longer exists in the Fleet secrets store. It introduces a typed “not found” error from the secrets read path and updates policy secret resolution to treat missing secrets as non-fatal, keeping Fleet Server available while allowing only the affected integration to degrade.

Changes:

• Introduce an exported ErrSecretNotFound sentinel returned on GET /_fleet/secret/{id} 404s.
• Update Bulker.ReadSecrets to warn and continue when a secret is missing instead of aborting policy loading.
• Add unit tests validating 404 behavior and successful secret decoding, plus a changelog fragment.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
internal/pkg/bulk/secret.go	Adds ErrSecretNotFound and returns it on 404 responses from the Fleet secrets API.
internal/pkg/bulk/secret_test.go	Adds tests for 404 sentinel behavior, generic server errors, and successful reads.
internal/pkg/bulk/engine.go	Skips missing secrets (warn + continue) in ReadSecrets while preserving fatal behavior for other errors.
changelog/fragments/1786053515-graceful-missing-secrets.yaml	Documents the behavior change as a bug fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MichelLosier]

LGTM!

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

internal/pkg/bulk/engine.go:341

• ReadSecrets now tolerates ErrSecretNotFound by skipping the missing secret. There are unit tests for ExtendedAPI.Read returning the sentinel, but there isn't a test that asserts ReadSecrets actually continues and returns a partial result map (and does not return an error) when one ID is missing. Adding a focused unit test would help prevent regressions in this crash-loop fix.

// read secrets one by one as there is no bulk API yet to read them in one request
func (b *Bulker) ReadSecrets(ctx context.Context, secretIds []string) (map[string]string, error) {
	result := make(map[string]string)
	esClient := b.Client()
	for _, id := range secretIds {
		val, err := ReadSecret(ctx, esClient, id)
		if err != nil {
			if errors.Is(err, ErrSecretNotFound) {
				zerolog.Ctx(ctx).Warn().Str("secret_id", id).Msg("secret not found; policy will load without it")
				continue
			}
			return nil, err
		}
		result[id] = val
	}
	return result, nil

internal/pkg/bulk/engine.go:336

• The warning message here is policy-specific ("policy will load without it"), but Bulker.ReadSecrets is also used for output API key resolution (internal/pkg/policy/policy_output.go:427). In that path, a missing secret is fatal, so this log line can be misleading during incidents. Consider making the message generic (or moving logging to the policy-monitor call site) so it remains accurate for all callers.

			if errors.Is(err, ErrSecretNotFound) {
				zerolog.Ctx(ctx).Warn().Str("secret_id", id).Msg("secret not found; policy will load without it")
				continue

[github-actions]

TL;DR

The failing :gcloud: Cloud e2e FIPS Test is most likely a configuration/infrastructure failure in cloud provisioning, not a regression in this PR. The provided log shows fleet_url is empty and the test script exits with status 1 in that condition.

Remediation

• Ensure mage test:cloudE2EUp produces a non-empty Terraform output for fleet_url before mage test:cloudE2ERun.
• Re-run the job with full pre-cleanup logs retained, since the provided excerpt only includes teardown and final exit status.

Investigation details

Root Cause

.buildkite/scripts/cloud_e2e_test.sh:35-45 exits with code 1 when FLEET_SERVER_URL is empty.

The failing job log excerpt includes:

• /tmp/gh-aw/buildkite-logs/fleet-server-gcloud-cloud-e2e-fips-test.txt:107 → - fleet_url = "" -> null
• /tmp/gh-aw/buildkite-logs/fleet-server-gcloud-cloud-e2e-fips-test.txt:136 → Error: The command exited with status 1

PR #7571 changes are confined to internal/pkg/bulk/secret.go, internal/pkg/bulk/engine.go, and tests/changelog, which do not directly control cloud Terraform output generation for fleet_url.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16128
• Job/step: :gcloud: Cloud e2e FIPS Test
• Key log excerpt:

  - fleet_url              = "" -> null
  ...
  Error: The command exited with status 1
  

Verification

• Not run locally (cloud Buildkite provisioning path is external to this sandbox).

Follow-up

• I did not find an accessible open flaky-test issue in this repo that explicitly matches this fleet_url-empty cloud E2E failure mode.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[github-actions]

@Mergifyio backport 9.5 9.4 8.19

[mergify]

backport 9.5 9.4 8.19

✅ Backports have been created

Details

• #7574 [9.5](backport #7571) fix: gracefully handle missing secrets in policy monitor has been created for branch 9.5 but encountered conflicts

Cherry-pick of 5ece0ba has failed:

On branch mergify/bp/9.5/pr-7571
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit 5ece0ba.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1786053515-graceful-missing-secrets.yaml
	modified:   internal/pkg/bulk/engine.go
	new file:   internal/pkg/bulk/secret_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/bulk/secret.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7575 [9.4](backport #7571) fix: gracefully handle missing secrets in policy monitor has been created for branch 9.4 but encountered conflicts

Cherry-pick of 5ece0ba has failed:

On branch mergify/bp/9.4/pr-7571
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 5ece0ba.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1786053515-graceful-missing-secrets.yaml
	modified:   internal/pkg/bulk/engine.go
	new file:   internal/pkg/bulk/secret_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/bulk/secret.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7576 [8.19](backport #7571) fix: gracefully handle missing secrets in policy monitor has been created for branch 8.19 but encountered conflicts

Cherry-pick of 5ece0ba has failed:

On branch mergify/bp/8.19/pr-7571
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 5ece0ba.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1786053515-graceful-missing-secrets.yaml
	modified:   internal/pkg/bulk/engine.go
	new file:   internal/pkg/bulk/secret_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/bulk/secret.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally