[9.4](backport #7571) fix: gracefully handle missing secrets in policy monitor

[mergify]

What is the problem this PR solves?

Fleet Server crash-loops when any secret referenced in an agent policy's secret_references cannot be found in the Fleet secrets store. A single missing secret — from a deleted or orphaned integration credential — causes the entire policy monitor to abort with:

Error: failed to get secret values: unexpected status 404 from fleet secret read: ...
Fleet Server failed

Fleet Server then restarts, re-reads the same stale policy document from the policy monitor (the checkpoint hasn't advanced), encounters the same 404, and crashes again — indefinitely. Every agent on that fleet-server instance goes offline for the duration.

This is a disproportionate blast radius: one deleted integration credential (unrelated to fleet-server's own connectivity) can take the entire Fleet Server offline.

The root causes that produce missing secrets are tracked in Kibana (elastic/kibana#282911 and elastic/kibana#282912) and in fleet-server (#7533, #7534). This PR is a defense-in-depth fix: even after those root causes are resolved, fleet-server should not treat a single missing package-policy credential as a reason to crash-loop and take all agents offline.

How does this PR solve the problem?

Two changes:

1. bulk/secret.go — ExtendedAPI.Read: Returns a typed ErrSecretNotFound sentinel when the Fleet secrets API responds with 404. This lets callers distinguish "this secret document doesn't exist" from network/auth failures (403, 500, etc.), which should still be fatal.

2. bulk/engine.go — ReadSecrets: When ReadSecret returns ErrSecretNotFound, logs a warning with the secret ID and skips it (continues the loop) rather than aborting. The returned map contains only the secrets that were successfully fetched.

With this change, a missing secret causes the policy to load without that secret value — the raw $co.elastic.secret{id} placeholder remains unsubstituted in the affected integration's config. That integration's credentials won't resolve (it will fail to authenticate), but fleet-server itself stays up and continues serving all other agents.

Callers that need hard failure for missing secrets (e.g. output API key resolution, where an empty key would silently break agent→ES connectivity) can check whether the expected ID is present in the returned map — as the second commit in #7416 already does.

How to test this PR locally

1. Start a local stack with fleet-server
2. Create an integration that uses secrets (e.g. any OAuth-based integration)
3. Directly delete the secret document from Elasticsearch: DELETE /_fleet/secret/<id>
4. Restart fleet-server and observe that it:
   • Logs WARN secret not found; policy will load without it secret_id=<id>
   • Continues to start successfully
   • Serves other agents normally
5. Verify (via agent diagnostics) that the affected integration's credential field contains the raw $co.elastic.secret{id} placeholder rather than the secret value

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer. (Each instance evaluates ReadSecrets independently per policy load; skipping is per-request with no shared state.)
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected. (The change reduces work on failure — skipping a secret is cheaper than returning an error — so scale impact is neutral or positive.)
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc. (No new load introduced; 404s are now fast-path skips.)

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes Add graceful handling for missing secrets in policy monitor #7536
• Root cause (how secrets become dangling in the first place):
  • [Fleet] Orphaned .fleet-policies documents survive agent policy deletion and can crash-loop fleet-server kibana#282911 — agentPolicyService.delete() can leave orphaned .fleet-policies documents when the cleanup call after package-policy deletion fails silently
  • [Fleet] Secret cleanup only checks package policy saved objects, can delete a secret still referenced by a compiled policy kibana#282912 — deleteSecretsIfNotReferenced only checks live package-policy saved objects, not .fleet-policies, so secrets are deleted even when an orphaned policy document still references them
• Mitigation for the agent-update-failure source of dangling secrets: Retain output secret when agent update fails #7533, Reconcile orphaned output secrets after agent update failures #7534
• Context: the >= 400 check in ExtendedAPI.Read was introduced in Store output API key secrets in .fleet-secrets instead of .fleet-agents #7416. That PR's own second commit described the original tolerant design — ReadSecrets was not expected to fail on a missing secret; hard failure for the output API key was added at the caller level. This PR restores that layered design by making 404 non-fatal in ReadSecrets while preserving fatal behavior for other 4xx/5xx responses.

  ---

  This is an automatic backport of pull request fix: gracefully handle missing secrets in policy monitor #7571 done by Mergify.

[mergify]

Cherry-pick of 5ece0ba has failed:

On branch mergify/bp/9.4/pr-7571
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 5ece0ba.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1786053515-graceful-missing-secrets.yaml
	modified:   internal/pkg/bulk/engine.go
	new file:   internal/pkg/bulk/secret_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/bulk/secret.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

TL;DR

All four Buildkite unit-test jobs are failing due to a deterministic unit-test regression: ExtendedAPI.Read no longer returns an error for HTTP 500, so TestExtendedAPIRead_ServerError_ReturnsGenericError fails. Restore non-404 error handling (or update the new test to match intended 9.4 behavior) and rerun unit jobs.

Remediation

• Reconcile internal/pkg/bulk/secret.go and internal/pkg/bulk/secret_test.go semantics: either reintroduce res.StatusCode >= 400 error handling in ExtendedAPI.Read (recommended), or adjust/remove the server-error test if silent 500 handling is truly intended for 9.4.
• Re-run the same failing jobs after the fix (Run unit tests, Run fips140=on unit tests..., Run fips140=only unit tests..., Run unit tests: MacOS 13).

Investigation details

Root Cause

Commit 3c7adc08e36f40ee91dc631f85913bdb1bb4ce49 removed the >= 400 branch in ExtendedAPI.Read, leaving only 404 special-casing and JSON decode:

• internal/pkg/bulk/secret.go#L45-L54 (at the failing SHA)

The same PR adds a test that expects HTTP 500 to return an error:

• internal/pkg/bulk/secret_test.go#L50-L54

With the current code, a 500 response body like {"error":"internal"} decodes into SecretResponse without decode error, so err == nil, violating require.Error(t, err).

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16151
• Jobs/steps: all four failed unit-test steps (Linux, FIPS-on, FIPS-only, MacOS 13).
• Buildkite log excerpt (all failing jobs):
  • Error: running "go test -tags=grpcnotrace ... ./..." failed with exit code 1
• Local reproduction on the exact failing SHA (3c7adc08e36f40ee91dc631f85913bdb1bb4ce49):
  • go test -tags=grpcnotrace -race ./internal/pkg/bulk -v
  • --- FAIL: TestExtendedAPIRead_ServerError_ReturnsGenericError
  • secret_test.go:53: Error: An error is expected but got nil.

No matching flaky-test issue was found for TestExtendedAPIRead_ServerError_ReturnsGenericError.

Verification

• Ran targeted test reproduction against the failing SHA in this workflow environment (./internal/pkg/bulk with race).

Follow-up

If you want strict backport minimalism, drop/adjust only the new server-error assertion; if you want parity with mainline safety behavior, restore >= 400 error handling and keep the test as-is.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏