Skip to content

fix(charts): explicit lenient probe tolerances (sse-proxy, otel wrappers) + Quantity schema typing - #32

Merged
braghettos merged 1 commit into
mainfrom
harden-probes
Aug 7, 2026
Merged

fix(charts): explicit lenient probe tolerances (sse-proxy, otel wrappers) + Quantity schema typing#32
braghettos merged 1 commit into
mainfrom
harden-probes

Conversation

@braghettos

Copy link
Copy Markdown
Collaborator

Applies the org probe-tolerance standard to this repo's charts and runs the Quantity schema-typing scan, in one PR.

Problem 1: default probe tolerances kill healthy pods under CPU contention

The Kubernetes probe defaults (timeoutSeconds: 1, failureThreshold: 3) mean a slow-but-alive health reply counts as a probe failure — under CPU contention healthy pods get liveness-killed, and the resulting restart storm worsens the contention. The org standard (precedent: the chart-inspector block in the core-provider chart, authn PR #13 and the sibling sweep PRs merged with it):

  • livenessProbe: timeoutSeconds: 5, failureThreshold: 6 — lenient, so only a truly dead process is restarted.
  • readinessProbe: timeoutSeconds: 5, failureThreshold: 3 — endpoint removal stays prompt, but load alone must not flap it.
  • Existing explicit periodSeconds kept; resources untouched; no probes added where none exist.

Per-chart changes

Chart Where Liveness (before → after) Readiness (before → after)
krateo-sse-proxy templates/deployment.yaml (probes hardcoded there) t=1, fail=3, per=10 (defaults) → t=5, fail=6, per=10 = 60s window t=1, fail=3, per=5 (defaults) → t=5, fail=3, per=5
otel-collector-deployment values.yaml passthrough to upstream opentelemetry-collector 0.158.1 (templates/_pod.tpl renders these keys) defaults → t=5, fail=6, implicit per=10 = 60s window defaults → t=5, fail=3, implicit per=10
otel-collector-daemonset same passthrough defaults → t=5, fail=6, implicit per=10 = 60s window defaults → t=5, fail=3, implicit per=10

Both wrappers' values.schema.json are extended to type the new livenessProbe/readinessProbe keys (matching the upstream chart's own schema key set). Rendering was verified end-to-end through the upstream subchart: the tolerances land in the rendered Deployment/DaemonSet pod specs.

krateo-observability ships no workload templates of its own (only Services + a Secret; pods come from the upstream clickstack subchart), so per the standard nothing was added there.

Problem 2: Quantity fields typed as integer/number in values.schema.json

Bare integer/number typing on Kubernetes Quantity fields (cpu/memory/storage) propagates into crdgen-generated composition CRDs and breaks the krateo-composition-version MutatingAdmissionPolicy typed conversion on Quantity strings (krateo-core-provider#66; snowplow had it, fixed in snowplow PR #140).

Scan result: CLEAN. All four charts' schemas already type every cpu/memory/persistence-size field as "string". The only integer size-like fields (max_log_size, send_batch_size) are OTel collector config counts, not k8s Quantities. No changes needed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LJsLqtryCgWwEt8FnPE1se

…ers) + Quantity schema typing

Two org-standard sweeps in one pass over this repo's charts:

1) Probe tolerances. The Kubernetes probe defaults (timeoutSeconds: 1,
   failureThreshold: 3) kill HEALTHY pods under CPU contention: a
   slow-but-alive health reply counts as a failure and the resulting
   restart storm worsens the contention. Per the org standard (precedent:
   the chart-inspector block in the core-provider chart, authn PR #13 and
   the sibling sweep PRs):
   - livenessProbe: timeoutSeconds: 5, failureThreshold: 6 — lenient, so
     only a truly dead process is restarted (sse-proxy: 10s x 6 = 60s
     window; otel wrappers: implicit 10s default x 6 = 60s window).
   - readinessProbe: timeoutSeconds: 5, failureThreshold: 3 — endpoint
     removal stays prompt, but load alone must not flap it.
   krateo-sse-proxy gets this directly in templates/deployment.yaml
   (probes are hardcoded there, existing periodSeconds kept). The two
   otel-collector wrappers get it via values passthrough to the upstream
   opentelemetry-collector chart 0.158.1, whose templates/_pod.tpl
   renders exactly these keys; the wrappers' values.schema.json is
   extended to type the new livenessProbe/readinessProbe keys.
   Resources untouched; no probes added to containers that have none
   (krateo-observability ships no workload templates of its own).

2) Quantity schema typing scan. values.schema.json fields for Kubernetes
   Quantities (cpu/memory/storage) typed as bare integer/number propagate
   into crdgen-generated composition CRDs and break the
   krateo-composition-version MutatingAdmissionPolicy typed conversion on
   Quantity strings (krateo-core-provider#66; snowplow had it, fixed in
   snowplow PR #140). Scanned all four charts' schemas: CLEAN — every
   cpu/memory/persistence-size field is already typed "string" (the only
   integer size-like fields, max_log_size/send_batch_size, are OTel
   collector config counts, not k8s Quantities). No changes needed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LJsLqtryCgWwEt8FnPE1se
@braghettos
braghettos merged commit d7a992c into main Aug 7, 2026
7 of 10 checks passed
@braghettos
braghettos deleted the harden-probes branch August 7, 2026 08:42
braghettos added a commit that referenced this pull request Aug 7, 2026
krateo-sse-proxy 0.1.6 -> 0.1.7, otel-collector-deployment 0.3.3 -> 0.3.4,
otel-collector-daemonset 0.1.5 -> 0.1.6. PR #32 shipped the template/schema
changes without bumping the literal chart versions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LJsLqtryCgWwEt8FnPE1se
braghettos added a commit that referenced this pull request Aug 7, 2026
Convert the bundle to the invariant OKF file set (ChartRepo index): thin
README skeleton, docs/{index,overview,usage,configuration,api,examples,
release,log}.md + rewritten docs/llms.txt, and a runnable
examples/observability-composition. Every claim re-derived from the four
charts (rendered via helm template with vendored deps) and the installer
component pins; docs/wiring.md and docs/crds.md folded into
configuration.md / api.md.

Divergences fixed while re-verifying:
- compositiondefinition.yaml pointed at the nonexistent OCI artifact
  charts/observability@0.1.8 -> charts/krateo-observability@0.1.11 (the
  release workflow publishes by chart name; verified against GHCR).
- README/old docs still described the pre-migration layout (kagent/ agent
  chart, CHART_VERSION placeholder versioning, stale image tags 1.0.0,
  fullnameOverride krateo-observability, the inert http-handlers ConfigMap
  as if live) — all rewritten to current source.
- The stale template-comment claim that upstream emits a
  krateo-clickstack-app Service corrected in docs: the rendered upstream
  Service is krateo-clickstack (3000/app, 4320/opamp).

CI: lint.yaml now also calls the shared lint-docs reusable
(krateo-platformops/.github). Chart values/templates untouched (PR #32
owns the probe changes).


Claude-Session: https://claude.ai/code/session_01LJsLqtryCgWwEt8FnPE1se

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant