Signed ContainerProfile fragment bundles - #63
Open
ConstanzeTU wants to merge 19 commits into
Open
Conversation
…tainerProfile Bring the well-tested pkg/signature core (cosign Signer/Verifier, signature.kubescape.io annotations, VerifyObjectAllowUntrusted) and cmd/sign-object onto the migrated ContainerProfile world. AP/NN deprecation collapses the former AP+NN adapter pair and the two tamper verify methods into a single ContainerProfile adapter and one verifyUserContainerProfile: one fetch, one verify, one R1016 site. Wire the tamper-alert exporter in main.go and gate the user-defined CP overlay on load. Adds enableSignatureVerification config (default off). Flat single-signed-CP behavior — the foundation the fragment-bundle layer builds on.
Add pkg/signature/bundle: multiple independently-signed partial ContainerProfiles (fragments) authored by different parties are each verified against a per-class trust policy (allowed signer identities + allowed spec paths, so a client-class signer cannot inject execs into the server profile), deterministically assembled into one composite (append+dedup, ingress/egress by identifier, class-precedence ordering — order-independent), and bound to the exact admissible leaf set via a Merkle leaf-tree manifest. SignComposite re-signs the assembled result with the cluster key so the R1016 tamper path protects it. Signer identity is the signing public-key fingerprint (key-based) or OIDC subject (keyless). Adds HashSignableContent to expose the canonical leaf digest. Unit tests cover the happy path, order-independence, tamper, untrusted signer, class confinement, and unsigned.
…e cache Resolve a user-defined-profile label that names a bundle by listing its fragments (new ListContainerProfiles client method), verifying + assembling + re-signing them into the authoritative composite, which then flows through the existing verify/tamper gate. A present-but-broken bundle suppresses the single-CP fallback; a tampered fragment raises R1016. Enabled by bundleTrustPolicyPath + bundleSigningKeyPath config (mounted ConfigMap + Secret), loaded in main.go via bundle.LoadTrustPolicy/LoadSigningKey. Adds bundle.SignerID for authoring trust policies. Runtime path unit-tested against the storage mock (happy path, not-a-bundle fallback, tampered→R1016).
Test_29_SignedContainerProfile: a signed user-authored CP loads + enforces, and an unlisted exec fires R0001. Test_31_TamperDetectionAlert: modifying a signed CP's spec in storage without re-signing fires R1016 on reload. Both use the sign-after-roundtrip pattern (sign the storage-normalised form so the signed hash matches what node-agent recomputes on load). Adds the curl-signed-cp deployment fixture.
The fork's mirrormain storage removed ApplicationProfile/NetworkNeighborhood from v1beta1 (AP/NN deprecation), so the carried-over AP/NN adapters + sign-object AP/NN branches no longer compile against it (the image build resolves storage via the go.mod replace to the fork commit, not the published v0.0.290 that still has them). Delete the AP/NN adapters, port the generic signer tests + cmd/sign-object to ContainerProfile (seccomp + rules adapters kept), and fix the CLI help text. Builds + tests pass against the fork storage.
verifyUserContainerProfile ran only on initial load (tryPopulateEntry), so a signed CP tampered AFTER it was cached reloaded through the reconciler without re-verification — no R1016 (Test_31 failure). Add the verify gate to refreshOneEntry, right after the RV fast-skip, so it runs exactly once per CP change and R1016 fires on post-load tampering (deduped per resourceVersion).
Test_38_SignedBundleOverlay: three fragments signed by two different keys (vendor: base; operator: admission ingress + overlay execs) round-trip storage signed, assemble into one enforced composite (union proof: an exec allowed only by the overlay fragment stays quiet while an unlisted exec fires R0001), a fragment tampered in storage without re-signing fires R1016 via reconciler re-assembly, and re-signing it recovers the composite (fresh unlisted exec alerts again, overlay-allowed exec stays quiet). Reconciler now re-assembles bundles on refresh instead of degrading the cached composite to a same-named fragment via the single-CP GET; the composite carries the bundle Merkle root as its ResourceVersion so the RV fast-skip holds while fragments are unchanged. Test chart gains the bundle trust policy (per-class signer fingerprints + allowed spec paths) and the cluster signing key, mounted at /etc/bundle (throwaway CI test keys).
The trust policy ConfigMap (per-class signer fingerprints + allowed spec paths) and cluster signing-key Secret that Test_38 depends on; mounted at /etc/bundle. Force-added: the root .gitignore's unanchored 'node-agent' binary pattern also matches this template directory.
…ripped The storage server's List serves items from its metadata table without loading the payload, so listed fragments carry empty specs. Assembly hashed those and flagged every signed fragment as tampered (spurious R1016, bundle never loaded — first Test_38 run). Use the List only to discover the fragment set and Get each fragment for its full spec; a transient Get failure is operational (retry next tick, no R1016). Adds a spec-stripped-List regression test.
The curl image is alpine/busybox: ls lives at /bin/ls, so the absolute /usr/bin/ls exec failed and produced no exec event — starving the R0001 gate (the bundle itself assembled cleanly: 3 fragments, stable root, no R1016). Exec plain names; id still resolves to /usr/bin/id, the path the overlay fragment allows. Log the first probe's exec result for future diagnosis.
37 was the next free number; 38 left a confusing gap. Function, fixture, labels, and doc references renamed — no behaviour change.
… ignores label selectors The storage server returns every CP in the namespace regardless of the List label selector, so assembleUserBundle assembled ALL class-labeled fragments for ANY user-defined-profile name (observed live: a client workload's profile lookup assembled the server's fragments, shadowing its own flat profile). Filter listed items on the bundle label before fetching/verifying. Regression test uses a selector-ignoring mock List.
…erver normalisation Vendors must ship SIGNED fragments, but the storage server normalises specs on save (deflate, with cluster-configurable collapse settings) — so a signature over the shipped form breaks on ingestion, and offline pre-normalisation is unsound because the collapse configuration is per-cluster. Instead the signer embeds the exact canonical signed content in the signature.kubescape.io/content annotation (base64+gzip; annotations are never mutated by storage): verification binds the embedded bytes, and the bundle layer treats them as the verified source of truth — name, labels (class AND bundle membership, so stored-label flips cannot escalate or cross bundles), spec, and leaf digest all come from the signed content. The stored object is a carrier whose spec drift is irrelevant to the chain. Opt-in via WithEmbedContent / sign-object --embed-content (CLI default: on). Legacy sign-after-roundtrip artifacts keep working unchanged.
The assembled-bundle line (fragments + Merkle root) was debug-only, forcing operators to enable global debug logging — which floods the log — just to observe bundle lifecycle. Log root TRANSITIONS (first assembly, fragment-set changes) at info; unchanged per-tick re-assemblies stay at debug.
…m adversarial review Two independent adversarial reviews of the signing/bundle layer surfaced real issues; fixes with regression tests: - V1/C3 (CRITICAL) trust-policy bypass: signerIdentity trusted the unsigned identity/issuer annotations for keyless, so anyone could spoof a trusted signer by stamping two strings (verification is allow-untrusted → no Fulcio attestation). Now the identity is ALWAYS the fingerprint of the public key the signature verified against; the OIDC branch is removed. - C2/V5 embedded-content decoupling: verification hashed the embedded bytes but never bound them to the carrier, and the flat path then enforced the mutable LIVE spec. Now verify.go binds embedded content to the object's name+namespace (mismatch = tamper), and verifyUserContainerProfile enforces the embedded (verified) spec, not the live one. - H1 malformed embedded content on a signed object is now classified as tamper (ErrSignatureMismatch → R1016 + fail-closed), not swallowed as operational. - H2 DecodeSignatureFromAnnotations rejects non-base64 instead of falling back to raw bytes (which could feed a bare attacker public key). - M1/V8 DoS: bound decompressed embedded content (io.LimitReader, 8MiB) and cap fragments per bundle (64) — assembly runs every reconcile tick. - V3 seccomp confinement: any non-zero seccomp content confines to the class, not just DefaultAction. - V7 bundle R1016 dedup keys on the fragment-set fingerprint (name@RV) so distinct tamper states re-alert instead of being masked. - V2 defense-in-depth: AssembleAndVerify self-checks the Merkle root; enforcement remains continuous per-tick re-verification (the root is a provenance record + external-verifier commitment).
…overy The previous fingerprint-keyed dedup stored a key that the clean-recovery path never cleared (it deleted a stale empty key), so a later tamper whose fragment-set fingerprint recurred — which happens because sign-fragment.sh delete/recreates fragments, resetting resourceVersions — was silently deduped and NO R1016 fired (caught in live retest: tamper detected + fail-closed, but zero alerts). Hold the last-alerted fingerprint per bundle and CLEAR it on clean re-assembly, so distinct tampers alert and a recurring one re-alerts after recovery. Regression test covers the tamper→recover→re-tamper cycle.
…rence) The fingerprint-keyed dedup re-alerted only if the recurring fragment-set fingerprint was cleared by an observed clean tick — fragile when delete/recreate resets resourceVersions to the same values. Switch to an edge-trigger: alert once per OBSERVED clean->tamper transition, re-arm on the next observed clean assembly. Keys on tamper STATE, not fragment identity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multi-party signed profile fragments assembled into one enforced ContainerProfile.
pkg/signatureported to the ContainerProfile world (single CP adapter; AP/NN adapters removed — the fork storage dropped those types), R1016 tamper detection on user-authored CPs (verify on load + on reconciler refresh).pkg/signature/bundle: per-class trust policy (allowed signer fingerprints + allowed spec paths), deterministic order-independent assembly, Merkle leaf-tree manifest, internal re-sign of the composite with the cluster key.signature.kubescape.io/bundlelabel (membership re-checked client-side — the storage List ignores label selectors and returns spec-stripped items, so fragments are fetched individually), fail-closed on any inadmissible/tampered fragment, composite carries its Merkle root as ResourceVersion for the refresh fast-skip.bundleTrustPolicyPath+bundleSigningKeyPath(+enableSignatureVerificationfor strict mode).cmd/sign-objectported to ContainerProfile; binaries released assign-object-v0.1.0.Validated: component tests green (31/31, run 31244214794 + rename-confirm run) and a full live demo dry-run against a k3s cluster (bob PR kubescape#197).