feat(container-images): collect hidden bytes from containerd snapshots - #56823
Stephanie0829 wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
AI review by Codex (OpenAI) - workflow run
patch is correct within the requested SKILL.md review scope. One skill changed: the update removes --no-edit and explains the compatibility reason. Overall score: 83/100 (Approve). The suggestions below concern the whole-file rubric; they are not regressions introduced by this patch.
| `reno new` does not open an editor unless `--edit` is passed. Do not use | ||
| `--no-edit`; released versions may reject that flag. |
There was a problem hiding this comment.
[P3] Skill assessment — 83/100, Approve
Scores: Description 17/25; Writing philosophy 22/25; Structure and progressive disclosure 24/25; Output definition and examples 20/25.
The 137-line skill has a clear purpose, imperative steps, short commands, a YAML output template, and explicit lint-based success criteria. The changed instructions explain why the unsupported flag should be omitted. It does not operate on production. Ownership exists in CODEOWNERS (@DataDog/agent-devx), but is not declared in the skill. create-pr mentions release notes without duplicating this workflow.
Top 3 improvements:
- Expand the description (line 3) with explicit triggering language and user phrases to improve discovery.
- Add a complete input/output example alongside the usage section: a request to fix an NTP timeout → generated filename, concrete
fixesYAML, and validation result. Existing examples show fragments rather than the full outcome. - Declare
@DataDog/agent-devxas owner and list prerequisites together, includingrenoand theddaenvironment required for validation.
Suggested description: “Create and validate a reno release note for Datadog Agent or Cluster Agent changes. Use when users ask to ‘add a release note’, ‘write a changelog entry’, or ‘create a reno note’, or when a PR needs a customer-facing release note. Skip changes eligible for changelog/no-changelog.”
Overall recommendation: Approve. Priority: 3.
BenchmarksBenchmark execution time: 2026-09-24 17:49:57 Comparing candidate commit b2caeb8 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.
|
Files inventory check summaryFile checks results against ancestor 5061317e: Results for datadog-agent_7.85.0~devel.git.543.b2caeb8.pipeline.139917600-1_amd64.deb:No change detected Results for datadog-iot-agent_7.85.0~devel.git.543.b2caeb8.pipeline.139917600-1_amd64.deb:No change detected |
Static quality checks✅ Please find below the results from static quality gates Successful checksInfo
2 successful checks with minimal change (< 2 KiB)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 5061317 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_cpu | % cpu utilization | +1.15 | [+0.90, +1.39] | 1 | Logs |
| ➖ | quality_gate_security_no_fs_load | memory utilization | +0.33 | [+0.26, +0.41] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | +0.25 | [+0.21, +0.29] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_metrics_logs | memory utilization | +0.15 | [-0.07, +0.38] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_private_action_runner | memory utilization | +0.12 | [-0.02, +0.25] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | +0.03 | [-0.00, +0.06] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | -0.32 | [-0.35, -0.28] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.43 | [-0.46, -0.39] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | -0.48 | [-1.35, +0.39] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_memory | memory utilization | -0.89 | [-1.10, -0.68] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 4 ≤ 5 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 176.42MiB ≤ 179MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 748.91KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 4 ≤ 5 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 525.77MiB ≤ 537MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.14MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 20 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 214.16MiB ≤ 228MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 263.54MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 384.78 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 19 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 425.67MiB ≤ 455MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.94GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_private_action_runner | memory_usage | 10/10 | 72.93MiB ≤ 75MiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 30.82 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 324.88MiB ≤ 355MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 65.39 ≤ 200 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 303.55MiB ≤ 335MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 24.39 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 326.06MiB ≤ 345MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3980f26f3e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if err := checkOverlayMetadataAccess(); err != nil { | ||
| return nil, err | ||
| } | ||
| return calculateHiddenBytes(ctx, layers, limits, readOverlayMetadata) |
There was a problem hiding this comment.
Skip privileged checks for user-xattr layers
When containerd returns an overlay mount with userxattr, BuildImageLayers explicitly marks the layers to read user.overlay.*, but this unconditional check still rejects any process outside the initial user namespace or without CAP_SYS_ADMIN. Those privileges are required for trusted.overlay.*, not user xattrs, so rootless/user-xattr overlayfs images that otherwise pass mount validation can never produce hidden-byte results. Apply the privilege check only when at least one layer uses trusted xattrs.
Useful? React with 👍 / 👎.
| Size: layer.SizeBytes, | ||
| Digest: layer.DiffID, | ||
| Size: layer.SizeBytes, | ||
| HiddenBytes: layer.HiddenBytes, |
There was a problem hiding this comment.
Add a fakeintake E2E assertion for hidden bytes
This changes the customer-visible container-image payload, but a repo-wide check found only unit tests and the manually driven test/integration/container_image_hidden_bytes setup—there is no test/new-e2e assertion for hidden_bytes. Consequently, a break anywhere in the real containerd → workloadmeta → container-image check → fakeintake path can pass CI; add and wire an E2E test that deploys the Agent and verifies nil, zero, and nonzero values in fakeintake.
AGENTS.md reference: AGENTS.md:L98-L106
Useful? React with 👍 / 👎.
| `reno new` does not open an editor unless `--edit` is passed. Do not use | ||
| `--no-edit`; released versions may reject that flag. |
There was a problem hiding this comment.
Add trigger phrases and ownership to the release-note skill
Skill score: Description 15/25, Writing philosophy 20/25, Structure/progressive disclosure 24/25, Output/examples 19/25; total 78/100 — Approve with suggestions. Top improvements: (1) make the frontmatter description state when to invoke the skill using concrete prompts, (2) declare the owning team, and (3) add one complete user-input-to-generated-note example and explicit success criteria. Suggested description: “Create and validate a reno release note for an Agent or Cluster Agent change; use when a user asks to add, write, generate, or fix a changelog or release note.”
Useful? React with 👍 / 👎.
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0641fde6b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if c.latestImages == nil { | ||
| c.latestImages = make(map[string]*workloadmeta.ContainerImageMetadata) | ||
| } | ||
| c.latestImages[img.ID] = img |
There was a problem hiding this comment.
Evict stale IDs when mutable image tags move
When containerd emits an image-update event because a mutable tag is repointed to a new config digest, knownImages.addReference removes that name from the previous ID, but this map and the corresponding hidden-byte job are only cleared by an image-delete event. Containerd can update the image record without emitting that delete, so every old digest remains in latestImages and hiddenBytes.jobs; an unfinished job also retries against the now-moved tag before becoming permanently retained. Nodes that repeatedly refresh tags such as latest therefore accumulate full image metadata (including compressed SBOMs) and jobs for the lifetime of the Agent. Remove the previous ID's hidden-byte state when the update leaves it without references.
Useful? React with 👍 / 👎.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a2f0e1d39
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| index++ | ||
| } | ||
| job.finished = true | ||
| c.publishImageLocked(&updated) |
There was a problem hiding this comment.
Avoid retriggering pending SBOM scans
When SBOM collection is enabled and hidden-byte collection finishes while the image's SBOM is still Pending, this publishes another SET event containing that pending state. handleEventBundle treats every such event as an unattempted SBOM and calls Scanner.Scan; if the original request is already being processed, the workqueue marks it dirty and processes it again after Done, potentially doubling expensive Trivy scans and startup CPU/disk work. Suppress the SBOM trigger for hidden-only publications or track scans already in flight.
Useful? React with 👍 / 👎.
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
What does this PR do?
Add default-on container-image measurements for Linux/containerd:
hidden_bytes: file data stored in older layers but hidden by later deletion or replacement.uncompressed_size: regular-file data across all layers, including hidden versions and counting same-layer hardlinks once. Excludes tar overhead, symlinks, directories, whiteout markers, and image metadata.No percentage is sent. The existing image
sizefield is unchanged.Enabled automatically with container-image collection. Set
container_image.hidden_bytes.enabled: false(orDD_CONTAINER_IMAGE_HIDDEN_BYTES_ENABLED=false) to opt out of both measurements.Collection checks the immutable-image cache, then prefers unpacked overlayfs snapshots through SBOM's access helpers. If those cannot be scanned, it streams layer archives already in containerd's local ContentStore: no registry downloads or extraction. Complete measurements are cached and published together; unavailable results are omitted, not zero. Older caches without the total require a fresh scan.
Supported hardlinks count once; their bytes become hidden only when their last visible name disappears. Snapshot scans read metadata; archive scans read and decompress contents but discard them. Scan time, entries, paths, metadata, and archive bytes are bounded.
Motivation
Identify image layers containing hidden file data and provide a comparable uncompressed total for later analysis. These are logical file bytes, not compressed download savings or guaranteed reclaimable disk space. No efficiency score, backend aggregation, or UI changes.
Describe how you validated your changes
Ran the rebuilt Agent in ARM64 minikube with containerd and fakeintake. Assertions checked received config IDs, ordered DiffIDs, every hidden-byte value, and the exact image total against an independent Docker-export/tar-header oracle.
Reproducible QA instructions · Recorded results and reviewer rerun commands.
Additional Notes
451811d2b5dcacross existing module references.🤖 Generated with Claude Code