aws_securityhub: raise total_fields limit to prevent transform failures - #20603
aws_securityhub: raise total_fields limit to prevent transform failures#20603kcreddy wants to merge 4 commits into
Conversation
The finding data stream already declares ~1,500 fields against a 2000 total_fields.limit, leaving little room for dynamically mapped nested resources. The latest_findings transform destination hits that ceiling and goes health red. Raise the limit to 5000 on the finding stream and both transform destinations.
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
…limit is applied on upgrade.
kcreddy
added
bugfix
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| changes: | ||
| - description: Raise finding and transform destination `total_fields.limit` from 2000 to 5000 to prevent transform failures as dynamic resource fields grow. | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/0 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/aws_securityhub/changelog.yml:6
Both new changelog entries link to placeholder PR pull/0; replace them with this PR's URL (pull/20603) before merge.
Details
The two changes added under version 1.2.2 (changelog.yml lines 6 and 9) both use https://github.com/elastic/integrations/pull/0 as their link. PR 0 does not exist, so the published package changelog for 1.2.2 ships two dead links, unlike every earlier entry in this file which points at its real PR (e.g. 1.2.1 -> pull/20421).
Recommendation:
Point both entries at this PR:
- version: "1.2.2"
changes:
- description: Raise finding and transform destination `total_fields.limit` from 2000 to 5000 to prevent transform failures as dynamic resource fields grow.
type: bugfix
link: https://github.com/elastic/integrations/pull/20603
- description: Bump transform versions and destination index names so the new field limit is applied on upgrade.
type: bugfix
link: https://github.com/elastic/integrations/pull/20603🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Raise finding and transform destination `total_fields.limit` from 2000 to 5000 to prevent transform failures as dynamic resource fields grow. | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/0 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/aws_securityhub/changelog.yml:6
Changelog links a different PR number
Details
This changelog entry's link: points at pull/0, but it was added in PR #20603. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20603🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/0 | ||
| - description: Bump transform versions and destination index names so the new field limit is applied on upgrade. | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/0 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/aws_securityhub/changelog.yml:9
Changelog links a different PR number
Details
This changelog entry's link: points at pull/0, but it was added in PR #20603. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20603🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
✅ All changelog entries have the correct PR link. |
| mapping: | ||
| total_fields: | ||
| limit: 2000 | ||
| limit: 5000 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/aws_securityhub/elasticsearch/transform/latest_findings/manifest.yml:16
The transform destination index templates raise the limit but still have no ignore_dynamic_beyond_limit, so an over-budget mapping update fails the transform hard instead of degrading; add index.mapping.total_fields.ignore_dynamic_beyond_limit: true alongside the new limit in both transform manifests.
Details
The linked issue's own analysis is that the finding data stream survives an exhausted mapping budget only because Fleet composes logs@settings into logs index templates, and that template sets index.mapping.total_fields.ignore_dynamic_beyond_limit: true. Transform destination index templates are built by a different code path that composes only the package component templates plus ecs@mappings, so they get no such escape hatch — which is exactly why the transform goes health red with Limit of total fields [2000] has been exceeded while ingest keeps working.
This PR raises the ceiling on both destination templates but does not close that asymmetry: the failure mode is unchanged, only deferred. aws_securityhub.finding.resources.tags.* is a wildcard object over customer-defined tag keys and resources.data declares only 2 of the many ASFF *Details schemas, so the dynamic portion of the mapping is unbounded by the package. Setting the flag turns the remaining risk from 'latest view goes stale and the transform retry-loops' into 'a few late dynamic fields are left unmapped' — the same trade-off the data stream already makes.
The same line applies to packages/aws_securityhub/elasticsearch/transform/latest_cdr_vulnerabilities/manifest.yml (line 7), whose destination_index_template declares no mappings block at all and therefore also maps dynamically. Both destination indexes are being recreated by this PR's index-name bump, so the setting would take effect on upgrade at no extra cost.
Note this is additive to, not a replacement for, the flattened remap of resources.data/resources.tags that the issue records as deliberately out of scope.
Recommendation:
Add the flag next to the raised limit in both transform manifests:
start: true
destination_index_template:
mappings:
dynamic: true
dynamic_templates:
- strings_as_keyword:
match_mapping_type: string
mapping:
ignore_above: 1024
type: keyword
date_detection: true
settings:
index:
mapping:
total_fields:
limit: 5000
ignore_dynamic_beyond_limit: trueAnd in latest_cdr_vulnerabilities/manifest.yml:
start: true
destination_index_template:
settings:
index:
mapping:
total_fields:
limit: 5000
ignore_dynamic_beyond_limit: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| mapping: | ||
| total_fields: | ||
| limit: 2000 | ||
| limit: 5000 |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/aws_securityhub/elasticsearch/transform/latest_cdr_vulnerabilities/manifest.yml:7
limit is indented four spaces under total_fields instead of two in both transform manifests; re-indent it to two while touching this line so it matches the data stream manifest and the rest of the repo.
Details
total_fields: sits at 8 spaces and limit: at 12, a 4-space step where every other level in the file steps by 2. It parses fine, but these two transform manifests are the only files in the repository with this indentation for the total_fields/limit pair — packages/aws_securityhub/data_stream/finding/manifest.yml, edited in the same PR, uses the standard 2-space step. The same off-by-two exists at packages/aws_securityhub/elasticsearch/transform/latest_findings/manifest.yml line 16.
Purely cosmetic; worth correcting only because both lines are already being modified here.
Recommendation:
Re-indent limit by two spaces relative to total_fields:
start: true
destination_index_template:
settings:
index:
mapping:
total_fields:
limit: 5000🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits 8c806b7, cad2e59 — 1 medium, 1 low
Issues found across earlier commits 04dec84 — 3 medium
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
cc @kcreddy |
1 participant
Proposed commit message
Checklist
changelog.ymlfile.Related issues