Feat: Make system index auto-expand replicas setting configurable

[theundefined]

Description

This PR makes the replica configuration of system indices created by the Security Analytics plugin configurable.

System indices created by the Security Analytics plugin have had hardcoded 'index.auto_expand_replicas' values (such as '0-20' or '0-all'). In large production clusters, this causes indices to replicate across many or all nodes, consuming excessive cluster resources (disk space and JVM heap), while overriding matching index templates configured by administrators.

This commit replaces the hardcoded replica settings with three new dynamic cluster settings:

• plugins.security_analytics.auto_expand_system_index_replicas (defaults to true)
• plugins.security_analytics.min_system_index_replicas (defaults to 0)
• plugins.security_analytics.max_system_index_replicas (defaults to 20)

By setting auto_expand_system_index_replicas to false, administrators can disable programmatic auto-expansion on index creation and allow custom index templates (such as those overriding number_of_replicas or auto_expand_replicas) to be fully respected.

Related Issues

Resolves # [Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 947c99e)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Invalid setting value When AUTO_EXPAND_SYSTEM_INDEX_REPLICAS is false, getSystemIndexAutoExpandReplicas returns the string "false", but OpenSearch's index.auto_expand_replicas setting only accepts either false (boolean/no expansion) or a range like min-max. More importantly, applying .put("index.auto_expand_replicas", "false") on every index creation still overrides any matching admin-configured index templates, which contradicts the stated goal in the PR description ("allow custom index templates ... to be fully respected"). To honor templates, the setting should be omitted entirely when auto-expand is disabled, rather than explicitly set to "false". public static String getSystemIndexAutoExpandReplicas(ClusterService clusterService) { if (clusterService.getClusterSettings().get(AUTO_EXPAND_SYSTEM_INDEX_REPLICAS) == true) { int min = clusterService.getClusterSettings().get(MIN_SYSTEM_INDEX_REPLICAS); int max = clusterService.getClusterSettings().get(MAX_SYSTEM_INDEX_REPLICAS); return min + "-" + max; } else { return "false"; } } Missing validation MIN_SYSTEM_INDEX_REPLICAS and MAX_SYSTEM_INDEX_REPLICAS are declared without min-value or cross-setting validation. A user can set min > max, or negative values, producing an invalid index.auto_expand_replicas string (e.g., "5-2" or "-1-20"), which will cause subsequent system index creation calls to fail. Consider adding intSetting lower bounds and a validator ensuring min <= max. public static final Setting<Integer> MIN_SYSTEM_INDEX_REPLICAS = Setting.intSetting( "plugins.security_analytics.min_system_index_replicas", 0, Setting.Property.NodeScope, Setting.Property.Dynamic ); public static final Setting<Integer> MAX_SYSTEM_INDEX_REPLICAS = Setting.intSetting( "plugins.security_analytics.max_system_index_replicas", 20, Setting.Property.NodeScope, Setting.Property.Dynamic ); Behavior change Previously initCorrelationAlertIndex did not set auto_expand_replicas at all (only the two correlation indices did in the old code path with the hardcoded value). Now it always sets it. Verify this is intentional; if the prior omission was deliberate for the alert index, this changes replica behavior for existing deployments upgrading. public void initCorrelationAlertIndex(ActionListener<CreateIndexResponse> actionListener) throws IOException { Settings correlationAlertSettings = Settings.builder() .put("index.hidden", true) .put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1) .put("index.auto_expand_replicas", SecurityAnalyticsSettings.getSystemIndexAutoExpandReplicas(clusterService)) .build();

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 947c99e
Explore these optional code suggestions:

Category	Suggestion	Impact
General	Enforce non-negative bounds for replica settings Add lower bounds to these int settings to prevent negative values from being accepted at setting-update time (OpenSearch's intSetting supports a minValue overload). This ensures invalid replica counts are rejected immediately rather than only failing later during index creation. src/main/java/org/opensearch/securityanalytics/settings/SecurityAnalyticsSettings.java [25-35] public static final Setting<Integer> MIN_SYSTEM_INDEX_REPLICAS = Setting.intSetting( "plugins.security_analytics.min_system_index_replicas", - 0, + 0, 0, Setting.Property.NodeScope, Setting.Property.Dynamic ); public static final Setting<Integer> MAX_SYSTEM_INDEX_REPLICAS = Setting.intSetting( "plugins.security_analytics.max_system_index_replicas", - 20, + 20, 0, Setting.Property.NodeScope, Setting.Property.Dynamic ); Suggestion importance[1-10]: 6 __ Why: Adding a minValue of 0 to the intSetting definitions is a valid and simple improvement that rejects invalid negative replica counts at setting-update time, providing earlier and clearer failure feedback.	Low
	Validate replica range bounds before use Validate that min is non-negative and max >= min before constructing the range string. Without validation, misconfigured settings (e.g., max < min, or negative values) will produce an invalid auto_expand_replicas value like "5-2" that will fail index creation at runtime with a confusing error. Consider falling back to defaults or throwing a clear IllegalArgumentException. src/main/java/org/opensearch/securityanalytics/settings/SecurityAnalyticsSettings.java [37-45] public static String getSystemIndexAutoExpandReplicas(ClusterService clusterService) { if (clusterService.getClusterSettings().get(AUTO_EXPAND_SYSTEM_INDEX_REPLICAS) == true) { int min = clusterService.getClusterSettings().get(MIN_SYSTEM_INDEX_REPLICAS); int max = clusterService.getClusterSettings().get(MAX_SYSTEM_INDEX_REPLICAS); + if (min < 0 || max < min) { + throw new IllegalArgumentException( + "Invalid system index replica bounds: min=" + min + ", max=" + max); + } return min + "-" + max; } else { return "false"; } } Suggestion importance[1-10]: 5 __ Why: Adding validation for the replica range bounds is a reasonable defensive improvement that would produce clearer error messages for misconfigurations, though the underlying index creation would eventually fail with a similar error.	Low

---

Previous suggestions

Suggestions up to commit ee61441

Category	Suggestion	Impact
General	Add bounds validation to replica settings The MIN_SYSTEM_INDEX_REPLICAS and MAX_SYSTEM_INDEX_REPLICAS settings lack bounds validation. Users could set negative values or a min greater than max, causing invalid auto_expand_replicas values like "5-2" or "-1-20" which will cause index creation failures. Add minValue and validators to ensure min >= 0 and min <= max. src/main/java/org/opensearch/securityanalytics/settings/SecurityAnalyticsSettings.java [25-35] public static final Setting<Integer> MIN_SYSTEM_INDEX_REPLICAS = Setting.intSetting( "plugins.security_analytics.min_system_index_replicas", - 0, + 0, 0, Setting.Property.NodeScope, Setting.Property.Dynamic ); public static final Setting<Integer> MAX_SYSTEM_INDEX_REPLICAS = Setting.intSetting( "plugins.security_analytics.max_system_index_replicas", - 20, + 20, 0, Setting.Property.NodeScope, Setting.Property.Dynamic ); Suggestion importance[1-10]: 6 __ Why: Adding a minimum bound of 0 to the replica settings is a reasonable validation that prevents negative values which would cause invalid auto_expand_replicas strings and index creation failures.	Low
Possible issue	Prevent invalid replica range when min>max If a user sets MIN_SYSTEM_INDEX_REPLICAS greater than MAX_SYSTEM_INDEX_REPLICAS, the produced string (e.g. "10-5") will be rejected by OpenSearch and index creation will fail. Guard against this by swapping or clamping the values, or by validating and falling back to safe defaults. src/main/java/org/opensearch/securityanalytics/settings/SecurityAnalyticsSettings.java [37-45] public static String getSystemIndexAutoExpandReplicas(ClusterService clusterService) { if (clusterService.getClusterSettings().get(AUTO_EXPAND_SYSTEM_INDEX_REPLICAS) == true) { int min = clusterService.getClusterSettings().get(MIN_SYSTEM_INDEX_REPLICAS); int max = clusterService.getClusterSettings().get(MAX_SYSTEM_INDEX_REPLICAS); + if (min > max) { + max = min; + } return min + "-" + max; } else { return "false"; } } Suggestion importance[1-10]: 5 __ Why: The suggestion correctly identifies a potential issue where min > max would produce an invalid string. However, silently clamping may hide user misconfiguration; validation at setting-time might be preferable.	Low

[github-actions]

Persistent review updated to latest commit 947c99e