Integration test suite duration (~5 hours 40 minutes) - identify optimization opportunities

[shreyah963]

Description

The SQL integration tests are taking around 5 hours and 40 minutes to complete. The secured integration tests are taking substantially longer than the unsecured tests. Total integ test runtime across all distributions is around 5 hours and 40 minutes, with the secured tests appearing to account for a disproportionate share of that time.

Context

• 5 hours for a single component's integ tests puts pressure on the overall release pipeline since all components must pass before RC
• As the test suite grows, this will only get worse without structural changes

Ask

Would appreciate investigation into:

1. What's driving the secured vs unsecured gap?
2. Which secured test suites are the most expensive?
3. Optimization opportunities

Examples

https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/11474/pipeline/156

• https://ci.opensearch.org/ci/dbc/integ-test/3.8.0/12113/linux/x64/tar/test-results/11474/integ-test/sql/with-security/stdout.txt
• https://ci.opensearch.org/ci/dbc/integ-test/3.8.0/12113/linux/x64/tar/test-results/11474/integ-test/sql/without-security/stdout.txt

https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/11453/pipeline/156

• https://ci.opensearch.org/ci/dbc/integ-test/3.8.0/12102/linux/arm64/tar/test-results/11453/integ-test/sql/with-security/stdout.txt
• https://ci.opensearch.org/ci/dbc/integ-test/3.8.0/12102/linux/arm64/tar/test-results/11453/integ-test/sql/without-security/stdout.txt

https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/11458/pipeline/153

https://build.ci.opensearch.org/blue/organizations/jenkins/integ-test/detail/integ-test/11457/pipeline/153

cc: @opensearch-project/engineering-effectiveness