fix: populate index-pattern field lists

[TackAdam]

Summary

Pre-generated index patterns (logs-otel-v1*, otel-v1-apm-span*, otel-v2-apm-service-map*) were being created by the Dashboards initialization script with an empty fields attribute. In the UI this appeared as "Fields (0)", and visualizations failed with errors such as:

Could not locate that index-pattern-field (id: endTime)

Recreating the dataset through the UI repaired the issue because the dataset creation flow populates the field list, but the pre-generated index patterns created during stack initialization remained permanently broken.

This PR populates index pattern fields during initialization and refactors the delayed refresh into a separate background process so it never blocks stack startup or Helm install/upgrade operations.

---

Root cause

Two issues combined to produce the bug.

1. The initialization script never populated fields at creation time

The script creates index patterns directly through the Saved Objects API:

POST /api/saved_objects/index-pattern

Unlike the dataset creation wizard, this code path never performs a _fields_for_wildcard request, so the saved object is persisted with an empty fields attribute.

OpenSearch Dashboards previously compensated by lazily fetching fields on first use, but that behavior was removed upstream in OpenSearch-Dashboards#11653. PR #12236 restored field prefetching for datasets created through the UI wizard, but pre-generated index patterns created out-of-band still bypass that logic.

2. The _fields_for_wildcard refresh call itself was broken

The existing refresh_index_pattern_fields() implementation passed repeated meta_fields query parameters:

meta_fields=_source
meta_fields=_id
...

This version of OpenSearch Dashboards rejects those parameters with a 400 response (definition for this key is missing), so even when the refresh function executed, it silently failed and never populated any fields.

The fix removes the unnecessary meta_fields parameters entirely and sends only the pattern parameter using requests' params dictionary for proper URL encoding.

---

Changes

1. Fix the _fields_for_wildcard API call

Updated refresh_index_pattern_fields() to:

• Remove the unsupported meta_fields query parameters.
• Send only the pattern parameter.
• Use requests' params dictionary for proper URL encoding.

This restores compatibility with current OpenSearch Dashboards versions.

---

2. Populate fields immediately after creation

After each index pattern is created, main() now immediately invokes refresh_index_pattern_fields(), mirroring the behavior of the dataset creation wizard:

• Fetch fields via _fields_for_wildcard
• Persist the populated field list back into the saved object

This fixes environments where matching indices already exist.

---

3. Replace the delayed refresh with a bounded retry loop

On a fresh deployment, matching indices often exist before any documents have been indexed, so the initial refresh legitimately returns no fields.

Previously the script:

• Slept for 10 minutes.
• Retried once.
• Silently gave up if fields were still unavailable.

This has been replaced with a bounded retry loop that:

• Performs up to 12 attempts
• Waits 5 minutes between attempts
• Runs for approximately 1 hour maximum
• Retries only index patterns that are still empty
• Exits immediately once every pattern has been populated
• Resolves index-pattern IDs by title on each attempt, making the process resilient regardless of startup ordering

The first refresh occurs immediately, so environments that already contain data complete within seconds.

---

4. Run the retry loop outside the initialization hook

The initialization script runs as a Helm post-install / post-upgrade hook, and Helm waits for hook Jobs to complete before finishing an install or upgrade.

Running an hour-long retry loop inside that hook would unnecessarily block every deployment.

The script now supports two execution modes:

• Default (main)

  • Performs the normal initialization
  • Fast one-shot execution
  • Used by the existing Helm hook

• refresh-loop

  • Runs only the bounded field refresh retry loop
  • Executes independently of the initialization hook

Docker Compose

Added a new service:

• opensearch-dashboards-field-refresh

Unlike the previous implementation, it intentionally does not depend on the initialization container completing successfully. A service_completed_successfully dependency on a one-shot container causes docker compose up --wait (used by test/e2e.sh) to exit non-zero when the init container finishes.

Instead, the refresh loop:

• Waits for Dashboards to become available.
• Resolves index-pattern IDs by title on each retry.
• Works correctly whether it starts before or after the initialization container.

Helm

Added:

• field-refresh-job.yaml
• field-refresh-configmap.yaml

The refresh Job is intentionally not a Helm hook so it does not delay installs or upgrades.

Additional implementation details:

• Job name is suffixed with .Release.Revision to avoid Job spec.template immutability during upgrades.
• Uses a dedicated ConfigMap because the existing initialization ConfigMap is itself a hook resource and cannot reliably be mounted by a non-hook Job.
• Adds activeDeadlineSeconds and ttlSecondsAfterFinished for automatic cleanup.

---

5. Enable unbuffered logging

All Python invocations now use:

python -u

This applies to:

• Docker Compose initialization
• Docker Compose field refresh
• Helm initialization Job

Previously Python buffered stdout until process exit, making the initialization flow difficult to monitor and debug.

---

Testing

Verified with a clean Docker Compose deployment:

docker compose down -v
docker compose up -d

Results:

• opensearch-dashboards-init executes main() and exits successfully.
• opensearch-dashboards-field-refresh starts independently.
• All three index patterns are populated on the first retry:
  • logs-otel-v1* → 135 fields
  • otel-v1-apm-span* → 280 fields
  • otel-v2-apm-service-map* → 13 fields
• Background refresh completes in approximately 10 seconds.
• Ran test/e2e.sh end-to-end locally; all checks passed (EXIT 0).
• helm template renders both Jobs, both ConfigMaps, correct hook/non-hook annotations, correct command arguments, deadlines, and unbuffered Python execution.
• Both Python scripts compile successfully.
• docker compose config validates.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.