[QA EPIC] OpenSearch Migration — End-to-End QA Validation (Phases 0–3)

[fabrizzio-dotCMS]

Description

QA sub-epic for the ES → OpenSearch migration pipeline under #34164. Covers end-to-end validation of the dual-write pipeline across Phases 0, 1, and 2 (partial), plus acceptance criteria for PR #35632 (startup hardening).

Parent epic: #34164 — Migrate Mapping Layer classes
Fix PR: #35632 — Startup validation hardening, haltMigration(), phase-aware index init, thread-safe formatter
Test plans:

• docs/backend/OPENSEARCH_MIGRATION_TEST_PLAN.md — base scenarios A–F (Phases 0–3)
• local/test-plans/35631-35632-test-plan.md — extended scenarios for PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632 (Groups 1–8, TC-001–TC-025)

Environment Setup

# Start migration stack (ES 7.10 + Kibana + OpenSearch 3.x + Dashboards)
docker compose -f docker/docker-compose-examples/os-migration/docker-compose.yml up -d

Service	URL	Purpose
Elasticsearch 7.10	http://localhost:9200	ES REST API
Kibana	http://localhost:5601	ES index inspection / queries
OpenSearch 3.x	http://localhost:9201	OS REST API
OpenSearch Dashboards	http://localhost:5602	OS index inspection / queries

# dotmarketing-config.properties — minimal OS connection
OS_ENDPOINTS=http://localhost:9201
OS_AUTH_TYPE=BASIC
OS_AUTH_BASIC_USER=admin
OS_AUTH_BASIC_PASSWORD=admin
OS_TLS_ENABLED=false

# Phase flag: 0=ES only  1=dual-write+ES reads  2=dual-write+OS reads  3=OS only (NOT TESTABLE)
FEATURE_FLAG_OPEN_SEARCH_PHASE=0

Migration Phase Reference

Phase	Writes	Reads	ES Status	OS Write Failures	Testable?
0	ES only	ES only	Active	n/a	Yes — fully
1	ES + OS	ES only	Active	Fire-and-forget (logged)	Yes — fully
2	ES + OS	OS only	Active (fallback)	Fire-and-forget (logged)	Partially — writes only
3	OS only	OS only	Decommissioned	Propagate to caller	No — query layer not migrated

Scope

1. Zero regression on ES-backed functionality (Phase 0 baseline)
2. Correct dual-write in Phase 1 — every ES write is mirrored to OS; OS failures are fire-and-forget
3. NEW (PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632): haltMigration() auto-resets phase to 0 when OS is unreachable or has the wrong version at startup
4. NEW (PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632): hasEmptyIndices() is phase-aware — empty OS index in Phase 1 does not trigger a false ES reindex
5. NEW (PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632): threadSafeTimestampFormatter prevents corrupted index names under concurrent requests
6. Index lifecycle operations (activate, deactivate, delete, reindex) — including ES-only vs OS-only name variants
7. Content Type and contentlet create/delete sync between ES and OS shadow index
8. Multi-node cluster startup consistency in Phase 1
9. Invalid OS_ENDPOINTS graceful degradation
10. REST API security: DELETE /v1/esindex requires admin authentication
11. Complex query equivalence between Kibana and OpenSearch Dashboards

DB Validation Queries

-- All index rows
SELECT * FROM indicies ORDER BY index_name;
-- OS rows only (os:: prefix = DB artifact, never visible to callers)
SELECT * FROM indicies WHERE index_name LIKE 'os::%';
-- Count per provider
SELECT CASE WHEN index_name LIKE 'os::%' THEN 'OS' ELSE 'ES' END AS provider, COUNT(*) FROM indicies GROUP BY 1;
-- Expected Phase 0: 2 rows (ES only). Expected Phase 1 fresh: 4 rows (2 ES + 2 OS).

Child Tasks

PR #35632 Acceptance & Startup Hardening

• [QA-G1] OpenSearch Migration — PR #35632 Acceptance: Startup Hardening (TC-001–TC-005) #35635 — G1: PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632 Acceptance — Startup Hardening (TC-001–TC-005)
• [QA-G2] OpenSearch Migration — Invalid DOT_OS_ENDPOINTS Values (TC-006–TC-008) #35636 — G2: Invalid DOT_OS_ENDPOINTS Values (TC-006–TC-008)

Cluster & Infrastructure

• [QA-G3] OpenSearch Migration — Multi-Node Cluster Phase 1 (TC-009–TC-011) #35637 — G3: Multi-Node Cluster Phase 1 (TC-009–TC-011)

Content Lifecycle Sync

• [QA-G4] OpenSearch Migration — Content Type Lifecycle Sync (TC-012–TC-014) #35638 — G4: Content Type Lifecycle Sync (TC-012–TC-014)
• [QA-G5] OpenSearch Migration — Contentlet Delete Sync (TC-015) #35639 — G5: Contentlet Delete Sync (TC-015)

Index Lifecycle

• [QA-G6] OpenSearch Migration — Index Delete: ES vs OS Behavior (TC-016–TC-020) #35640 — G6: Index Delete — ES vs OS Behavior (TC-016–TC-020)

Security & Query Validation

• [QA-G7] OpenSearch Migration — REST Security: 401 / 403 on Index Endpoints (TC-021–TC-022) #35641 — G7: REST Security — 401/403 on Index Endpoints (TC-021–TC-022)
• [QA-G8] OpenSearch Migration — Complex Query Equivalence: Kibana vs OS Dashboards (TC-023–TC-025) #35642 — G8: Complex Query Equivalence — Kibana vs OS Dashboards (TC-023–TC-025)

Acceptance Criteria

• All G1–G8 child tasks pass with documented results
• Scenario B-7 in docs/backend/OPENSEARCH_MIGRATION_TEST_PLAN.md updated to reflect haltMigration() behavior (PR fix(os-migration): harden startup validation, add emergency halt path, fix thread-safe formatter #35632)
• DB validation queries pass at each phase transition
• Phase 3 scenarios remain deferred (blocked on query layer migration)
• Any match_phrase_prefix divergence (TC-023) filed as a separate bug

Additional Context

Depends on: #34164
Fix PR: #35632