feat: add auto-fixture for Istanbul E2E coverage collection

[gustavolira]

Summary

• Adds a _coverageCollector auto-fixture to the shared test object that automatically collects window.__coverage__ from the browser after each test
• Guarded by E2E_COLLECT_COVERAGE=1 — zero overhead when disabled (no page.evaluate, no fs operations)
• Writes Istanbul coverage JSON files to coverage/istanbul/ (configurable via COVERAGE_OUTPUT_DIR)

Why

The rhdh-plugin-export-overlays repo builds Istanbul-instrumented dynamic plugins and runs E2E tests against them. The instrumented plugins populate window.__coverage__ in the browser, but nothing collects it today.

Since all 18 workspace test files import test from @red-hat-developer-hub/e2e-test-utils/test, adding an auto-fixture here gives every test automatic coverage collection without modifying any test file.

How it works

1. Instrumented plugin loaded in browser → window.__coverage__ populated
2. Test runs and exercises plugin UI
3. _coverageCollector auto-fixture teardown fires (after test, before page closes)
4. page.evaluate() reads __coverage__ from browser
5. JSON written to coverage/istanbul/<testId>-<timestamp>.json
6. Overlay repo's coverage-reporter merges all JSONs → lcov.info
7. upload-coverage.sh sends lcov to Codecov with --slug --sha --flag

Design decisions

• auto: true — runs for every test without test file changes
• scope: "test" — collects per-test, not per-worker
• await use() first — teardown runs AFTER the test but BEFORE page closes (Playwright tears down in reverse dependency order)
• try/catch with empty catch — best-effort; if the page crashed or plugin wasn't instrumented, silently skip
• _ prefix — Playwright convention for internal auto-fixtures
• No new dependencies — uses only node:fs and existing path import

Related PRs

• Instrumentation pipeline: feat: E2E coverage instrumentation with CI pipeline for all dynamic plugins rhdh-plugin-export-overlays#2383
• Codecov flags: feat: add E2E coverage flags for overlay cross-repo uploads rhdh-plugins#2997
• Jira: RHIDP-13411

Test plan

• yarn build succeeds
• yarn typecheck passes
• With E2E_COLLECT_COVERAGE=1 and an instrumented plugin: JSON files appear in coverage/istanbul/
• Without E2E_COLLECT_COVERAGE: no JSON files, no performance impact
• When page crashes mid-test: no error thrown, test result unaffected

🤖 Generated with Claude Code

[subhashkhileri]

Review:

[subhashkhileri]

@gustavolira

---

Hey, here's what needs to change in rhdh-e2e-test-utils to support the __coverage tag swap for PR checks.

Everything is in src/utils/plugin-metadata.ts — three small changes:

1. Add backstage.role to the types (so we know which plugins are frontend)

PackageCRD interface (line 19) needs backstage.role:

interface PackageCRD {
  spec?: {
    packageName?: string;
    dynamicArtifact?: string;
    backstage?: {
      role?: string;
    };
    appConfigExamples?: Array<{ ... }>;
  };
}

PluginMetadata interface (line 12) needs role:

export interface PluginMetadata {
  packagePath: string;
  pluginConfig: Record<string, unknown>;
  packageName: string;
  sourceFile: string;
  role?: string;
}

2. Parse the role in parseMetadataFile (line 212)

Just read it from the parsed YAML and include in the return:

const role = parsed?.spec?.backstage?.role;
// ... in the return:
return { packagePath, pluginConfig: pluginConfig || {}, packageName, sourceFile: filePath, role };

3. The actual swap — in resolvePluginPackages (line 472-479)

This is the existing PR branch that already only runs when GIT_PR_NUMBER is set:

if (prOciUrls) {
  const prUrl = prOciUrls.get(displayName);
  if (prUrl) {
    const usesCoverage =
      process.env.E2E_COLLECT_COVERAGE === "1" &&
      metadata.role === "frontend-plugin";
    const resolved = usesCoverage
      ? prUrl.replace(/(:[^!]+)/, "$1__coverage")
      : prUrl;
    console.log(`[PluginMetadata] PR: ${pkg} → ${resolved}`);
    return { ...plugin, package: resolved };
  }
}

That regex appends __coverage to the tag while keeping the !alias intact:
plugin:pr_1845__1.5.0!plugin → plugin:pr_1845__1.5.0__coverage!plugin

That's it — only affects PR checks, only when E2E_COLLECT_COVERAGE=1, only for frontend plugins. Nightly, {{inherit}}, and local dev paths are untouched.

---

[gustavolira]

✅ Implemented Coverage Image Swap

Implemented all changes requested in #95 (comment)

Changes Made

1. Added role field to types (src/utils/plugin-metadata.ts)

export interface PluginMetadata {
  packagePath: string;
  pluginConfig: Record<string, unknown>;
  packageName: string;
  sourceFile: string;
  role?: string;  // ← Added
}

interface PackageCRD {
  spec?: {
    packageName?: string;
    dynamicArtifact?: string;
    backstage?: {  // ← Added
      role?: string;
    };
    appConfigExamples?: Array<{...}>;
  };
}

2. Parse role in parseMetadataFile()

const role = parsed?.spec?.backstage?.role;
// ... in return:
return {
  packagePath,
  pluginConfig: pluginConfig || {},
  packageName,
  sourceFile: filePath,
  role,  // ← Added
};

3. Implemented __coverage tag swap in resolvePluginPackages()

if (prOciUrls) {
  const prUrl = prOciUrls.get(displayName);
  if (prUrl) {
    const usesCoverage =
      process.env.E2E_COLLECT_COVERAGE === "1" &&
      metadata.role === "frontend-plugin";
    const resolved = usesCoverage
      ? prUrl.replace(/(:[^!]+)/, "$1__coverage")
      : prUrl;
    console.log(`[PluginMetadata] PR: ${pkg} → ${resolved}`);
    return { ...plugin, package: resolved };
  }
}

How It Works

When running E2E tests in PR mode with E2E_COLLECT_COVERAGE=1:

• ✅ Frontend plugins → auto-swap to __coverage tag
• ✅ Backend plugins → keep original tag (no browser coverage)
• ✅ Preserves OCI alias: plugin:pr_123__1.0.0!alias → plugin:pr_123__1.0.0__coverage!alias

Scope

Only affects PR mode (GIT_PR_NUMBER set). Nightly mode, {{inherit}}, and local dev paths are unchanged.

---

Commit: 82061d5

Ready for review!