Skip to content

Commit 6f7c21c

Browse files
feat: Add e2e test documentation site with generation scripts and VitePress configuration. (#24837)
* feat: Add e2e test documentation site with generation scripts and VitePress configuration. * feat: migrate e2e documentation generation to Playwright and update service entity permission tests. * docs: Improve Playwright E2E test documentation with JSDoc comments and update doc generation. * docs: Add link to E2E Test Documentation in README. * fix: Correctly escape backslashes in Playwright generated markdown tables and refine the documentation CI workflow. * minor fix * fix node version issue * minor change
1 parent b1e577e commit 6f7c21c

32 files changed

Lines changed: 8188 additions & 148 deletions
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
name: Verify Playwright Documentation
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- 'openmetadata-ui/src/main/resources/ui/playwright/e2e/**/*.spec.ts'
7+
- 'openmetadata-ui/src/main/resources/ui/playwright/doc-generator/**'
8+
branches:
9+
- main
10+
11+
permissions:
12+
contents: write
13+
pull-requests: write
14+
15+
jobs:
16+
verify-docs:
17+
runs-on: ubuntu-latest
18+
steps:
19+
- uses: actions/checkout@v3
20+
21+
- name: Setup Node.js
22+
uses: actions/setup-node@v3
23+
with:
24+
node-version-file: 'openmetadata-ui/src/main/resources/ui/.nvmrc'
25+
cache: 'yarn'
26+
cache-dependency-path: 'openmetadata-ui/src/main/resources/ui/yarn.lock'
27+
28+
- name: Install Dependencies
29+
working-directory: openmetadata-ui/src/main/resources/ui
30+
run: yarn install --frozen-lockfile --ignore-scripts
31+
32+
- name: Install Playwright Browsers
33+
# We need browsers installed for 'playwright test --list' to work in some versions,
34+
# although strictly speaking just listing shouldn't require binaries if configured right.
35+
# Adding just in case.
36+
working-directory: openmetadata-ui/src/main/resources/ui
37+
run: npx playwright install --with-deps
38+
39+
- name: Generate Documentation
40+
working-directory: openmetadata-ui/src/main/resources/ui
41+
run: node playwright/doc-generator/generate.js
42+
43+
- name: Check for Changes
44+
id: git-check
45+
run: |
46+
git diff --exit-code --quiet openmetadata-ui/src/main/resources/ui/playwright/docs || echo "changes=true" >> $GITHUB_OUTPUT
47+
48+
- name: Commit and Push Changes
49+
if: steps.git-check.outputs.changes == 'true'
50+
run: |
51+
git config --global user.name 'github-actions[bot]'
52+
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
53+
git add openmetadata-ui/src/main/resources/ui/playwright/docs
54+
git commit -m "docs: auto-generate playwright documentation"
55+
git push
56+
57+
- name: Comment on PR
58+
if: steps.git-check.outputs.changes == 'true'
59+
uses: peter-evans/create-or-update-comment@v4
60+
with:
61+
issue-number: ${{ github.event.pull_request.number }}
62+
body: |
63+
## 📝 Documentation Auto-Updated
64+
65+
The Playwright documentation has been automatically updated to match the changes in this PR.
66+
67+
* **Generated by**: `playwright-docs-check.yml`
68+
* **Status**: ✅ Updated and pushed to branch.

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,3 +181,5 @@ hive-mind-prompt-*.txt
181181
.claude/
182182
.claude-flow/
183183
memory
184+
185+

openmetadata-ui/src/main/resources/ui/.eslintignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,6 @@ src/jsons/connectionSchemas/
3939

4040
# generated type
4141
src/generated
42+
43+
# docs
44+
playwright/doc-generator/

openmetadata-ui/src/main/resources/ui/.eslintrc.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,8 @@ overrides:
244244
- off
245245
react-hooks/rules-of-hooks:
246246
- off
247+
max-len:
248+
- off
247249

248250
# i18next rule is not required for js, jsx, json and test file
249251
- files:

openmetadata-ui/src/main/resources/ui/.lintstagedrc.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,7 @@
2525

2626
# Run Prettier
2727
'src/**': prettier --write
28+
29+
# Auto-generate Playwright documentation
30+
'playwright/e2e/**/*.spec.ts':
31+
- node playwright/doc-generator/generate.js

openmetadata-ui/src/main/resources/ui/README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,10 @@ Playwright is already included in the project dependencies. If you need to insta
140140
npx playwright install
141141
```
142142

143+
### Documentation
144+
145+
For a detailed breakdown of all end-to-end tests, including test steps and coverage metrics, please refer to the **[E2E Test Documentation](./playwright/docs/README.md)**.
146+
143147
### Running Tests
144148

145149
#### 1. Run All Tests

openmetadata-ui/src/main/resources/ui/package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,8 @@
3737
"playwright:run": "playwright test",
3838
"playwright:open": "playwright test --ui",
3939
"playwright:codegen": "playwright codegen",
40-
"generate:app-docs": "node generateApplicationDocs.js"
40+
"generate:app-docs": "node generateApplicationDocs.js",
41+
"generate:e2e-docs": "node playwright/doc-generator/generate.js"
4142
},
4243
"dependencies": {
4344
"@analytics/session-utils": "^0.1.17",
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Playwright Documentation Generator
2+
3+
This directory contains the custom tooling used to generate the E2E test documentation found in `../docs`.
4+
5+
## 🏗️ Architecture
6+
7+
The generator consists of three main components:
8+
9+
1. **`generate.js`** (Orchestrator)
10+
* Main entry point.
11+
* Manages the `DOMAIN_MAPPING` to categorize tests into domains (e.g., Governance, Platform).
12+
* Calls the loader to get test data.
13+
* Calls the markdown generator to create strings.
14+
* Writes the final `.md` files to the output directory.
15+
16+
2. **`playwright-loader.js`** (Data Provider)
17+
* **Test Discovery**: Uses `npx playwright test --list --reporter=json` to get an accurate list of all tests from Playwright itself.
18+
* **Temp Config**: Created a temporary `playwright.docs.temp.config.ts` during execution to ensure *all* tests (including ignored/nightly ones) are included in the discovery phase.
19+
* **AST Parsing**: Uses the TypeScript Compiler API (`typescript` package) to parse the source code of test files.
20+
* Extracts `test.step('Step Name', ...)` calls to provide granular details.
21+
* Extracts JSDoc comments for test descriptions.
22+
23+
3. **`markdown.js`** (Renderer)
24+
* Contains template functions (`generateIndexMarkdown`, `generateDomainMarkdown`).
25+
* Handles the calculation of metrics like "Total Scenarios".
26+
* **Metrics Logic**:
27+
* **Tests**: Raw count of `test(...)` blocks.
28+
* **Total Scenarios**: Calculated as `(Steps > 0 ? Steps : 1)`. If a test has steps, each step is a scenario. If not, the test itself counts as one.
29+
30+
## 🚀 Usage
31+
32+
To regenerate the documentation manually:
33+
34+
```bash
35+
# From the project root
36+
node openmetadata-ui/src/main/resources/ui/playwright/doc-generator/generate.js
37+
```
38+
39+
## 🛠️ Key Features
40+
41+
* **Total Scenarios Metric**: Provides a more accurate measure of test depth by counting individual steps as scenarios.
42+
* **Granular Step Display**: test steps are extracted and displayed as `↳ Step Name` in the documentation tables.
43+
* **Domain Categorization**: Tests are automatically grouped into domains (Discovery, Governance, etc.) based on file paths and explicit overrides in `generate.js`.
44+
* **Zero New Dependencies**: Uses existing project dependencies (`playwright`, `typescript`, `dotenv`).
45+
46+
## 🔄 Workflow Integration (Rock Solid Automation)
47+
48+
Documentation is automatically kept in sync through two layers of automation:
49+
50+
### 1. Pre-commit Hook (Local)
51+
Defined in `.lintstagedrc.yaml`.
52+
* **Trigger**: Any change to `playwright/e2e/**/*.spec.ts`.
53+
* **Action**: Automatically runs `node playwright/doc-generator/generate.js`.
54+
* **Result**: The updated `docs/` are generated and `git add`ed to your commit automatically. You don't need to do anything manually.
55+
56+
### 2. GitHub Actions (CI Safety Net)
57+
Defined in `.github/workflows/playwright-docs-check.yml`.
58+
* **Trigger**: Pull Requests to `main`.
59+
* **Action**: Checks if the documentation matches the code.
60+
* **Auto-Fix**: If docs are outdated (e.g., if the pre-commit hook was bypassed), the CI job will:
61+
1. Fail the initial check.
62+
2. Regenerate the documentation.
63+
3. **Automatically commit and push** the fixes to your PR branch.
64+
Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
/*
2+
* Copyright 2025 Collate.
3+
* Licensed under the Apache License, Version 2.0 (the "License");
4+
* you may not use this file except in compliance with the License.
5+
* You may obtain a copy of the License at
6+
* http://www.apache.org/licenses/LICENSE-2.0
7+
* Unless required by applicable law or agreed to in writing, software
8+
* distributed under the License is distributed on an "AS IS" BASIS,
9+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
10+
* See the License for the specific language governing permissions and
11+
* limitations under the License.
12+
*/
13+
14+
const fs = require('fs');
15+
const path = require('path');
16+
const { execSync } = require('child_process');
17+
const { generateDomainMarkdown, generateIndexMarkdown } = require('./markdown.js');
18+
const { loadTestsFromPlaywright } = require('./playwright-loader.js');
19+
20+
// Constants
21+
// Script is in: openmetadata-ui/src/main/resources/ui/playwright/doc-generator/
22+
const PLAYWRIGHT_DIR = path.resolve(__dirname, '../e2e');
23+
const OUTPUT_DIR = path.resolve(__dirname, '../docs');
24+
25+
const DOMAIN_MAPPING = {
26+
// Overrides (Must be at the top to take precedence over generic keys like 'Pipeline')
27+
'TestSuitePipeline': { domain: 'Observability', name: 'Data Quality' },
28+
'TestSuiteMultiPipeline': { domain: 'Observability', name: 'Data Quality' },
29+
'ObservabilityAlerts': { domain: 'Observability', name: 'Alerts & Notifications' },
30+
'NotificationAlerts': { domain: 'Observability', name: 'Alerts & Notifications' },
31+
'DataQualityAndProfiler': { domain: 'Observability', name: 'Profiler' },
32+
'ProfilerConfigurationPage': { domain: 'Observability', name: 'Profiler' },
33+
'TestCases': { domain: 'Observability', name: 'Data Quality' },
34+
'TestCase': { domain: 'Observability', name: 'Data Quality' }, // Matches TestCaseVersionPage, AddTestCaseNewFlow
35+
'IncidentManager': { domain: 'Observability', name: 'Incident Manager' },
36+
37+
// Governance
38+
'Automator': { domain: 'Governance', name: 'Automator' },
39+
'Glossary': { domain: 'Governance', name: 'Glossary' },
40+
'Tag': { domain: 'Governance', name: 'Tags' },
41+
'Classification': { domain: 'Governance', name: 'Tags' }, // Alias
42+
'Workflow': { domain: 'Governance', name: 'Workflows' },
43+
'Metric': { domain: 'Governance', name: 'Metrics' },
44+
'KnowledgeCenter': { domain: 'Governance', name: 'Knowledge Center' },
45+
'CustomProperty': { domain: 'Governance', name: 'Custom Properties' },
46+
'Customproperties': { domain: 'Governance', name: 'Custom Properties' }, // Fix for casing
47+
'Domain': { domain: 'Governance', name: 'Domains & Data Products' },
48+
'DataProduct': { domain: 'Governance', name: 'Domains & Data Products' }, // Alias
49+
'DataContract': { domain: 'Governance', name: 'Data Contracts' },
50+
51+
// Platform
52+
'RBAC': { domain: 'Platform', name: 'RBAC' },
53+
'Role': { domain: 'Platform', name: 'RBAC' },
54+
'Policy': { domain: 'Platform', name: 'RBAC' },
55+
'Policies': { domain: 'Platform', name: 'RBAC' },
56+
'SSO': { domain: 'Platform', name: 'SSO' },
57+
'User': { domain: 'Platform', name: 'Users & Teams' },
58+
'Team': { domain: 'Platform', name: 'Users & Teams' },
59+
'Persona': { domain: 'Platform', name: 'Personas & Customizations' },
60+
'Customization': { domain: 'Platform', name: 'Personas & Customizations' },
61+
'Customize': { domain: 'Platform', name: 'Personas & Customizations' },
62+
'Theme': { domain: 'Platform', name: 'Personas & Customizations' },
63+
'AppMarketplace': { domain: 'Platform', name: 'App Marketplace' },
64+
'Application': { domain: 'Platform', name: 'App Marketplace' },
65+
'Settings': { domain: 'Platform', name: 'Settings' },
66+
'Cron': { domain: 'Platform', name: 'Settings' },
67+
'Lineage': { domain: 'Platform', name: 'Lineage (UI)' }, // Default UI Lineage
68+
'Impact': { domain: 'Platform', name: 'Lineage (UI)' },
69+
'Entity': { domain: 'Platform', name: 'Entities' },
70+
'Bulk': { domain: 'Platform', name: 'Entities' },
71+
'Navbar': { domain: 'Platform', name: 'Navigation' },
72+
'Navigation': { domain: 'Platform', name: 'Navigation' },
73+
'PageSize': { domain: 'Platform', name: 'Navigation' },
74+
'Pagination': { domain: 'Platform', name: 'Navigation' },
75+
'Login': { domain: 'Platform', name: 'Authentication' },
76+
'Auth': { domain: 'Platform', name: 'Authentication' },
77+
'Tour': { domain: 'Platform', name: 'Onboarding' },
78+
79+
// Discovery
80+
'Search': { domain: 'Discovery', name: 'Search' },
81+
'DataInsight': { domain: 'Discovery', name: 'Data Insights' },
82+
'Feed': { domain: 'Discovery', name: 'Feed' },
83+
'Conversation': { domain: 'Discovery', name: 'Feed' },
84+
'Chat': { domain: 'Discovery', name: 'Feed' },
85+
'DataAsset': { domain: 'Discovery', name: 'Data Assets' }, // Generic bucket
86+
'Table': { domain: 'Discovery', name: 'Data Assets' },
87+
'Topic': { domain: 'Discovery', name: 'Data Assets' },
88+
'Dashboard': { domain: 'Discovery', name: 'Data Assets' },
89+
'Pipeline': { domain: 'Discovery', name: 'Data Assets' },
90+
'Container': { domain: 'Discovery', name: 'Data Assets' },
91+
'Database': { domain: 'Discovery', name: 'Data Assets' },
92+
'Schema': { domain: 'Discovery', name: 'Data Assets' },
93+
'Explore': { domain: 'Discovery', name: 'Explore' },
94+
'MyData': { domain: 'Discovery', name: 'My Data' },
95+
'Curated': { domain: 'Discovery', name: 'Curated Assets' },
96+
'Home': { domain: 'Discovery', name: 'Home Page' },
97+
'Landing': { domain: 'Discovery', name: 'Home Page' },
98+
'RecentlyViewed': { domain: 'Discovery', name: 'Home Page' },
99+
'Following': { domain: 'Discovery', name: 'Home Page' },
100+
101+
// Observability
102+
'Quality': { domain: 'Observability', name: 'Data Quality' },
103+
'Dim': { domain: 'Observability', name: 'Data Quality' }, // Dimensionality
104+
'TestSuite': { domain: 'Observability', name: 'Data Quality' },
105+
'TestCase': { domain: 'Observability', name: 'Data Quality' },
106+
'Profiler': { domain: 'Observability', name: 'Profiler' },
107+
'RCA': { domain: 'Observability', name: 'Root Cause Analysis' },
108+
'Incident': { domain: 'Observability', name: 'Incident Manager' },
109+
'Alert': { domain: 'Observability', name: 'Alerts & Notifications' },
110+
'Notification': { domain: 'Observability', name: 'Alerts & Notifications' },
111+
112+
// Integration
113+
'Connector': { domain: 'Integration', name: 'Connectors' },
114+
'Service': { domain: 'Integration', name: 'Connectors' },
115+
'Ingestion': { domain: 'Integration', name: 'Connectors' },
116+
'Query': { domain: 'Integration', name: 'Connectors' }, // QueryEntity
117+
};
118+
119+
function getComponentInfo(fileName) {
120+
for (const [key, def] of Object.entries(DOMAIN_MAPPING)) {
121+
if (fileName.includes(key)) return def;
122+
}
123+
return { domain: 'Platform', name: 'Other' };
124+
}
125+
126+
function main() {
127+
console.log(`🚀 Starting Documentation Generation (Node.js)`);
128+
console.log(` Input: ${PLAYWRIGHT_DIR}`);
129+
console.log(` Output: ${OUTPUT_DIR}`);
130+
131+
if (!fs.existsSync(PLAYWRIGHT_DIR)) {
132+
console.error(`❌ Playwright directory not found!`);
133+
process.exit(1);
134+
}
135+
136+
// 1. Find and Parse Files using Native Playwright Loader
137+
console.log(`📝 asking Playwright to list tests...`);
138+
const parsedFiles = loadTestsFromPlaywright(PLAYWRIGHT_DIR);
139+
console.log(` Received ${parsedFiles.length} file suites from Playwright.`);
140+
141+
// 2. Group by Domain + Component
142+
const groupings = new Map();
143+
144+
parsedFiles.forEach(file => {
145+
const { domain, name } = getComponentInfo(file.fileName);
146+
const key = `${domain}:${name}`;
147+
148+
if (!groupings.has(key)) {
149+
groupings.set(key, { domain, name, files: [] });
150+
}
151+
groupings.get(key).files.push(file);
152+
});
153+
154+
// 3. Convert to Component Objects
155+
const components = Array.from(groupings.values()).map(g => ({
156+
name: g.name,
157+
domain: g.domain,
158+
slug: g.name.toLowerCase().replace(/[^a-z0-9]+/g, '-'),
159+
files: g.files,
160+
totalTests: g.files.reduce((s, f) => s + f.totalTests, 0),
161+
totalSteps: g.files.reduce((s, f) => s + f.totalSteps, 0),
162+
totalScenarios: g.files.reduce((s, f) => s + f.totalScenarios, 0),
163+
}));
164+
165+
// 4. Generate Content
166+
console.log(`⚙️ Generating Markdown for ${components.length} components...`);
167+
168+
// Clean output directory
169+
if (fs.existsSync(OUTPUT_DIR)) {
170+
fs.rmSync(OUTPUT_DIR, { recursive: true, force: true });
171+
}
172+
fs.mkdirSync(OUTPUT_DIR, { recursive: true });
173+
174+
// Stats
175+
const stats = {
176+
components: components.length,
177+
files: parsedFiles.length,
178+
tests: components.reduce((s,c) => s + c.totalTests, 0),
179+
steps: components.reduce((s,c) => s + c.totalSteps, 0)
180+
};
181+
182+
// Group Components by Domain for Generation
183+
const componentsByDomain = {};
184+
components.forEach(c => {
185+
if (!componentsByDomain[c.domain]) componentsByDomain[c.domain] = [];
186+
componentsByDomain[c.domain].push(c);
187+
});
188+
189+
// Generate Consolidated Domain Pages
190+
Object.entries(componentsByDomain).forEach(([domain, comps]) => {
191+
const content = generateDomainMarkdown(domain, comps);
192+
fs.writeFileSync(path.join(OUTPUT_DIR, `${domain}.md`), content);
193+
console.log(` ✓ ${domain}.md`);
194+
});
195+
196+
// Generate Main Index (README.md)
197+
fs.writeFileSync(path.join(OUTPUT_DIR, 'README.md'), generateIndexMarkdown(components, stats));
198+
console.log(` ✓ README.md`);
199+
200+
// 5. Stage files in Git
201+
try {
202+
console.log(`\n📦 Staging generated docs...`);
203+
execSync('git add .', { cwd: OUTPUT_DIR, stdio: 'inherit' });
204+
console.log(` ✓ git add completed for ${OUTPUT_DIR}`);
205+
} catch (error) {
206+
console.error(` ⚠️ Warning: Failed to stage files with git.`);
207+
console.error(error.message);
208+
}
209+
210+
console.log(`\n✅ Success! Documentation generated in:`);
211+
console.log(` ${OUTPUT_DIR}`);
212+
}
213+
214+
main();

0 commit comments

Comments
 (0)