-
Notifications
You must be signed in to change notification settings - Fork 1
test(e2e): add a Playwright harness for the Ask Agent capabilities #421
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
38e0e6b
test(e2e): add a Playwright harness for the Ask Agent capabilities
seonghobae 11a60b3
fix(frontend): repair the inherited login/admin-panel build break
seonghobae 8b256b8
Merge remote-tracking branch 'origin/worktree-fix-frontend-build-brea…
seonghobae 33ec5cd
test(frontend): keep Playwright specs out of Vitest
seonghobae 8d914dd
fix(e2e): stop Ask Agent lineage/image tests silently skipping their …
seonghobae 2fc0883
fix(e2e): stabilize Ask Agent locale and multi-match checks
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| import { expect, test } from "@playwright/test"; | ||
| import { loginAsDemoAnalyst } from "./support/auth.ts"; | ||
|
|
||
| /** | ||
| * Exercises the four Ask Agent capabilities end to end: relative-time-scoped | ||
| * retrieval (#415), git-branch-style multi-lineage rendering (#418), image | ||
| * citation (#419), and the evidence Layer Popup (#420). | ||
| * | ||
| * Requires all four PRs merged to `main` and the backend/frontend images | ||
| * rebuilt from it -- an ad-hoc `docker compose` stack still running an | ||
| * older or unrelated branch will not satisfy these selectors (verified: the | ||
| * stack running during this checkpoint's development was built from a | ||
| * different, more advanced branch with its own conversation-history UI, not | ||
| * `main`). `smoke.spec.ts`'s login flow is the one assertion here proven to | ||
| * pass against arbitrary deployments, since the Keycloak-hosted login form | ||
| * is shared across every branch. | ||
| */ | ||
| test.beforeEach(async ({ page }) => { | ||
| await loginAsDemoAnalyst(page); | ||
| await page.locator(".language-switcher select").selectOption("en"); | ||
| await page.getByRole("button", { name: "Ask Agent" }).click(); | ||
| }); | ||
|
|
||
| test("answers a relative-time-scoped question and cites at least one post", async ({ page }) => { | ||
| await page.getByRole("textbox", { name: "Ask a question" }).fill("어제 무슨 일이 있었나요?"); | ||
| await page.getByRole("button", { name: "Ask", exact: true }).click(); | ||
| await expect(page.getByRole("heading", { name: "Answer" })).toBeVisible(); | ||
| await expect(page.getByRole("heading", { name: "Cited posts" })).toBeVisible({ timeout: 15000 }); | ||
| }); | ||
|
|
||
| test("renders a cited lineage thread as a git-branch-style graph", async ({ page }) => { | ||
| await page.getByRole("textbox", { name: "Ask a question" }).fill("What happened between these events?"); | ||
| await page.getByRole("button", { name: "Ask", exact: true }).click(); | ||
| await expect(page.getByRole("heading", { name: "Cited posts" })).toBeVisible({ timeout: 15000 }); | ||
| const lineage = page.getByLabel("Reconstructed lineage"); | ||
| // Fail loudly (not silently skip) if the answer stops citing a | ||
| // multi-post lineage -- the whole point of this test. | ||
| await expect(lineage).not.toHaveCount(0); | ||
| await expect(lineage.first()).toBeVisible(); | ||
| await expect(page.getByRole("img", { name: /lineage$/ }).first()).toBeVisible(); | ||
| }); | ||
|
|
||
| test("cites persisted image evidence when a cited post has an embedded image", async ({ page }) => { | ||
| await page.getByRole("textbox", { name: "Ask a question" }).fill("Which project?"); | ||
| await page.getByRole("button", { name: "Ask", exact: true }).click(); | ||
| await expect(page.getByRole("heading", { name: "Cited posts" })).toBeVisible({ timeout: 15000 }); | ||
| const imageEvidence = page.getByText(/^Image evidence:/); | ||
| // Fail loudly (not silently skip) if the answer stops citing image | ||
| // evidence -- the whole point of this test. | ||
| await expect(imageEvidence).not.toHaveCount(0); | ||
| await expect(imageEvidence.first()).toBeVisible(); | ||
|
seonghobae marked this conversation as resolved.
|
||
| }); | ||
|
|
||
| test("opens cited-post evidence in a Layer Popup without leaving the answer", async ({ page }) => { | ||
| await page.getByRole("textbox", { name: "Ask a question" }).fill("Which project?"); | ||
| await page.getByRole("button", { name: "Ask", exact: true }).click(); | ||
| await expect(page.getByRole("heading", { name: "Cited posts" })).toBeVisible({ timeout: 15000 }); | ||
|
|
||
| const viewEvidence = page.getByRole("button", { name: "View evidence" }).first(); | ||
| await viewEvidence.click(); | ||
|
|
||
| const dialog = page.getByRole("dialog"); | ||
| await expect(dialog).toBeVisible(); | ||
| await page.getByRole("button", { name: "Close evidence panel" }).click(); | ||
| await expect(dialog).not.toBeVisible(); | ||
| // The answer is still on screen -- the layer never navigated away. | ||
| await expect(page.getByRole("heading", { name: "Cited posts" })).toBeVisible(); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| import { expect, test } from "@playwright/test"; | ||
| import { loginAsDemoAnalyst } from "./support/auth.ts"; | ||
|
|
||
| test("logs in and reaches an authenticated destination", async ({ page }) => { | ||
| await loginAsDemoAnalyst(page); | ||
| await expect(page.getByRole("button", { name: "Ask Agent" })).toBeVisible(); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| import type { Page } from "@playwright/test"; | ||
|
|
||
| /** | ||
| * Synthetic demo credentials seeded by `make seed` -- never a real account. | ||
| * See `backend/tests/test_api.py`'s `_fetch_demo_analyst_token` for the | ||
| * same login this drives through the real Keycloak realm. | ||
| */ | ||
| const DEMO_USERNAME = "demo.analyst"; | ||
| const DEMO_PASSWORD = "lineageweave-demo-only"; | ||
|
|
||
| /** | ||
| * Logs in through the real Keycloak-hosted login form (OIDC redirect, | ||
| * not a token injected into storage) so the e2e suite exercises the same | ||
| * authorization-code flow a reader actually goes through. | ||
| * | ||
| * Next action: call this once per test before interacting with any | ||
| * authenticated destination. | ||
| */ | ||
| export async function loginAsDemoAnalyst(page: Page): Promise<void> { | ||
| await page.goto("/"); | ||
| await page.getByRole("button", { name: "Log in" }).click(); | ||
| await page.waitForURL(/\/realms\/lineageweave-demo\/protocol\/openid-connect\/auth/); | ||
| await page.getByLabel("Username or email").fill(DEMO_USERNAME); | ||
| await page.getByLabel("Password", { exact: true }).fill(DEMO_PASSWORD); | ||
| await page.getByRole("button", { name: "Sign In" }).click(); | ||
| await page.waitForURL((url) => !url.pathname.includes("/realms/")); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| import { defineConfig, devices } from "@playwright/test"; | ||
|
|
||
| /** | ||
| * Runs against the already-running docker-compose stack (`make up`), not a | ||
| * dev-server Playwright starts itself -- the app needs Postgres, Keycloak, | ||
| * Valkey, and the orchestrator alongside it, which `webServer` can't provide. | ||
| * Point `LINEAGEWEAVE_E2E_BASE_URL` at a different origin if the compose | ||
| * port mapping changes. | ||
| */ | ||
| export default defineConfig({ | ||
| testDir: "./e2e", | ||
| fullyParallel: false, | ||
| forbidOnly: !!process.env.CI, | ||
| retries: process.env.CI ? 1 : 0, | ||
| workers: 1, | ||
| reporter: [["list"]], | ||
| use: { | ||
| baseURL: process.env.LINEAGEWEAVE_E2E_BASE_URL ?? "http://localhost:15173", | ||
| trace: "retain-on-failure", | ||
| }, | ||
| projects: [ | ||
| { | ||
| name: "chromium", | ||
| use: { ...devices["Desktop Chrome"] }, | ||
| }, | ||
| ], | ||
| }); |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "compilerOptions": { | ||
| "tsBuildInfoFile": "./node_modules/.tmp/tsconfig.e2e.tsbuildinfo", | ||
| "target": "es2023", | ||
| "lib": ["ES2023"], | ||
| "types": ["node"], | ||
| "skipLibCheck": true, | ||
|
|
||
| "module": "nodenext", | ||
| "allowImportingTsExtensions": true, | ||
| "verbatimModuleSyntax": true, | ||
| "moduleDetection": "force", | ||
| "noEmit": true, | ||
|
|
||
| "noUnusedLocals": true, | ||
| "noUnusedParameters": true, | ||
| "erasableSyntaxOnly": true, | ||
| "noFallthroughCasesInSwitch": true | ||
| }, | ||
| "include": ["e2e", "playwright.config.ts"] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,13 @@ | ||
| /// <reference types="vitest/config" /> | ||
| import { defineConfig } from 'vite' | ||
| import react from '@vitejs/plugin-react' | ||
| import { configDefaults } from 'vitest/config' | ||
|
|
||
| // https://vite.dev/config/ | ||
| export default defineConfig({ | ||
| plugins: [react()], | ||
| test: { | ||
| environment: 'jsdom', | ||
| setupFiles: ['./src/setupTests.ts'], | ||
| exclude: [...configDefaults.exclude, 'e2e/**'], | ||
| }, | ||
| }) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.