notch-aqa

Playwright + pytest + Allure automation for the Notch AI support platform. Page Object Model, typed Python, multi-environment runs (local, staging, preprod, prod).

Resources

QA Role — questions & answers	rowan-kendal.vercel.app/qa-role
Automation Audit test plan	rowan-kendal.vercel.app/notch

Quick start

# 1. Python environment
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# 2. Playwright browsers
playwright install --with-deps

# 3. Local config (secrets + auth)
cp secured_env_files/.env.example secured_env_files/.env
# edit secured_env_files/.env — see secured_env_files/README.md

# 4. First run — headed, complete Google login once
make test-smoke HEADED=1 ENV=staging

# 5. Later runs reuse storage_state.json (headless OK)
make test-smoke ENV=staging

Layout

.
├── docs/                # Architecture + feature notes
├── downloads/           # Browser downloads (gitignored)
├── fixtures/            # pytest fixtures (browser, auth, audit)
├── pages/               # Page Object Model
│   ├── base/
│   └── notch/           # guardrails, playground, …
├── reports/             # Allure, traces, screenshots (gitignored)
├── resources/           # Test data (e.g. automation_audit_cases.jsonc)
├── secured_env_files/   # .env + storage_state.json (gitignored)
├── tests/               # Test suites only
├── utilities/           # env, drivers, auth, verifications, reports
└── workflows/           # Multi-page business flows

Make targets

Target	Description
make install	Editable install with dev deps
make browsers	playwright install --with-deps
make capture-auth	Headed Chrome login → storage_state.json
make test-smoke	-m smoke
make test-regression	-m regression
make test-audit	-m automation_audit
make allure-serve	Open Allure report from last run
make lint / make format	black + isort + flake8

Make variables: ENV (default staging), HEADED=1, SLOWMO=250, WORKERS=4.

pytest CLI

pytest \
  --env=staging \          # staging | preprod | prod | local
  --browser=chromium \       # chromium | firefox | webkit
  --headed \                 # visible browser (Chrome channel for SSO)
  --slowmo=250 \             # ms delay between actions
  -n=4                       # parallel workers (pytest-xdist)

Environments

Env	Default base URL
staging	https://guardio.app.getnotch.dev
preprod	https://guardio.app.preprod.getnotch.dev
prod	https://guardio.app.getnotch.com
local	http://localhost:3000

Override with NOTCH_BASE_URL in secured_env_files/.env (or optional repo-root .env).

Failure artifacts

On test failure:

• Screenshot → reports/screenshots/{CaseId}_{timestamp}.png (e.g. F6_20260529_160512_843.png)
• Trace zip → reports/traces/{CaseId}_{timestamp}.zip
• Allure attachments → screenshot, URL, DOM snapshot

Automation Audit

Guardrails decide whether the AI agent may reply. Four rule types:

1. Email patterns — sender address
2. Subjects — subject line keywords
3. Words in user message — customer body
4. Words in assistant reply — generated draft

Each case in resources/test_data/automation_audit_cases.jsonc creates a Guardrails draft, pins it in Playground, sends a customer message, and asserts blocked vs reply sent.

Test plan: rowan-kendal.vercel.app/notch · Implementation notes: docs/automation-audit.md