Skip to content

About

QA automation framework for the ShortBeyond API using Playwright, Artillery load/spike testing, PostgreSQL CTEs, Gherkin BDD, OpenAPI, Postman, and GitHub Actions.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

ShortBeyond QA Automation & Performance Framework

ShortBeyond API & QA Suite Playwright Artillery PostgreSQL OpenAPI Postman

End-to-end quality assurance suite for the ShortBeyond URL shortener API (beyondtest/shortb-api:hotfix backed by PostgreSQL 15). Covers functional API contract testing with Playwright, BDD feature specifications in Gherkin, API documentation in OpenAPI/Swagger and Postman/Bruno, transactional PostgreSQL CTE data management, Artillery load/spike testing, and automated execution in GitHub Actions.


Table of Contents

  1. QA Strategy & Artifacts
  2. Repository Structure
  3. Architecture & Design Decisions
  4. API Contracts (Swagger, Postman & Bruno)
  5. Running Tests Locally
  6. Observed Results & Benchmarks
  7. CI/CD Pipeline (GitHub Actions)
  8. Author

1. QA Strategy & Artifacts

Rather than treating test automation in isolation, this repository documents the full testing lifecycle from planning to defect analysis:

Artifact Purpose Location
Test Plan Scope, risk matrix, traceability, and exit criteria docs/qa/TEST_PLAN.md
Test Case Inventory 24 mapped scenarios (20 functional + 4 performance) docs/qa/TEST_CASES.md
BDD Specifications Gherkin acceptance criteria (health, auth, links) specs/features/
Bug Report Example Root-cause investigation of 100 req/s spike failures docs/qa/bug-reports/BUG-001
Architecture Guide Service objects, CTE SQL queries, and SLA definitions docs/ARCHITECTURE_GUIDE.md
Performance Report Latency percentile analysis and Artillery runbooks performance/README.md

2. Repository Structure

.
├── .github/workflows/
│   └── ci.yml                          # CI pipeline (PostgreSQL 15 + API container + Playwright)
├── docs/
│   ├── auth/                           # Auth endpoint specs (Bruno / OpenCollection)
│   ├── health/                         # Health endpoint spec (Bruno / OpenCollection)
│   ├── links/                          # Links CRUD specs (Bruno / OpenCollection)
│   ├── postman/
│   │   ├── ShortBeyond.postman_collection.json
│   │   └── ShortBeyond.local.postman_environment.json
│   ├── qa/
│   │   ├── bug-reports/
│   │   │   └── BUG-001-spike-500-and-auth-degradation.md
│   │   ├── TEST_CASES.md               # Test case matrix with IDs and contract expectations
│   │   └── TEST_PLAN.md                # Test strategy, risk matrix, and traceability
│   ├── swagger/
│   │   └── openapi.yaml                # OpenAPI 3.0.3 API specification
│   ├── ARCHITECTURE_GUIDE.md           # Technical patterns and database guide
│   └── opencollection.yml              # Bruno collection entrypoint
├── performance/
│   ├── data/usuarios.csv               # Seeded user dataset for load tests
│   ├── reports/                        # Raw Artillery JSON reports (login, register, spike)
│   ├── tests/                          # Artillery workload definitions (YAML)
│   ├── insert-users.js                 # Bulk DB seeding script entrypoint
│   └── README.md                       # Load and spike test documentation
├── playwright/
│   ├── e2e/                            # API contract specs (auth, health, links)
│   └── support/
│       ├── factories/                  # Faker + ULID data generators
│       ├── fixtures/                   # Playwright custom fixtures (auth, links)
│       └── services/                   # HTTP service clients & PostgreSQL CTE controller
├── specs/features/                     # BDD scenarios written in Gherkin (.feature)
├── global-setup.js                     # Pre-suite DB sanitization via SQL CTE
├── playwright.config.js                # Runner config (API project, retries, HTML reporter)
├── shortbeyond.yaml                    # Pod specification (postgres:15, adminer, shortb-api)
└── package.json

3. Architecture & Design Decisions

  • API-First Playwright Runner (APIRequestContext): playwright.config.js configures only the api-tests project without browser binaries, reducing execution time and eliminating UI flakiness while keeping Playwright's fixtures, assertions, and HTML reporting.
  • Service Objects & Dependency Injection: HTTP interactions live in playwright/support/services/ (auth.js, links.js) and are injected into tests through custom fixtures (playwright/support/fixtures/index.js).
  • Collision-Free Factories: user.js and utils.js combine @faker-js/faker with 26-character Crockford ULIDs and tag generated test emails with @lfernandobra.dev for safe bulk cleanup.
  • Transactional PostgreSQL Cleanup & Seeding (database.js):
    • global-setup.js invokes cleanupTestData() before every test run using a SQL Common Table Expression (CTE) to remove test links (including orphaned rows) and users atomically.
    • insertTestUsers() computes a single Bcrypt hash (cost 10) upfront and inserts 500-row batches directly into PostgreSQL, seeding 2,000–4,000 test accounts in under 1 second while generating performance/data/usuarios.csv.

4. API Contracts (Swagger, Postman & Bruno)

The API exposes /health, /api/auth/register, /api/auth/login, and /api/links (GET/POST/DELETE):


5. Running Tests Locally

Prerequisites

  • Node.js >= 18.x
  • Running PostgreSQL 15 (ShortDB) and beyondtest/shortb-api:hotfix containers (see shortbeyond.yaml)

Setup & Functional Suite

# 1. Install dependencies
npm ci

# 2. Configure local environment variables
cp .env.example .env

# 3. Run Playwright API test suite
npm test

# 4. Open Playwright HTML report
npm run test:report

Database Seeding & Performance Suite

# Reset test records and seed 2,000 users into PostgreSQL + CSV
node playwright/support/services/database.js

# Run authentication workload test (4 req/s for 60s)
npm run test:perf

# Run 5-phase traffic spike test (up to 100 req/s)
npm run test:spike

6. Observed Results & Benchmarks

Functional Coverage (Playwright)

  • 20 automated API test cases across health.spec.js, auth/register.spec.js, auth/login.spec.js, links/post.spec.js, links/get.spec.js, and links/delete.spec.js.
  • Validates HTTP status codes, JSON payload schemas, password non-exposure, 5-char short_code regex (^([a-zA-Z0-9_-]){5}$), and Portuguese API validation messages.

Performance Telemetry (Artillery)

Metrics captured directly in performance/reports/:

Test Suite Workload Profile Error Rate p95 Latency Outcome
health.yaml 5 req/s for 30s 0% 13.9 ms Passed (target < 50 ms)
register.yaml 3 req/s for 45s (80% 201 / 20% 400) 0% unexpected 100.5 ms Passed (target < 500 ms)
login.yaml 4 req/s for 60s (70% 200 / 30% 401) 0% unexpected 94.6 ms Passed (target < 400 ms)
spike.yaml 5 phases up to 100 req/s (4,000 VUs) 162 HTTP 500s, 3 timeouts 6,312.2 ms (peak) Degraded under peak; recovered to 89.1 ms

Detailed bucket-by-bucket spike analysis and root-cause hypotheses are documented in BUG-001.


7. CI/CD Pipeline (GitHub Actions)

Workflow ci.yml runs on every push and pull_request to main:

  1. Spins up postgres:15 and beyondtest/shortb-api:hotfix containers on ubuntu-latest.
  2. Waits for pg_isready and GET /health readiness checks.
  3. Validates Postman JSON integrity and executes npm test (with retries: 2, workers: 1, forbidOnly: true).
  4. Dumps container logs on failure and archives playwright-report/ for 14 days.

8. Author

Luis Fernando Brandão
QA Automation Engineer

About

QA automation framework for the ShortBeyond API using Playwright, Artillery load/spike testing, PostgreSQL CTEs, Gherkin BDD, OpenAPI, Postman, and GitHub Actions.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages