Skip to content
BranchesTags

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
node_modules
node_modules
 
 
public
public
 
 
src
src
 
 
.DS_Store
.DS_Store
 
 
Dockerfile
Dockerfile
 
 
README.md
README.md
 
 
package-lock.json
package-lock.json
 
 
package.json
package.json
 
 

Repository files navigation

BDD Coverage App

A local web application that generates BDD (Behaviour-Driven Development) coverage intelligence reports by cross-referencing three data sources:

  • Azure DevOps - fetches all test cases matching your query via WIQL
  • Jenkins REST API - auto-fetches the latest pipeline artifact files (Executed + Excluded) directly from Jenkins builds, no manual download needed
  • GHE REST API - scans .feature files directly from GitHub Enterprise repos via the Trees + Blobs API (no local clone required), or optionally from a local path

All through a browser UI, in under 60 seconds, with no scripting required.

Contents

The Problem It Solves

In large BDD projects - think 1,000+ automated test cases across dozens of feature files and two CI pipelines - answering basic coverage questions becomes surprisingly difficult:

Question Without this tool
Which TCs are actually running in the pipeline? Open each TC in ADO, search pipeline logs by name/ID - one at a time
Which TCs have a .feature file but never ran in any pipeline? Manually grep across hundreds of .feature files
Which TCs are completely uncovered? No reliable way without scripting
What is our BDD coverage % right now? Unknown unless someone compiles it manually
Who owns the most uncovered TCs? Manual ADO query + cross-reference

For a suite of 1,000 test cases, a manual coverage check takes 3–5 hours of tedious, error-prone work. Results go stale the moment the next pipeline run completes.

This tool answers all of those questions in under 60 seconds:

  • Fetches pipeline artifact data automatically from Jenkins - no downloading, no uploading
  • Runs the ADO WIQL query in bulk - no browsing test case by test case
  • Scans .feature files via GHE REST API - no local repo clone needed
  • Produces a full HTML report with coverage %, tables, platform filter, and breakdowns by module and assignee
  • Re-runnable in one click whenever you need a fresh snapshot

What the Report Tells You

Metric What it means
Total ADO Test Cases All TCs matching your query (e.g. a specific Area Path)
General Pipeline TCs whose TC ID was found in General pipeline execution logs
Upgrade Pipeline TCs whose TC ID was found in Upgrade pipeline execution logs
Pipeline Excluded TCs present in the pipeline excluded list (skipped intentionally)
Feature File Only TCs tagged in a .feature file - but not yet running in any pipeline
Not Covered TCs with no pipeline match and no feature file match
BDD Coverage % (Covered / Total) x 100

The report also includes:

  • Platform filter bar - filter all tables and counts by Android, Apple, Windows, Linux, or Chrome (searches TC title, ADO tags, and feature file path)
  • Coverage by Solution - which modules / BDD sub-folders have the most coverage
  • Not-covered by Assignee - who owns the most uncovered test cases
  • Full searchable, scrollable tables for every category

Time Savings

Task Manual With this tool
Get pipeline artifact files Navigate to Jenkins build, download artifact zip, extract, locate .txt files 1 click - Jenkins REST API fetches all sub-folders automatically
Check which TCs ran in pipeline Open each TC in ADO, search logs - one at a time Bulk cross-reference, all 1,000+ TCs in seconds
Fetch ADO test case details Browse ADO area by area, apply filters manually Single WIQL query fetches everything at once
Scan feature files for TC tags grep -r TC_ across hundreds of files, tally manually GHE Trees API - entire repo tree in 1 call, blobs fetched 50 at a time
Calculate coverage % Count covered TCs in a spreadsheet Auto-calculated - shown instantly in the report
Generate a shareable report Compile findings into a doc or spreadsheet by hand Self-contained timestamped HTML file, one click to share
Re-run for a different area Repeat the entire process from scratch Change the Area Path, click Generate

Typical time saved: 3–5 hours of manual lookup → under 60 seconds. Coverage % accuracy: manual estimates ±20% → tool-generated exact figure.

Screenshots

Screenshots coming soon

How It Works

ADO REST API          Jenkins REST API         GHE REST API (or local path)
(WIQL query ->        (auto-fetch latest        Trees API: fetch full repo
 fetch all TC          SUCCESS build ->          tree in 1 call ->
 details, titles,      download all              parallel blob fetches
 assignees,            Analysis.*/Executed.txt   (50 concurrent) ->
 priority, state)      + Excluded.txt)           parse @TC-XXXXXX tags
        |                     |                         |
        +---------------------+-------------------------+
                              |
                    TC ID-based cross-reference
                    Priority: General > Upgrade > Excluded > Feature-only
                              |
               +--------------+--------------+
         Coverage %               Timestamped HTML Report
      breakdowns by             (searchable tables, platform filter,
      module + assignee          self-contained, shareable)

Feature file scanning - GHE API vs Local Path

GHE Repos (default) Local Path
Requires repo clone on server No Yes
Works on any hosted VM Yes Only if repo is cloned there
Always scans latest code Yes (pinned to branch HEAD) Only if clone is up to date
Speed ~2–50s (Trees API + 50 parallel blobs) <1s (disk I/O)
Setup GHE PAT entered in UI (required) BDD_SCAN_ROOT in .env
Multiple repos Yes - add as many as needed One path only
If scan fails Warning shown in report, continues without feature matching Warning shown

GHE scan flow:

  1. GET /api/v3/repos/:owner/:repo/branches/:branch - resolve commit → tree SHA (1 call)
  2. If a sub-path is specified, walk the tree segments to resolve that sub-folder's SHA (avoids root tree truncation for large repos)
  3. GET /api/v3/repos/:owner/:repo/git/trees/:sha?recursive=1 - full file tree from that SHA (1 call)
  4. Filter to .feature blobs
  5. GET /api/v3/repos/:owner/:repo/git/blobs/:sha x N files - batched 50 at a time
  6. Parse each file for @TC-XXXXXX tags and TC_XXXXXX scenario names

Sub-path tip: For large repos, always set Sub-path to the BDD folder (e.g. Bdd) - this resolves the tree from that sub-folder directly rather than the entire repo root, avoiding the GHE recursive tree size limit and dramatically reducing scan time.

Test Connection button in the UI calls POST /api/ghe-test which resolves the branch and counts .feature files for each configured repo - validates token + repo access before running the full report.

GHE PAT is required - enter it directly in the Step 3 UI field (read:repo scope). If no token is provided or the scan fails (e.g. wrong branch, network error), a warning is shown in the report and generation continues with ADO + pipeline data only.

Jenkins auto-detection logic:

  • General build: latest SUCCESS build whose cause contains Push event to branch master
  • Upgrade build: latest SUCCESS build triggered by the upstream project matching JENKINS_UPGRADE_TRIGGER (configurable in .env)
  • All Analysis.* sub-folders are fetched automatically (e.g. Analysis.General-Main, Analysis.General-Security, Analysis.Upgrade-CurrentBuild, Analysis.Upgrade-v1.0, ...)

TC ID matching logic:

  • Pipeline artifact files contain one test method name per line, e.g. Bdd.General.Features.LoginFeature.TC_1017008_SomeScenario
  • The tool extracts all numeric IDs from TC_XXXXXX blocks - including combined ones like TC_1017008_1017050SomeScenario (both IDs extracted)
  • Feature files are scanned for @TC-XXXXXX tags and TC_XXXXXX_YYYYYY patterns in scenario names
  • Duplicate priority when a TC ID appears in multiple sources: General executed → Upgrade executed → Excluded → Feature file only

Prerequisites

  • Node.js 18+ installed
  • Azure DevOps access with a Personal Access Token (Work Items - Read)
  • Jenkins network access - JENKINS_URL reachable from this machine. If your Jenkins allows anonymous read, no credentials are needed.
  • GHE Personal Access Token with read:repo scope - entered in the Step 3 UI field when running a report (optional but required for feature file matching)

Setup

1. Clone and install

git clone <repo-url>
cd bdd-coverage-app
npm install

2. Configure your credentials

There are two ways to configure the app — pick whichever suits you:

Option A — Edit src/config/env.js directly (simplest, no extra files)

Open src/config/env.js and fill in the USER_CONFIG block at the top of the file:

const USER_CONFIG = {
  // Azure DevOps
  ADO_ORG:     'https://dev.azure.com/your-org',
  ADO_PROJECT: 'YourProject',
  ADO_PAT:     'your-pat-here',

  // Jenkins
  JENKINS_URL:             'https://your-jenkins.example.com',
  JENKINS_JOB_PATH:        'job/your-pipeline/job/master',
  JENKINS_UPGRADE_TRIGGER: 'Trigger-UpgradeTestExecution',

  // GHE (optional)
  GHE_HOST:  '',   // e.g. 'https://ghe.yourcompany.com'
  GHE_TOKEN: '',

  // Server
  PORT: 3001,      // ← change this if port 3001 is already in use
  HOST: '0.0.0.0',
};

This is the recommended approach for open-source / local use — one file, all settings, no extra config.

Option B — .env file (recommended for teams / shared servers)

# Windows
copy .env.example .env

# macOS / Linux
cp .env.example .env

Open .env and fill in your values:

ADO_ORG=https://dev.azure.com/your-org
ADO_PROJECT=YourProject
ADO_PAT=your-personal-access-token-here

JENKINS_URL=https://your-jenkins.example.com
JENKINS_JOB_PATH=job/your-pipeline/job/master

PORT=3001

.env is listed in .gitignore and is never committed. .env values always override USER_CONFIG.

3. Create ADO Personal Access Token

  1. Open your ADO organisation in a browser
  2. Click your avatar (top-right) → Personal access tokens
  3. Click New Token
  4. Name it (e.g. bdd-coverage-app), set expiry (90 days recommended)
  5. Under Scopes, choose Custom defined and enable Work Items → Read
  6. Click Create and copy the token immediately - it is shown only once
  7. Paste it into ADO_PAT (in USER_CONFIG or .env)

4. Start the app

npm start

Open http://localhost:3001 in your browser (or whatever port you configured). The server also prints your LAN IP on startup for access from other machines on the same network.

For auto-restart on code changes during development: npm run dev

Running on a Custom Port

The app defaults to port 3001. If that port is already in use (e.g. another tool is running), change it in one place:

Option A — USER_CONFIG in src/config/env.js:

PORT: 3002,   // ← pick any free port

Option B — .env file:

PORT=3002

Option C — inline when starting (no file changes needed):

# Windows PowerShell
$env:PORT=3002; npm start

# macOS / Linux
PORT=3002 npm start

Running two instances in parallel

If you need to run this app on a different port alongside another tool already on port 3000, just make sure each instance has a different PORT. Open a new terminal for each:

# Terminal 1 — existing tool stays untouched on port 3000
# (nothing to do — leave it running)

# Terminal 2 — this app on port 3001
cd bdd-coverage-app
npm start           # reads PORT=3001 from .env or USER_CONFIG

Both servers run independently. Changing the port in .env or USER_CONFIG only affects this app — the other tool is completely unaffected.

Using the App

Step 1 - ADO Query

Guided Builder (default): Build your query visually.

  • Area Path - filter to a specific ADO area (optional; leave blank to query all)
  • Title - filter by test case title with operator (Contains Words, Contains, etc.)
  • Automation Status - defaults to Automated API Test (BDD); check additional statuses as needed
  • State Filter - defaults to Exclude Closed; switch to No state filter to include all states
  • Advanced Filters - expand for priority, assigned to, tags, iteration path, and order by

Raw WIQL: paste any WIQL query directly. Useful for multi-area-path queries or complex conditions.

Step 2 - Pipeline Data (optional)

Auto-fetch (recommended): Click Fetch Latest General Build, Fetch Latest Upgrade Build, or Fetch Both Pipelines. The server queries Jenkins, finds the latest successful build, and downloads all Analysis.* artifact files automatically.

Manual upload: Expand Upload files manually and drop .txt files into the appropriate zone:

  • Left zone (General) - General pipeline Executed + Excluded .txt files
  • Right zone (Upgrade) - Upgrade pipeline Executed + Excluded .txt files

Step 3 - Feature Files (optional)

GHE Repos (default): Click + Add Repository to add a repo entry.

  • The first click adds a blank card for you to fill in with your repo URL, branch, and sub-path
  • Each subsequent click adds a blank card for you to fill in
  • Enter the full GHE repo URL, branch, and optional sub-path (strongly recommended for large repos)
  • Enter your GHE PAT (read:repo scope) in the token field - required for scanning
  • Click Test Connection to verify access and count .feature files per repo before generating
  • Multiple repos can be added - all are scanned and merged into one TC ID lookup
  • Duplicate repo URLs are highlighted with a validation error

If no token is provided or the GHE scan fails, a warning is shown in the report and generation continues with ADO + pipeline data only - the feature file step is fully non-blocking.

Local Path: Enter the server-side absolute path to your .feature files folder. Only works if this app is running on a machine with the BDD repo cloned locally.

Leave both blank to report on ADO + pipeline data only.

Step 4 - Generate

Click Generate Report. The app:

  1. Runs the WIQL query against ADO and fetches all matching test case details
  2. Parses pipeline files and indexes every TC ID found
  3. (If configured) Scans GHE repos or local path for .feature files and extracts TC ID tags
  4. Cross-references all sources and generates a timestamped HTML report
  5. Opens the report automatically in a new browser tab

Report Features

Platform Filter Bar

The generated report includes a Platform Filter bar at the top with buttons: All Platforms | Android | Apple | Windows | Linux | Chrome

Clicking a platform button filters all three tables (Covered, Not Covered, Feature File Only) to show only rows where the test case title, ADO tags, or feature file path contains that platform name. All summary cards (Total, Covered, General, Upgrade, Excluded, Feature File Only, Not Covered, Coverage %) and section counts update dynamically to reflect the filtered view. Clicking All Platforms resets everything.

Search Boxes

Each table section has a live search box - type to filter rows by any column value. Search and platform filter work together.

Download

A Download Report button in the report header saves the self-contained HTML file for sharing. No server is needed to view a saved report.

Project Structure

bdd-coverage-app/
|-- src/
|   |-- server.js               # Thin Express entry point - registers middleware, routes, starts server
|   |-- config/
|   |   |-- env.js              # Centralised environment config (single source of truth for all .env vars)
|   |   `-- ghe.js              # GHE repo URL validation + host allow-list enforcement
|   |-- middleware/
|   |   |-- auth.js             # Entra ID (OIDC) Bearer token validation middleware
|   |   `-- upload.js           # Multer config - memory storage, 50 MB limit
|   |-- routes/
|   |   |-- auth.js             # GET /api/auth-config, POST /api/preview-wiql
|   |   |-- report.js           # POST /api/generate-report (orchestrates ADO + pipeline + GHE + report)
|   |   |-- jenkins.js          # POST /api/jenkins-fetch
|   |   `-- ghe.js              # POST /api/ghe-test
|   `-- services/
|       |-- adoClient.js        # ADO REST API client - WIQL query + batched TC detail fetch
|       |-- jenkinsClient.js    # Jenkins REST API client - auto-detect latest build, download artifacts
|       |-- pipelineParser.js   # Parses pipeline .txt files, extracts TC IDs with priority rules
|       |-- featureScanner.js   # GHE Trees+Blobs API scanner + local filesystem scanner
|       `-- reportGenerator.js  # Generates the self-contained HTML coverage report
|-- public/
|   `-- index.html              # Browser UI - query builder, Jenkins fetch, GHE repo manager
|-- .env                        # Your local credentials - never committed (listed in .gitignore)
|-- .env.example                # Template - copy to .env and fill in your values
|-- .gitignore                  # Excludes .env and node_modules from git
|-- .dockerignore               # Excludes .env, node_modules, .git from Docker build context
|-- Dockerfile                  # Docker image definition - node:20-alpine, exposes port 3000
`-- package.json                # Dependencies + start scripts (start: node src/server.js)

Module Responsibilities

Layer Files Responsibility
Entry src/server.js Wires middleware + routes, starts the HTTP server
Config src/config/env.js One place to read all env vars - no process.env scattered elsewhere
Config src/config/ghe.js Validates GHE repo JSON and enforces GHE_HOST allow-list
Middleware src/middleware/auth.js Validates Entra ID tokens (no-op when ENTRA_CLIENT_ID not set)
Middleware src/middleware/upload.js Multer multipart file upload config
Routes src/routes/report.js Orchestrates the full report pipeline (ADO -> parse -> GHE -> generate)
Routes src/routes/jenkins.js Jenkins build auto-fetch endpoint
Routes src/routes/ghe.js GHE connectivity test endpoint
Routes src/routes/auth.js Auth config + WIQL preview endpoints
Services src/services/adoClient.js ADO REST API - WIQL, batched work item fetch, 1000-item pagination
Services src/services/jenkinsClient.js Jenkins - detect latest build, download Analysis.* artifacts
Services src/services/pipelineParser.js Parse .txt lines, extract TC IDs, apply General > Upgrade priority
Services src/services/featureScanner.js GHE Trees+Blobs API + local filesystem, parse @TC-XXXXXX tags
Services src/services/reportGenerator.js Build self-contained HTML report with platform filter + tables

Troubleshooting

Port 3000 already in use

# Windows PowerShell
Stop-Process -Id (Get-NetTCPConnection -LocalPort 3000).OwningProcess -Force
npm start
# macOS / Linux
lsof -ti:3000 | xargs kill
npm start

401 Unauthorized errors from ADO Your PAT has expired or has insufficient permissions. Recreate it with Work Items → Read scope and update ADO_PAT in .env.

Jenkins fetch fails or returns "Jenkins not configured" Ensure JENKINS_URL and JENKINS_JOB_PATH are set in .env. If your Jenkins allows anonymous read, no credentials are needed. Restart the server after editing .env.

Jenkins fetch finds no builds The auto-detection scans the last 60 builds for General and 500 builds for Upgrade (Upgrade runs are infrequent). If no successful build is found, increase the limit parameter in src/services/jenkinsClient.js → findLatestSuccessBuild.

GHE scan fails with 404 "Not Found" on branch The branch name is wrong for that repo - check the default branch name in GHE (some repos use main instead of master). Update the Branch field in the repo card and retry.

GHE scan fails - report still generates This is by design. GHE scanning is fully optional and non-blocking. A warning banner is shown in the report. The report still contains ADO + pipeline coverage data.

GHE tree truncated warning in report The repo's root tree exceeds the GHE recursive tree size limit. Set a Sub-path (e.g. Bdd) in the repo card to scope the scan to just the BDD folder - this resolves the tree from that sub-folder's SHA directly and avoids truncation.

Unexpected token '<' error on Generate The server returned an HTML error page instead of JSON - check the server console for the full error message.

Upgrade Pipeline shows 0 in the report When uploading manually, make sure Upgrade files go into the right zone (Upgrade). Both pipelines use the same filename - the zone determines attribution. When using Jenkins auto-fetch, the pipeline is detected automatically.

0 TC IDs from pipeline after uploading Open one of the .txt files and confirm each line contains TC_ followed by digits (e.g. ...TC_1017008...). The file must follow the Namespace.FeatureClass.TC_XXXXXX_ScenarioName format.

Feature file scan returns 0 files (Local Path)

  • Verify the path exists and is accessible from this machine
  • Confirm the folder contains .feature files (not .feature.cs or other extensions)
  • Directories named node_modules, .git, bin, obj are automatically skipped
  • Ensure BDD_SCAN_ROOT is set in .env if using the local path security restriction

About

Node.js tool that automates test coverage analysis across Azure DevOps and Jenkins, reducing analysis time from 3–5 hours to ~60 seconds.

Topics

azuredevopsjenkinsqa-automationtestcoverage

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Contributors

Languages