diff --git a/.cursor/rules/cloudinary.mdc b/.cursor/rules/cloudinary.mdc new file mode 100644 index 0000000..e78b368 --- /dev/null +++ b/.cursor/rules/cloudinary.mdc @@ -0,0 +1,8 @@ +--- +description: web-speed-test-server agent guidance +alwaysApply: true +--- + +This repo's agent guidance lives in [`AGENTS.md`](../../AGENTS.md) at the repo root. Read it before generating or editing code. + +Quick facts: deployable Node.js/Express service (not an npm package), Node 20, Yarn 1 (not npm), default branch `master`. Tests run with `WTP_API_KEY=mock yarn run test`. See `AGENTS.md` for setup, env vars (including the misspelled `CLOUDINARY_SEACRET`), the HTTP API, and gotchas. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..368c35c --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,5 @@ +# GitHub Copilot instructions — web-speed-test-server + +This repo's agent guidance lives in [`AGENTS.md`](../AGENTS.md) at the repo root. Read it before generating or editing code. + +Quick facts: deployable Node.js/Express service (not an npm package), Node 20, Yarn 1 (not npm), default branch `master`. Tests run with `WTP_API_KEY=mock yarn run test`. See `AGENTS.md` for setup, env vars (including the misspelled `CLOUDINARY_SEACRET`), the HTTP API, and gotchas. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..455657d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,73 @@ +# AGENTS.md — web-speed-test-server + +## What this repo is (one line) +A deployable Node.js/Express backend service — the server side of Cloudinary's Website Speed Test — that takes a page URL, measures it via the WebPageTest.org API, and uploads each loaded image to Cloudinary to report per-image byte savings. It's a service you run, not a package you install. + +## When to use this / when NOT to use this +- **Use this when:** you're deploying or modifying the API that powers the Website Speed Test (the `/test/*`, `/version`, `/locations` HTTP endpoints), or wiring it to WebPageTest and a Cloudinary product environment. +- **Do NOT use this when:** you want the browser UI (that's `cloudinary/web-speed-test-client`), or you're looking for a Cloudinary SDK to upload/transform assets in your own app (that's `cloudinary_npm` / `@cloudinary/url-gen`). This repo is `private: true` in `package.json` — it's not published to npm. +- **Architecture fact:** there's no local browser, Puppeteer, or Lighthouse here. The measurement engine is the external WebPageTest.org HTTP API; Cloudinary is used to analyze image savings. Both are called over HTTP. + +## Setup +```bash +git clone https://github.com/cloudinary/web-speed-test-server.git +cd web-speed-test-server +yarn install # Yarn 1 classic — postinstall runs patch-package +``` +Required environment variables (read by `config/default.js` via `dotenv`; put them in `.env` at the repo root): +```bash +WTP_API_KEY= # WebPageTest API key(s), comma-separated; picked at random per run +CLOUDINARY_NAME= +CLOUDINARY_API= +CLOUDINARY_SEACRET= # NOTE: misspelled in source — set it exactly as written +PORT=5000 # optional, default 5000 +``` + +## Run +```bash +yarn start # node --require ./instrumentation.js start.js — HTTP on PORT (default 5000) +``` +The `--require ./instrumentation.js` flag loads OpenTelemetry before the app; a Prometheus metrics endpoint runs on port `6060`. The HTTP server socket timeout is 3 minutes to accommodate slow WebPageTest runs. + +## Build / test commands (from CI — `.github/workflows/ci.yml`) +CI runs on Node `20.16`: +```bash +yarn install --immutable # matches CI verbatim (Yarn 1 classic ignores --immutable; use --frozen-lockfile to actually enforce the lockfile) +WTP_API_KEY=mock yarn run test # mocha suite; CI sets WTP_API_KEY=mock +``` +`yarn run test` maps to `mocha` (see `package.json` scripts). Tests use mocha + chai + chai-http + sinon + nock (HTTP mocking), so they don't hit real WebPageTest or Cloudinary. Run a single spec file with mocha directly: +```bash +WTP_API_KEY=mock npx mocha test/wptTests.js # test files: apiTests.js, cloudinaryTests.js, wptTests.js +``` +There is **no lint script** and no lint workflow — don't invoke `yarn lint`. + +## Conventions & gotchas +- **Package manager: Yarn 1 (classic).** Use `yarn`, not `npm`. The lockfile is `yarn.lock`; there's no `package-lock.json`. If `yarn` isn't on PATH: `corepack enable && corepack prepare yarn@1.22.22 --activate`. +- **`patch-package` postinstall.** The pinned `cloudinary` dependency is patched via `patches/cloudinary+2.9.0.patch`, applied on install. Bumping `cloudinary` means regenerating that patch. +- **`CLOUDINARY_SEACRET` is misspelled on purpose** in `config/default.js`. Set the env var with that spelling; don't "fix" it. +- **No `engines` field.** `package.json` declares no Node floor. CI pins `20.16`; the deps (`got` 14 ESM, OpenTelemetry, Express 4.22) need a modern Node (20). The upstream README's "Node v5 or newer" is wrong. +- **PR titles are semantic** (`.github/workflows/pr.yml`, `amannn/action-semantic-pull-request`). Allowed types: `fix`, `feat`, `chore`. +- **Releases are automated** via `release-please` (`.github/workflows/release.yml`, `release-type: node`) on push to `main`/`master`. Versioning follows Conventional Commits; a security-only fix is a PATCH bump. +- **Default branch is `master`.** Branch off `master` for changes. +- **OpenTelemetry packages move as a set.** Stable (`@opentelemetry/api`, `resources`, `sdk-trace-base`) are `1.x`/`2.x`; experimental (`exporter-prometheus`, `instrumentation-*`, `sdk-node`) are `0.x`. Bumping past `0.218.0` removes `new Resource({...})` — use `resourceFromAttributes({...})` in `instrumentation.js`. + +## Repo already ships its own Claude skill — respect it +`.claude/skills/fix-dependency-security/SKILL.md` is the repo's own workflow for resolving yarn dependency vulnerabilities. Don't contradict it: keep Yarn 1 (not npm), `master` as the default branch, `patch-package` postinstall, no Chrome/chromedriver install step, and PATCH bumps for security-only fixes. Follow that skill for any dependency-security task. + +## Key files +- `start.js` — entry point (`http.createServer(app)`, listens on `PORT`). +- `instrumentation.js` — OpenTelemetry setup, loaded before the app; Prometheus on `6060`. +- `app.js` — Express app: open CORS, JSON body parser, mounts `routes/wpt.js`, catch-all 404. +- `routes/wpt.js` — the HTTP routes (`/test/run`, `/test/:testId`, `/version`, `/locations`, `/locations/current`). +- `config/default.js` — all config, read from env vars. +- `wtp/` — WebPageTest client (`apiCaller.js`, `apiKey.js`, `locationSelector.js`). +- `cloudinary/apiCaller.js` — per-image Cloudinary upload + eager-transformation analysis. + +## Canonical docs +- Cloudinary documentation: https://cloudinary.com/documentation +- Documentation llms.txt index: https://cloudinary.com/documentation/llms.txt + +## Commit / PR conventions +- Branch off `master`; open PRs against `master`. +- PR titles follow Conventional Commits with type `fix`, `feat`, or `chore` (enforced in CI). +- Keep CI green: `yarn install --immutable` then `WTP_API_KEY=mock yarn run test` on Node 20. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f599929 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,36 @@ +@AGENTS.md + +# CLAUDE.md — web-speed-test-server + +## Claude Code-specific notes + +**Primary reference:** `AGENTS.md` (imported above) covers setup, run, build/test commands, conventions, and gotchas. Read it before touching any file. + +This repo already ships a `.claude/` directory (`.claude/skills/fix-dependency-security/`). This file is additive — it doesn't replace or conflict with that skill. For any dependency-security or CVE/JIRA task, use the repo's `fix-dependency-security` skill and follow it exactly. + +## What this repo is + +A deployable Node.js/Express service — the backend for Cloudinary's Website Speed Test. It takes a page URL, measures it through the external WebPageTest.org API, and uploads each loaded image to Cloudinary to report per-image byte savings. It's a service you run, not a package you install (`package.json` is `private: true`). + +## Key constraints + +- **Yarn 1 (classic), not npm.** Lockfile is `yarn.lock`; there's no `package-lock.json`. `postinstall` runs `patch-package`. +- **Node 20** (CI pins `20.16`). No `engines` field in `package.json`; the upstream README's "Node v5" is wrong. +- **`CLOUDINARY_SEACRET`** is the literal (misspelled) env var name in `config/default.js`. Don't correct it. +- **No lint script.** Don't run `yarn lint` — it doesn't exist. +- **Default branch: `master`.** PR titles are semantic (`fix`/`feat`/`chore`); releases are automated via `release-please`. +- **No local browser.** Measurement is the external WebPageTest.org API; there's no Puppeteer/Lighthouse/chromedriver here. + +## Verified run + test commands + +```bash +yarn install # Yarn 1; runs patch-package postinstall +yarn start # node --require ./instrumentation.js start.js — HTTP on PORT (5000) + +# Tests (CI: Node 20.16) +yarn install --immutable # exact CI install +WTP_API_KEY=mock yarn run test # mocha suite (CI sets WTP_API_KEY=mock) +WTP_API_KEY=mock npx mocha test/wptTests.js # single test file (also apiTests.js, cloudinaryTests.js) +``` + +`yarn run test` maps to `mocha`. Tests mock HTTP via `nock`, so they don't call real WebPageTest or Cloudinary. diff --git a/README.md b/README.md index 2a398b9..d804e02 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,133 @@ -# web-speed-test Image Performance Analysis Server App +# web-speed-test-server -This is the server app for Cloudinary image analysis tool. +[![license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/cloudinary/web-speed-test-server/blob/master/LICENSE) +[![CI](https://github.com/cloudinary/web-speed-test-server/actions/workflows/ci.yml/badge.svg)](https://github.com/cloudinary/web-speed-test-server/actions/workflows/ci.yml) +The backend service for Cloudinary's Website Speed Test image performance tool. It's a deployable Node.js/Express server (not an installable package), run on Node 20. It takes a page URL, drives a load-and-render measurement through the WebPageTest.org API, then uploads each image the page loaded to Cloudinary with a set of eager transformations to report the byte savings you'd get from `q_auto` and modern formats (WebP, AVIF, JP2, JPEG XR, PNG). -## Getting Started +## Setup -### Installation +Clone the repo, install with Yarn 1 (the lockfile is `yarn.lock`; there's no `package-lock.json`), and run on Node 20: -**Step 1**. Make sure that you have [Node.js](https://nodejs.org/) v5 or newer and -[Yarn](https://yarnpkg.com/) installed on your development machine. +```bash +git clone https://github.com/cloudinary/web-speed-test-server.git +cd web-speed-test-server +yarn install +``` + +`yarn install` runs a `patch-package` postinstall step that patches the pinned `cloudinary` dependency, so install with Yarn rather than npm. + +## Configuration + +The server reads its config from environment variables (via `dotenv` — put them in a `.env` file at the repo root, or export them). The Cloudinary and WebPageTest variables are read in `config/default.js`; `PORT` is read in `start.js`: + +| Variable | Purpose | +|---|---| +| `WTP_API_KEY` | WebPageTest.org API key. Accepts a comma-separated list; one key is picked at random per test run. Defaults to `dummy`, which won't run real tests. | +| `CLOUDINARY_NAME` | Cloudinary cloud name. | +| `CLOUDINARY_API` | Cloudinary API key. | +| `CLOUDINARY_SEACRET` | Cloudinary API secret. | +| `PORT` | HTTP listen port. Defaults to `5000`. | + +Note the variable name `CLOUDINARY_SEACRET` is misspelled in the source (`config/default.js` reads `process.env.CLOUDINARY_SEACRET`). Set it with that exact spelling — "correcting" it to `CLOUDINARY_SECRET` leaves the API secret undefined and breaks the Cloudinary calls. + +A Prometheus metrics endpoint (OpenTelemetry) is also served on port `6060`, separate from the HTTP API. + +Keep the API secret and WebPageTest key out of client-side code and out of version control. + +```bash +# .env at the repo root +WTP_API_KEY=, +CLOUDINARY_NAME= +CLOUDINARY_API= +CLOUDINARY_SEACRET= +PORT=5000 +``` + +Start the server: + +```bash +yarn start +``` + +That runs `node --require ./instrumentation.js start.js` — the instrumentation module loads first to set up OpenTelemetry, then the Express app listens on `PORT` (default `5000`). + +## API + +The server exposes a JSON HTTP API (CORS is open to all origins). The examples below assume it's running on `http://localhost:5000`. + +### Start a test + +`POST /test/run` starts a WebPageTest run for a URL. Body: `url` (required) and optional `mobile` (boolean, toggles mobile emulation). It returns a `testId` you poll for results: + +```bash +curl -X POST http://localhost:5000/test/run \ + -H 'Content-Type: application/json' \ + -d '{"url":"https://example.com","mobile":false}' +``` -**Step 2**. Clone this repository +```json +{ "status": "success", "data": { "testId": "250719_AbC_123" } } +``` + +A missing body or missing `url` returns HTTP 400. + +### Get test results + +`GET /test/:testId` polls WebPageTest for the run, and once it's done, uploads each image the page loaded to Cloudinary and returns the per-image savings analysis. The optional `quality` query parameter maps to Cloudinary's `q_auto:` for the analysis: + +```bash +curl 'http://localhost:5000/test/250719_AbC_123?quality=80' +``` + +While the run is still in progress it returns a not-finished marker — poll again: + +```json +{ "status": "success", "message": "test not finished", "code": 150 } +``` + +When the run is done, it returns `{ "status": "success", "data": { ... } }` with the analyzed image payload. -```shell -$ git clone git@github.com:CloudinaryLtd/web-speed-test-server.git web-speed-test-server -$ cd web-speed-test -$ yarn install # Install project dependencies listed in package.json +### Get the server version + +`GET /version` returns the running server version from `package.json`: + +```bash +curl http://localhost:5000/version ``` -**Step 3**. Set CLOUDINARY_URL +```json +{ "version": "1.3.12" } +``` -You will need to add your CLOUDINARY_URL to an `.env` file under `/server` +### List WebPageTest locations +`GET /locations` returns the cached list of WebPageTest agent locations used by the location selector. `GET /locations/current` returns the currently selected location and when it was last refreshed. Both are populated only when the location selector is enabled (`WTP_LS_ENABLED=true`); otherwise the list is empty and the timestamp is the epoch: -**Step 4**. Compile and launch your app by running: +```bash +curl http://localhost:5000/locations +curl http://localhost:5000/locations/current +``` -```shell -$ yarn start # Run the server +```json +{ "location": "IAD_US_01", "lastUpdated": "1970-01-01T00:00:00.000Z" } ``` + +## For AI agents + +This repo is the **server** half of Cloudinary's Website Speed Test — a deployable Express service that orchestrates WebPageTest.org and the Cloudinary SDK to measure per-image performance. It does not run a browser or Lighthouse locally; the measurement engine is the external WebPageTest.org API. Pick the right piece: + +| You want | Go to | +|---|---| +| The backend/API service (this repo) | `cloudinary/web-speed-test-server` | +| The browser UI that calls this API | [`cloudinary/web-speed-test-client`](https://github.com/cloudinary/web-speed-test-client) | +| The hosted tool, no setup | Cloudinary Website Speed Test | + +## Links + +- [Repository](https://github.com/cloudinary/web-speed-test-server) +- [web-speed-test-client (frontend)](https://github.com/cloudinary/web-speed-test-client) +- [Cloudinary documentation](https://cloudinary.com/documentation) +- [Documentation llms.txt index](https://cloudinary.com/documentation/llms.txt) + +Released under the MIT license.