Spec 010 T006–T007: the Java-against-Java parity harness

specs/010-java-services/tasks.md

@@ -21,7 +21,7 @@
 
 - [ ] T001 [EXT] Ask the Reactome organisation's owners to create the library-service repository, licensed AGPL-3.0 (research R2), and record its name in this file and in `plan.md`
 - [ ] T002 [EXT] Ask the owners of `reactome/content-service` and `reactome/analysis-service` to merge the deployed branches (`7b9eceb` on `add-content-page-endpoints`; `9b147e1`, PR #25). The pins stay on these commits either way (research R1).
-- [x] T003 [P] Record the pinned commits of research R1 in one place, `deploy/java/PINS.md`, each with the reason it is pinned and how it was verified (byte comparison of classes)
+- [x] T003 [P] Record the pinned commits of research R1 in one place, `deploy/java/PINS.md`, each with the reason it is pinned and how it was verified (byte comparison of classes) — #505
 
 ---
 
@@ -31,22 +31,22 @@
 
 ### Tests first
 
-- [x] T004 Extend `deploy/nginx/routes.spec.ts`:
+- [x] T004 Extend `deploy/nginx/routes.spec.ts`: — #505
   - parse `deploy/nginx/common/routes.conf` as well as `routes-core.conf`, in nginx's order;
   - classify each `proxy_pass` as node, Java service or Tomcat;
   - assert that no URL in node's endpoint samples or the proxy-order node list reaches a Java service or Tomcat.
   - Show it failing by adding a location that sends a node URL to Tomcat.
-- [x] T005 Add a test in `deploy/nginx/routes.spec.ts` that an exporter address routed away from Tomcat keeps the exporter location's rate limit. Show it failing with a `routes-core.conf` location for `/ContentService/exporter/event/` (research R6 trap).
+- [x] T005 Add a test in `deploy/nginx/routes.spec.ts` that an exporter address routed away from Tomcat keeps the exporter location's rate limit. Show it failing with a `routes-core.conf` location for `/ContentService/exporter/event/` (research R6 trap). — #505
 
 ### Implementation
 
-- [ ] T006 Create the parity harness in `tools/java-parity/`, per research R7:
+- [x] T006 Create the parity harness in `tools/java-parity/`, per research R7: — #507
   - same request to a reference and a candidate, same Host;
   - compare status, a fixed header set and the SHA-256 of the body;
   - per-class normalisers, declared in `tools/java-parity/normalisers.mjs`;
   - JSON report of identical, different and both-error counts, with no bodies or token values in the output.
-- [ ] T007 Add a self-test to `tools/java-parity/parity.spec.mjs`. Two stub servers that differ in one byte must report a difference; identical stubs must not. Show it failing with the comparison disabled.
-- [ ] T008 [P] Add the PDF normaliser to `tools/java-parity/normalisers.mjs`:
+- [x] T007 Add a self-test to `tools/java-parity/parity.spec.mjs`. Two stub servers that differ in one byte must report a difference; identical stubs must not. Show it failing with the comparison disabled. — #507
+- [ ] T008 [P] (With the first PDF comparison, T030: it needs `qpdf` in a container.) Add the PDF normaliser to `tools/java-parity/normalisers.mjs`:
   - `qpdf --qdf --object-streams=disable --static-id`, run in a throwaway container;
   - mask CreationDate, ModDate and the cover-page date.
   - Test it in `tools/java-parity/parity.spec.mjs`: two PDFs differing only in those fields compare equal, and a PDF with a changed body does not.
@@ -205,12 +205,11 @@ Each cutover is one PR. It adds the group's patterns to node's table, one nginx
 
 ## Phase 8: Polish and side findings
 
-- [ ] T058 [EXT] Decide the analysis report's species overview figure. The PDF report embeds the retired Fireworks image, drawn by Java's fireworks-exporter; the site's overview is now the Reacfoam (Voronoi) map. Showing that in the report needs analysis-report changed upstream (its owners) or the figure supplied to it.
-
 - [ ] T054 [P] Set `client_max_body_size` in `deploy/nginx/local.conf` to match the other deployments (64 MB), so local uploads over 1 MB are not refused. Test it with `deploy/nginx/test-config.sh`.
 - [ ] T055 [P] Declare the proxy package that `projects/website-angular/src/scripts/serve-prod.js` uses in `package.json` (it is only present as a dev-only transitive dependency), and check the lockfile changes only that entry
 - [ ] T056 Update `specs/006-content-service-to-node/tasks.md` Phase 9 rows with the PR that moved each Java-homed operation, as each moves
 - [ ] T057 Update the curator-facing documents if any reader-visible behaviour is touched by a cutover (constitution: Development Workflow)
+- [ ] T058 [EXT] Decide the analysis report's species overview figure. The PDF report embeds the retired Fireworks image, drawn by Java's fireworks-exporter; the site's overview is now the Reacfoam (Voronoi) map. Showing that in the report needs analysis-report changed upstream (its owners) or the figure supplied to it.
 
 ---
 

tools/java-parity/normalisers.mjs

@@ -0,0 +1,35 @@
+/**
+ * The normalisers parity may apply, each declared here with what it removes
+ * and why. A difference is only ever hidden by one of these, by name, in the
+ * run that uses it -- never by loosening the comparison itself.
+ *
+ * Each removes exactly its target's bytes and leaves every other byte as it
+ * came. A JSON parse-and-print round trip would not: it loses digits past
+ * 2^53, folds -0 into 0, drops duplicate keys, reorders integer-like keys and
+ * rewrites number formats and escapes -- differences the API's R and Python
+ * users would see, hidden from the comparison.
+ */
+import { gunzipSync } from 'node:zlib';
+
+/**
+ * An analysis token holds the time and a per-process counter, so two runs
+ * never share one: its value -- only its value -- is blanked inside
+ * `summary`.
+ */
+function withoutToken(bytes) {
+  // latin1 maps bytes to code units one to one, so nothing else is re-encoded.
+  const text = bytes.toString('latin1');
+  const token = /("summary"\s*:\s*\{[^{}]*?"token"\s*:\s*)"(?:[^"\\]|\\.)*"/;
+  // No token to blank is not an answer this normaliser is for: an error, not a silent pass-through.
+  if (!token.test(text)) throw new Error('no summary.token');
+  return Buffer.from(text.replace(token, '$1""'), 'latin1');
+}
+
+export const normalisers = {
+  /** Byte for byte. */
+  identity: (body) => body,
+  /** An analysis result (JSON) with its token blanked. */
+  analysisResult: (body) => withoutToken(body),
+  /** The gzipped result download, decompressed, with its token blanked. Not gzip: an error. */
+  analysisResultGz: (body) => withoutToken(gunzipSync(body)),
+};

tools/java-parity/parity.mjs

@@ -0,0 +1,287 @@
+#!/usr/bin/env node
+/**
+ * Java-against-Java parity (spec 010, research R7): the same requests to a
+ * reference -- what answers today -- and a candidate -- what replaces it --
+ * compared on status, every response header but named noise, and the bytes
+ * of the body after a declared normaliser.
+ *
+ * Over node:http, not fetch: fetch drops a Host set by the caller, so each
+ * side saw its own address and every answer built from Host differed for a
+ * reason that was not a difference; and fetch hides Transfer-Encoding.
+ *
+ * Reports carry digests and verdicts, never bodies -- some answers hold
+ * people's records -- and errors carry only their kind, never a message,
+ * which can quote a body. With --redact not even the request, because some
+ * paths are tokens anyone holding them can read.
+ *
+ * Usage:
+ *   node tools/java-parity/parity.mjs --reference URL --candidate URL \
+ *     --requests requests.json [--normalise NAME] [--host HOST] [--timeout MS]
+ *     [--ignore-header NAME]... [--calibrate] [--redact] [--simultaneous]
+ * requests.json: [{ "method": "GET", "path": "/...", "headers": {},
+ *                   "body": "text" | "bodyBase64": "..." | "bodyFile": "path" }]
+ * Exit: 0 all identical; 1 differences; 2 errors, unstable requests, or
+ * nothing compared -- a run that compared nothing is not a pass; 3 usage.
+ */
+import { createHash } from 'node:crypto';
+import { readFileSync } from 'node:fs';
+import { request as httpRequest } from 'node:http';
+import { request as httpsRequest } from 'node:https';
+import { brotliDecompressSync, gunzipSync, inflateSync } from 'node:zlib';
+import { normalisers } from './normalisers.mjs';
+
+/** Headers that differ between any two servers by nature, and only these. */
+const NOISE = new Set(['date', 'connection', 'keep-alive', 'server']);
+
+function requestBody({ body, bodyBase64, bodyFile }) {
+  if (bodyFile) return readFileSync(bodyFile);
+  if (bodyBase64) return Buffer.from(bodyBase64, 'base64');
+  return body === undefined ? undefined : Buffer.from(body);
+}
+
+/**
+ * A base URL as the place to send a path: scheme, host (an IPv6 one without
+ * its brackets), port, and any context path, which is kept -- dropping it
+ * sent both sides to the same wrong resource, whose matching 404s compared
+ * as a pass. Anything it cannot honour is refused, never ignored.
+ */
+function baseOf(base) {
+  const url = new URL(base);
+  if (url.protocol !== 'http:' && url.protocol !== 'https:')
+    throw new Error(`base must be http or https: ${url.protocol}`);
+  if (url.search || url.hash) throw new Error('base must not carry a query or fragment');
+  return {
+    send: url.protocol === 'https:' ? httpsRequest : httpRequest,
+    hostname: url.hostname.replace(/^\[(.*)\]$/, '$1'),
+    port: url.port || (url.protocol === 'https:' ? 443 : 80),
+    prefix: url.pathname.replace(/\/$/, ''),
+  };
+}
+
+/**
+ * Headers as sent: from rawHeaders, every copy kept and joined in order --
+ * the parsed map keeps one Content-Type or Location of two, so a side that
+ * sent two compared equal to one that sent one.
+ */
+function rawHeaders(raw) {
+  const headers = {};
+  for (let at = 0; at < raw.length; at += 2) {
+    const name = raw[at].toLowerCase();
+    headers[name] = name in headers ? `${headers[name]}, ${raw[at + 1]}` : raw[at + 1];
+  }
+  return headers;
+}
+
+const failure = (kind) => Object.assign(new Error(kind), { kind });
+
+/** One answer: status, headers as sent, and the body as sent -- within `timeout` in all. */
+function answer(base, req, { host, timeout }) {
+  const { send, hostname, port, prefix } = baseOf(base);
+  const body = requestBody(req);
+  return new Promise((resolve, reject) => {
+    const outgoing = send(
+      {
+        hostname,
+        port,
+        method: req.method ?? 'GET',
+        path: `${prefix}${req.path}`,
+        headers: {
+          ...(req.headers ?? {}),
+          ...(host ? { host } : {}),
+          ...(body ? { 'content-length': body.length } : {}),
+        },
+      },
+      (response) => {
+        const chunks = [];
+        response.on('data', (chunk) => chunks.push(chunk));
+        response.on('end', () => {
+          clearTimeout(timer);
+          resolve({
+            status: response.statusCode,
+            headers: rawHeaders(response.rawHeaders),
+            body: Buffer.concat(chunks),
+          });
+        });
+        // A reset part way through: the run goes on at once, and no timer
+        // is left armed to keep the process alive until it fires.
+        response.on('error', (error) => {
+          clearTimeout(timer);
+          reject(error);
+        });
+      }
+    );
+    // The whole request, not a silence: a body trickled a byte at a time
+    // never goes quiet, and held the run open.
+    const timer = setTimeout(() => outgoing.destroy(failure('timeout')), timeout);
+    outgoing.on('error', (error) => {
+      clearTimeout(timer);
+      reject(error);
+    });
+    outgoing.end(body);
+  });
+}
+
+/** The body a client reads: Content-Encoding undone (the header itself is still compared). */
+function decoded({ headers, body }) {
+  const encoding = (headers['content-encoding'] ?? '').trim().toLowerCase();
+  try {
+    if (encoding === 'gzip') return gunzipSync(body);
+    if (encoding === 'br') return brotliDecompressSync(body);
+    if (encoding === 'deflate') return inflateSync(body);
+  } catch {
+    throw failure('body decoding');
+  }
+  return body;
+}
+
+/** A body through the normaliser; a body it cannot take is the normaliser's failure. */
+function normalised(normalise, answer) {
+  const body = decoded(answer);
+  try {
+    return normalise(body);
+  } catch {
+    throw failure('normaliser');
+  }
+}
+
+const digest = (bytes) => createHash('sha256').update(bytes).digest('hex');
+
+/** What differs between two answers, bodies normalised the same way. */
+function differences(a, b, normalise, ignored = new Set()) {
+  const found = [];
+  if (a.status !== b.status) found.push('status');
+  const names = new Set([...Object.keys(a.headers), ...Object.keys(b.headers)]);
+  for (const name of [...names].sort()) {
+    if (NOISE.has(name) || ignored.has(name)) continue;
+    // Content-Length follows the body; under a normaliser the body is compared instead.
+    if (name === 'content-length' && normalise !== normalisers.identity) continue;
+    if (a.headers[name] !== b.headers[name]) found.push(name);
+  }
+  // Different statuses are a difference whatever the bodies are: an error
+  // page the normaliser cannot take must not turn that into an error.
+  if (found.includes('status')) {
+    const raw = (answer) => {
+      try {
+        return decoded(answer);
+      } catch {
+        // An error page that claims an encoding it lacks: compare it as sent.
+        return answer.body;
+      }
+    };
+    if (digest(raw(a)) !== digest(raw(b))) found.push('body');
+    return found;
+  }
+  if (digest(normalised(normalise, a)) !== digest(normalised(normalise, b))) found.push('body');
+  return found;
+}
+
+/** Only the kind of a failure: a message can quote the body or the URL. */
+const kindOf = (error) => error?.kind ?? (error?.code ? `network ${error.code}` : 'failed');
+
+/**
+ * Compare every request. With `calibrate`, the reference is asked again:
+ * where it disagrees with itself (live third-party data), the request is
+ * "unstable" and counted neither way.
+ */
+export async function compareAll(
+  requests,
+  {
+    reference,
+    candidate,
+    normalise = normalisers.identity,
+    host,
+    timeout = 120_000,
+    calibrate = false,
+    redact = false,
+    simultaneous = false,
+    ignoreHeaders = [],
+  }
+) {
+  // Refused before anything is sent: a base it cannot honour is a usage error.
+  baseOf(reference);
+  baseOf(candidate);
+  const ignored = new Set(ignoreHeaders.map((name) => name.toLowerCase()));
+  const results = [];
+  const counts = { identical: 0, different: 0 };
+  const count = (key) => (counts[key] = (counts[key] ?? 0) + 1);
+  const ask = (base, req) => answer(base, req, { host, timeout });
+  for (const [index, req] of requests.entries()) {
+    const label = redact ? `#${index}` : `${req.method ?? 'GET'} ${req.path}`;
+    try {
+      const [ref, cand] = simultaneous
+        ? await Promise.all([ask(reference, req), ask(candidate, req)])
+        : [await ask(reference, req), await ask(candidate, req)];
+      const result = { request: label, status: [ref.status, cand.status] };
+      if (calibrate) {
+        result.calibrated =
+          differences(ref, await ask(reference, req), normalise, ignored).length === 0;
+        if (!result.calibrated) {
+          count('unstable');
+          results.push(result);
+          continue;
+        }
+      }
+      result.differs = differences(ref, cand, normalise, ignored);
+      count(result.differs.length ? 'different' : 'identical');
+      results.push(result);
+    } catch (error) {
+      count('errors');
+      results.push({ request: label, error: kindOf(error) });
+    }
+  }
+  // A declared exclusion is part of the result, never silent.
+  return { counts, ...(ignored.size ? { ignoredHeaders: [...ignored] } : {}), results };
+}
+
+/** The exit code a report earns: 0 only when something was compared and all of it was identical. */
+export function verdict({ counts }) {
+  if (counts.errors || counts.unstable || counts.identical + counts.different === 0) return 2;
+  return counts.different ? 1 : 0;
+}
+
+// Started directly rather than imported by a test.
+if (process.argv[1] && process.argv[1].endsWith('parity.mjs')) {
+  const args = process.argv.slice(2);
+  const option = (name) => {
+    const at = args.indexOf(`--${name}`);
+    return at === -1 ? undefined : args[at + 1];
+  };
+  const flag = (name) => args.includes(`--${name}`);
+  const normalise = normalisers[option('normalise') ?? 'identity'];
+  if (!option('reference') || !option('candidate') || !option('requests') || !normalise) {
+    console.error(
+      'usage: parity.mjs --reference URL --candidate URL --requests FILE [--normalise NAME] [--host HOST] [--timeout MS] [--ignore-header NAME]... [--calibrate] [--redact] [--simultaneous]'
+    );
+    process.exit(3);
+  }
+  // Usage errors exit 3, never 1 ("differences"): an unusable base, a
+  // dangling --ignore-header.
+  const usage = (message) => {
+    console.error(`parity.mjs: ${message}`);
+    process.exit(3);
+  };
+  for (const base of [option('reference'), option('candidate')]) {
+    try {
+      baseOf(base);
+    } catch {
+      usage('each base must be an http or https URL without a query');
+    }
+  }
+  args.forEach((arg, at) => {
+    if (arg === '--ignore-header' && (!args[at + 1] || args[at + 1].startsWith('--')))
+      usage('--ignore-header needs a header name');
+  });
+  const report = await compareAll(JSON.parse(readFileSync(option('requests'), 'utf8')), {
+    reference: option('reference'),
+    candidate: option('candidate'),
+    normalise,
+    host: option('host'),
+    timeout: option('timeout') ? Number(option('timeout')) : undefined,
+    calibrate: flag('calibrate'),
+    redact: flag('redact'),
+    simultaneous: flag('simultaneous'),
+    ignoreHeaders: args.flatMap((arg, at) => (arg === '--ignore-header' ? [args[at + 1]] : [])),
+  });
+  console.log(JSON.stringify(report, null, 2));
+  process.exit(verdict(report));
+}