refactor(global): the spec SDK, the S3 client and the credential libraries load on first use

capabilities/coordination/README.md

@@ -31,7 +31,7 @@ A run a caller starts, one whose context names no `trigger`, whether `run_defini
 
 `dslProblems` (`src/dsl/dsl-validation.ts`) checks a document against the DSL, before the policy's rejections join its problems, and answers a promise, which the document's reader awaits: the version it names, its names, the schema of the specification, the names of its tasks and the graph of its steps, in that order, each only when those before it found nothing. It reads the document without the runtime's own `output.shape` and without the `kind` and `because` of an error, the runtime's extensions to the DSL's errors.
 
-The schema is `workflowSchema` of `@openworkflowspec/sdk`, the specification's JSON Schema of draft 2020-12, and `@hyperjump/json-schema`, its `draft-2020-12` entry, checks a document against it by interpreting it (`src/dsl/specification-schema.ts`). The SDK's own validation, which `Classes.Workflow.deserialize` runs unless it is given `{ validate: false }`, is Ajv, which compiles each of the SDK's 171 validators into code generated from strings: it costs start-up time wherever it loads and is refused wherever no code may be generated. The interpreting validator reads the same schema as data, generates no code, and compiles the schema into the tree it interprets only when the first document is checked, in about 35 ms; its validation is asynchronous only for that, so the capability's check is a promise. It asserts the formats the schema names, `uri`, `uri-template`, `json-pointer` and `date-time`, registered with the checks of `@hyperjump/json-schema-formats` the library's own formats use, and asserted with `setShouldValidateFormat(true)`, a setting of the library for the whole process, since Ajv asserted them too and draft 2020-12 otherwise reads a format as an annotation. It gives the same answers: over the 499 workflow documents the runtime's tests check, its corpus and its input logs hold, and the 13,340 made from them by changing one value each, it accepts and refuses exactly the documents Ajv does. Its `date-time` is stricter than Ajv's: it refuses a space or a tab between the date and the time, which the grammar of RFC 3339 does not allow, and a leap second on a day that had none. Nothing calls the SDK's validation; the SDK stays for its classes and its graph, and importing it still compiles its validators, so this check alone does not take that cost out of loading the capability.
+The schema is `workflowSchema` of `@openworkflowspec/sdk`, the specification's JSON Schema of draft 2020-12, and `@hyperjump/json-schema`, its `draft-2020-12` entry, checks a document against it by interpreting it (`src/dsl/specification-schema.ts`). The SDK's own validation, which `Classes.Workflow.deserialize` runs unless it is given `{ validate: false }`, is Ajv, which compiles each of the SDK's 171 validators into code generated from strings: it costs start-up time wherever it loads and is refused wherever no code may be generated. The interpreting validator reads the same schema as data, generates no code, and compiles the schema into the tree it interprets only when the first document is checked, in about 35 ms; its validation is asynchronous only for that, so the capability's check is a promise. It asserts the formats the schema names, `uri`, `uri-template`, `json-pointer` and `date-time`, registered with the checks of `@hyperjump/json-schema-formats` the library's own formats use, and asserted with `setShouldValidateFormat(true)`, a setting of the library for the whole process, both made once, as the library is loaded (`src/dsl/dsl-libraries.ts`), since Ajv asserted them too and draft 2020-12 otherwise reads a format as an annotation. It gives the same answers: over the 499 workflow documents the runtime's tests check, its corpus and its input logs hold, and the 13,340 made from them by changing one value each, it accepts and refuses exactly the documents Ajv does. Its `date-time` is stricter than Ajv's: it refuses a space or a tab between the date and the time, which the grammar of RFC 3339 does not allow, and a leap second on a day that had none. Nothing calls the SDK's validation; the SDK stays for its schema, its classes and its graph, and importing it still compiles its validators. So the capability imports the SDK, and the validator with the formats it asserts, with the first document it reads, through `specificationSdk` and `schemaValidator` (`src/dsl/dsl-libraries.ts`), the one place each is loaded, and keeps them while the server runs; no module the capability's entry imports reaches either (`src/dsl/library-entries.test.ts`). A server that reads no workflow document never loads them, which took 130 ms, 222 files and 1.4 MiB of a server's start, and the first document read pays for them beside the compiling of the schema: the first save of a workflow without expressions took 175 ms against 34 ms when the two loaded with the capability, and the second 5.2 ms against 4.9 ms (measured on 2026-10-23 on one machine of 16 cores, with Node 26.10.0, at the median of 11 starts and of 9 servers).
 
 The schema is read with two changes that keep what it accepts. A `false` subschema is read as `{ not: {} }`, which refuses the same values, so that every refusal is a keyword of a schema at its place and a property the schema refuses outright is a `not` that allows nothing, wherever the schema refuses it. A task is checked on its own: each reference to `#/$defs/task` is read as any object, and each task `allTaskEntries` of `@on.auto/workflow-engine` walks is checked against `#/$defs/task`, so a document is checked once a task. Checked whole, a list of tasks is checked again for each type of task that may hold it, `do` and `for` among them, and the time doubles with each level the tasks nest, on the one thread every workflow shares: Ajv took 55 ms for 16 levels and about a second for 20; task by task, 30 levels take about a millisecond (`src/dsl/dsl-validation.test.ts`). A task under `use.functions`, which the policy refuses, is not checked further.
 

capabilities/coordination/src/dsl/dsl-libraries.ts

@@ -0,0 +1,49 @@
+import type { registerSchema, validate } from '@hyperjump/json-schema/draft-2020-12';
+import type { buildGraph, Classes } from '@openworkflowspec/sdk';
+
+export interface SpecificationSdk {
+  readonly workflowSchema: unknown;
+  readonly buildGraph: typeof buildGraph;
+  readonly Classes: typeof Classes;
+}
+
+export interface SchemaValidator {
+  readonly registerSchema: typeof registerSchema;
+  readonly validate: typeof validate;
+}
+
+let loadingSdk: Promise<SpecificationSdk> | undefined;
+
+let loadingValidator: Promise<SchemaValidator> | undefined;
+
+export function specificationSdk(): Promise<SpecificationSdk> {
+  loadingSdk ??= import('@openworkflowspec/sdk');
+  return loadingSdk;
+}
+
+export function schemaValidator(): Promise<SchemaValidator> {
+  loadingValidator ??= validatorAssertingFormats();
+  return loadingValidator;
+}
+
+async function validatorAssertingFormats(): Promise<SchemaValidator> {
+  const [validator, { addFormat }, formats] = await Promise.all([
+    import('@hyperjump/json-schema/draft-2020-12'),
+    import('@hyperjump/json-schema/experimental'),
+    import('@hyperjump/json-schema-formats'),
+  ]);
+  const formatChecks: ReadonlyMap<string, (text: string) => boolean> = new Map([
+    ['date-time', formats.isDateTime],
+    ['json-pointer', formats.isJsonPointer],
+    ['uri', formats.isUri],
+    ['uri-template', formats.isUriTemplate],
+  ]);
+  validator.setShouldValidateFormat(true);
+  for (const [name, isFormat] of formatChecks) {
+    addFormat({
+      id: `https://json-schema.org/format/${name}`,
+      handler: (value: unknown) => typeof value !== 'string' || isFormat(value),
+    });
+  }
+  return validator;
+}

capabilities/coordination/src/dsl/dsl-validation.ts

@@ -14,16 +14,16 @@ import {
   type JsonObject,
   type TaskList,
 } from '@on.auto/workflow-engine';
-import { buildGraph, Classes } from '@openworkflowspec/sdk';
 import specificationPackage from '@openworkflowspec/sdk/package.json' with { type: 'json' };
 import { Predicate, Schema } from 'effect';
 import satisfies from 'semver/functions/satisfies.js';
 
 import { withoutOutputShape } from '../document/workflow-output.ts';
+import { specificationSdk, type SpecificationSdk } from './dsl-libraries.ts';
 import { problemsOf, type Problem } from './schema-problems.ts';
 import { schemaChecksOf } from './specification-schema.ts';
 
-type Workflow = ReturnType<typeof Classes.Workflow.deserialize>;
+type Workflow = ReturnType<SpecificationSdk['Classes']['Workflow']['deserialize']>;
 
 type Reading = { readonly text: string; readonly document: JsonObject } | { readonly problems: readonly Problem[] };
 
@@ -44,9 +44,11 @@ export async function dslProblems(
     return unread;
   }
   const refused = (await schemaChecksOf(reading.document)).flatMap((check) => problemsOf(check));
-  return refused.length > 0
-    ? refused
-    : firstFound([() => duplicatedNames(reading.document), () => (connecting ? connectionProblems(reading.text) : [])]);
+  if (refused.length > 0) {
+    return refused;
+  }
+  const duplicated = duplicatedNames(reading.document);
+  return duplicated.length > 0 || !connecting ? duplicated : connectionProblems(reading.text);
 }
 
 function firstFound(checks: readonly (() => readonly Problem[])[]): readonly Problem[] {
@@ -130,8 +132,9 @@ function duplicatedNames(document: JsonObject): readonly Problem[] {
   );
 }
 
-function connectionProblems(text: string): readonly Problem[] {
-  const hydration = hydrated(text);
+async function connectionProblems(text: string): Promise<readonly Problem[]> {
+  const { buildGraph, Classes } = await specificationSdk();
+  const hydration = hydrated(() => Classes.Workflow.deserialize(text, { validate: false }));
   if ('problems' in hydration) {
     return hydration.problems;
   }
@@ -143,9 +146,11 @@ function connectionProblems(text: string): readonly Problem[] {
   }
 }
 
-function hydrated(text: string): { readonly workflow: Workflow } | { readonly problems: readonly Problem[] } {
+function hydrated(
+  deserialized: () => Workflow,
+): { readonly workflow: Workflow } | { readonly problems: readonly Problem[] } {
   try {
-    return { workflow: Classes.Workflow.deserialize(text, { validate: false }) };
+    return { workflow: deserialized() };
   } catch (error) {
     return { problems: [{ pointer: '', detail: messageOf(error) }] };
   }

capabilities/coordination/src/dsl/library-entries.test.ts

@@ -0,0 +1,24 @@
+import { fileURLToPath } from 'node:url';
+
+import { modulesReachedFrom } from '@on.auto/operations/testing';
+import { describe, expect, it } from 'vitest';
+
+const sources = fileURLToPath(new URL('..', import.meta.url));
+
+const libraries: ReadonlySet<string> = new Set([
+  '@hyperjump/json-schema-formats',
+  '@hyperjump/json-schema/draft-2020-12',
+  '@hyperjump/json-schema/experimental',
+  '@openworkflowspec/sdk',
+]);
+
+function librariesReachedFrom(module: string): readonly string[] {
+  return modulesReachedFrom(sources, module).packages.filter((name) => libraries.has(name));
+}
+
+describe('the SDK of the specification and the validator of its schema', () => {
+  it('are reached by no module the entry of the capability imports, and load with the first document it reads', () => {
+    expect(modulesReachedFrom(sources, 'index.ts').files).toContain('dsl/dsl-libraries.ts');
+    expect(librariesReachedFrom('index.ts')).toEqual([]);
+  });
+});

capabilities/coordination/src/dsl/specification-schema.ts

@@ -1,12 +1,4 @@
-import { isDateTime, isJsonPointer, isUri, isUriTemplate } from '@hyperjump/json-schema-formats';
-import {
-  registerSchema,
-  setShouldValidateFormat,
-  validate,
-  type SchemaObject,
-  type Validator,
-} from '@hyperjump/json-schema/draft-2020-12';
-import { addFormat } from '@hyperjump/json-schema/experimental';
+import type { SchemaObject, Validator } from '@hyperjump/json-schema/draft-2020-12';
 import {
   allTaskEntries,
   entriesOf,
@@ -18,9 +10,10 @@ import {
   type JsonEntry,
   type JsonObject,
 } from '@on.auto/workflow-engine';
-import { workflowSchema } from '@openworkflowspec/sdk';
 import { Schema } from 'effect';
 
+import { schemaValidator, specificationSdk, type SchemaValidator } from './dsl-libraries.ts';
+
 export interface SchemaRefusal {
   readonly keyword: string;
   readonly at: string;
@@ -55,14 +48,12 @@ interface Place {
   readonly location: string;
 }
 
-const taskDefinition = '#/$defs/task';
+interface Specification {
+  readonly id: string;
+  readonly schema: JsonObject;
+}
 
-const formatChecks: ReadonlyMap<string, (text: string) => boolean> = new Map([
-  ['date-time', isDateTime],
-  ['json-pointer', isJsonPointer],
-  ['uri', isUri],
-  ['uri-template', isUriTemplate],
-]);
+const taskDefinition = '#/$defs/task';
 
 const schemaKeywords: ReadonlySet<string> = new Set([
   'additionalItems',
@@ -114,11 +105,9 @@ function readMembers(members: JsonObject): JsonObject {
   );
 }
 
-const published = Schema.decodeUnknownSync(
+const publishedOf = Schema.decodeUnknownSync(
   Schema.StructWithRest(Schema.Struct({ $id: Schema.String }), [Schema.Record(Schema.String, Schema.Json)]),
-)(workflowSchema);
-
-const specification = readObject(published);
+);
 
 let compiling: Promise<Checks> | undefined;
 
@@ -136,31 +125,35 @@ export async function schemaChecksOf(document: JsonObject): Promise<readonly Sch
 }
 
 async function compiledChecks(): Promise<Checks> {
-  setShouldValidateFormat(true);
-  for (const [name, isFormat] of formatChecks) {
-    addFormat({
-      id: `https://json-schema.org/format/${name}`,
-      handler: (value: unknown) => typeof value !== 'string' || isFormat(value),
-    });
-  }
-  registerSchema(schemaObjectOf(specification));
-  const [document, task] = await Promise.all([checkAt('#'), checkAt(taskDefinition)]);
+  const [{ workflowSchema }, { registerSchema, validate }] = await Promise.all([specificationSdk(), schemaValidator()]);
+  const published = publishedOf(workflowSchema);
+  const specification: Specification = { id: published.$id, schema: readObject(published) };
+  registerSchema(schemaObjectOf(specification.schema));
+  const [document, task] = await Promise.all([
+    checkAt(validate, specification, '#'),
+    checkAt(validate, specification, taskDefinition),
+  ]);
   return { document, task };
 }
 
-async function checkAt(definition: string): Promise<Check> {
-  const root: Place = { at: '#', location: `${published.$id}${definition}` };
+async function checkAt(
+  validate: SchemaValidator['validate'],
+  specification: Specification,
+  definition: string,
+): Promise<Check> {
+  const root: Place = { at: '#', location: `${specification.id}${definition}` };
   const validator = await validate(root.location);
   return (instance) => {
     const output = validator(instanceOf(instance), 'DETAILED');
-    return output.valid ? [] : refusalsIn(output.errors, root, instance);
+    return output.valid ? [] : refusalsIn(output.errors, root, instance, specification.schema);
   };
 }
 
 function refusalsIn(
   units: readonly RefusedUnit[] | undefined,
   parent: Place,
   instance: Json,
+  schema: JsonObject,
 ): readonly SchemaRefusal[] {
   return (units ?? []).flatMap((unit) => {
     const place = placeOf(unit.absoluteKeywordLocation, parent);
@@ -169,10 +162,10 @@ function refusalsIn(
       keyword: place.at.slice(place.at.lastIndexOf('/') + 1),
       at: place.at,
       pointer,
-      rule: valueAtPointer(specification, decodeURI(fragmentOf(place.location))),
+      rule: valueAtPointer(schema, decodeURI(fragmentOf(place.location))),
       value: valueAtPointer(instance, pointer),
     };
-    return [refusal].concat(refusalsIn(unit.errors, place, instance));
+    return [refusal].concat(refusalsIn(unit.errors, place, instance, schema));
   });
 }