[ESSREDUCE] Minimal implementation-independent workflow specifications

[SimonHeybrock]

Fresh counter-proposal for #653, as announced in the review on #689 — I propose closing that PR in favor of this one.

The core difference to #689: the spec is pure interface. No workflow factory, no sciline keys, no use_workflow_default — nothing that presumes a particular implementation or execution location. The same spec must make sense whether the workflow runs as a local sciline pipeline, behind a web service, or as a cluster job; compute binding is a parallel mechanism, deliberately out of scope here.

The second commit extends the first with what the essapps architecture sketch needs (D13, see the comment below): outputs as a typed model and data-reference fields, so that any output of one workflow run can be checked against the parameter it feeds in the next.

Main decisions (full rationale in ADR 0001, included in this PR):

• One pydantic params model per workflow instead of per-sciline-key Parameter entries. Enables cross-parameter validation and JSON Schema for free. The graph-derived "select outputs → see relevant params" feature is dropped, not ported: it treats output selection as workflow slicing, which only the sciline implementation can express.
• Outputs are a pydantic model too, in the same vocabulary as params: title and description as field metadata, optional fields allowed, primary output first. Chaining is then a type check between an output field and a parameter field, and where a framework stores an output is decided by the field's type, not by the spec.
• Data fields: a parameter or output holding a file or array rather than a literal is a field of type Ref, a reference to an output of an earlier run or to a dataset the framework did not compute, annotated with the format of the bytes (raw NeXus file, scipp object, opaque file) and, for scipp data, an ArraySpec (dims, unit, coord units, binned). There is no separate input section: an input is a parameter of data-field type. Collections of data fields are allowed on both sides, and a reference may name one element of a collection output. The spec says nothing about how a workflow gets at the bytes, a path or an in-memory object; that is decided where the workflow is called, by the executor binding, so a framework can add an in-memory path for chained runs without touching a spec. The spec layer still imports no scipp; the structural check of a scipp object against its ArraySpec lives in spec.conversions.
• title and description mandatory; identity is name + version — no instrument, no WorkflowId class. Scoping and global uniqueness are the enumerating context's responsibility. An optional code_revision is provenance, not identity.
• Two forms with a one-way projection: WorkflowSpec (in-process, holds the model classes, full validation) and SerializedWorkflowSpec (plain data, params and outputs as JSON Schema, JSON-round-trippable). Data fields appear in the schema under a dataField key with their format and structure. No inverse — validators don't survive JSON Schema, so authoritative validation stays with the process owning the model classes; remote consumers validate optimistically. This matches the announcement-as-contract design from Rethink workflow registration: move schema ownership to domain packages esslivedata#889.
• Scipp-free shared parameter vocabulary (unit enums, range/edges models with cross-field validation, Quantity for small values with a unit); scipp conversions quarantined in spec.conversions. Value defaults are set by workflow authors at the use site, not by the vocabulary.
• No registry: enumeration mechanisms are out of scope, any works against the same spec type. One requirement on workflow packages: the module defining a spec must be importable without importing the workflow code.
• Explicit goal: ess.livedata WorkflowSpec eventually inherits from this. esslivedata already declares outputs as a model with field metadata, so the remaining difference is field types: its sc.DataArray template fields become data fields constrained by ArraySpec.

Deferred until a consumer exists, listed in the ADR so they are not lost: declared failure reasons, an intermediate flag for retention, declared collection keys, and spec discovery via entry points.

Adds pydantic>=2.5 as an essreduce dependency (already present in the locked CI environments). The existing ess.reduce.parameter/workflow/widgets machinery is superseded but untouched; removal is a later hard break — no one depends on it currently.

🤖 Generated with Claude Code

[SimonHeybrock]

Reminder of the extensions the essapps architecture sketch (scipp/essapps, branch architecture-sketch, docs/developer/architecture.md, decision D13) needs from this spec, so they are not lost while the PR is open. A prototype of all of them lives in ess.apps.spec on that branch.

• Outputs as a typed model. outputs becomes a pydantic model class in the same vocabulary as params, with title and description as field metadata and optional fields allowed. Chaining is then a type check between an output field and a parameter field. As proposed, non-array outputs have no type, which breaks "outputs can be inputs" for exactly the values that most often feed the next workflow (beam centre, direct beam).
• A data-reference field type. A field marked with a kind (raw NeXus file, opaque file, scipp array) holds a reference to another record's output at submission and the materialized value (path or scipp object) inside the workflow. Arrays are constrained by the same ArraySpec outputs use. A field may be a union of a literal and a reference. The prototype validates this without importing scipp in the spec layer.
• Collections on both sides. list[...] and dict[str, ...] of one declared type as parameters and as outputs; a reference may name one element of a collection output by key.
• ArraySpec.binned, so consumers know event data that must not be plotted directly.
• Two small spec fields: cheap: frozenset[str], the parameters a warm workflow can change without recomputing the expensive part (what lets a UI offer a slider), and an optional code_revision.
• Entry points split by role: specs in one group, workflow factories in another under the same name, so a service can load and validate every spec without importing workflow code. This matches the ADR's "the backend never imports a factory" and the esslivedata split.

None of this changes the ADR's core: pure interface, one params model, one-way serialization, identity by name and version.

[SimonHeybrock]

I do not like how this got complex in the latest commit -- need to think more about how to reference input (and output) data. Should inputs always be filenames and outputs DataArray (or similar)? The fundamental problem appears to be that we "hard-code" the file loading as part of our Sciline workflows, making it tricky to hand an DataArray into a workflow (at least on the spec level), as it feeds into a different workflow node (or nodes). We are avoiding similar problems on the output side by saying "workflows should not have side-effects, file writing happens outside workflows", but doing the same for inputs is much more challenging.