feat: add experimental data entities as mirrors of the computational ones

[timurbazhirov]

What this is

A minimal, working implementation of experimental data support: four root entities, the pieces they compose, and examples built from real instrument and lab-system records. It is deliberately the smallest thing that stands on its own. The concepts are outlined and each is exercised by a record that actually validates.

Design and dataset survey are in plan/upcoming/2026-09-02-experimental-data-*.md (on claude/experimental-data-support-s3bf63).

The idea: mirror, don't fork

Experimental work has the same shape as simulation and different nouns. Each new entity is built as the mirror of the computational one it corresponds to, composing the same mixins and keeping the fields that transfer.

Computational	Experimental	What changes
material	sample	basis/lattice become an optional _material reference, because a specimen may have no known structure
software/application	instrument	version becomes firmware, build becomes serialNumber
job	measurement	compute becomes instrument, _material becomes _sample
workflow	process	subworkflows become stages, units become steps

Properties do not fork. A measured film thickness is an ordinary property/holder: same data union, same exabyteId, same repetition. Only source.info widens, from the exabyte job reference to a union over exabyte, measurement and process. The measurement reference is shaped like the job reference (measurementId where a job has jobId, channel where it has unitId), so a consumer that already reads job provenance needs no new code path.

What is in the diff

• Four root entities and their components: sample (+ layer, position, library), instrument (+ component, and the instrument_property mixin mirroring job/compute_property), measurement (+ data channel, scanning probe parameters), process (+ stage, step, and the target/precursor/gas source union).
• Foundation: core/abstract/multidimensional_array and core/reusable/array_data for rasters and grids (core/abstract/3d_grid is a k-point mesh descriptor, not a data grid); core/reusable/file_reference for binary instrument output, which system/file_source cannot hold because it requires text; quantity/*, environment, identifier; system/activity, the status and timing mixin shared by measurement and process.
• techniques_category, narrowing the same CateCom tiers that categorize models and methods.
• Three properties: film_thickness, surface_roughness, hysteresis_loop, registered the usual five-touch way.
• Units: 18 new families (temperature, time, voltage, current, flow rate, deposition rate, and so on) plus Torr-scale pressures. Existing families are extended, never redefined.
• Wiring: ENTITY_DOMAINS gains the four nouns so the layer taxonomy stays total; the ontology map gains an experiment colour family; a concept page joins the docs.
• Pydantic models regenerated with the datamodel-code-generator==0.28.5 that pyproject.toml pins and synced into src/py/mat3ra/esse/models/ (details below).

Why it composes what it composes

process/step reuses workflow/unit/mixins/base unchanged, so a process is walked with the same head/next/flowchartId mechanics as a workflow: a deposition recipe and a calculation are the same kind of object to a traversal. system/activity exists so measurement and process cannot grow two different status vocabularies. instrument/instrument_property mirrors job/compute_property so "has an instrument" is defined once.

Examples are real records

Not invented: an HTEM combinatorial library (6705) and one of its 44 positions, an Asylum Research Jupiter DART piezoresponse scan and switching spectroscopy with their actual scan parameters, and a sputter-and-anneal run from the NLR lab system.

Two shapes corrected by real data

Both were found by validating a real record against the schemas, not by inspection:

1. activity.operators was reusing the bibliographic author schema, which requires a surname. A lab record identifies an operator by whatever the instrument logged, often a single username, so it is now a name and an optional affiliation.
2. vendor and model are no longer required on an instrument. Identity comes from the name the entity mixin supplies, and a lab record often knows a chamber only by the short name its operators use (pdac_com5).

Tightened after a review pass

Each of these was confirmed with a probe record against the resolved schemas before and after (commit 1104d14):

• Technique unions on process, stage and instrument use anyOf, not oneOf. A record classified only to tier1 satisfied both branches and oneOf rejected it as ambiguous, so a run not yet classified below tier1 could not be stored at all.
• The stage composes system/activity instead of re-declaring status and timing by hand, which was exactly the drift the mixin exists to prevent.
• A sample's _parent is a system/_sample reference, as measurement._sample already was; a parent pointing at a Material is now rejected.
• The technique subtype vocabularies were defined but referenced by nothing, leaving subtype a free string. They are now referenced from each branch, so a misspelt subtype fails validation.
• The step keeps the workflow-unit status vocabulary and its description says why: run-level states such as cancelled belong to the stage and process, which carry the activity vocabulary.

And the guard tests the existing suites did not make, in tests/js/experimentalSchemas.tests.ts (commits e06e451, 9b24c75):

• a holder whose source is a measurement validates, and the job-sourced example still does (the regression test for the one existing schema this PR widens);
• a measurement source without a measurement id is rejected;
• every schema this PR adds carries a description, because those become the pydantic docstrings and the map's detail panel;
• no example carries an inline array over the budget stated on array_data;
• the measurement example's parameters validate against measurement/parameters/scanning_probe_microscopy, which nothing narrowed until now.

How the models were synced

The pre-commit hook writes datamodel-codegen output to dist/py, which is gitignored, while the committed models live under src/py/mat3ra/esse/models/ as a real directory rather than the link pyproject.toml anticipates. Commit caec728 copies the hook's output over the committed tree, generated with the pinned 0.28.5 so untouched modules keep their headers. To make the 270-file diff reviewable, this is what it contains, measured against a regeneration on the base commit:

Files	What
50 new	modules for the schemas this PR adds; every one imports, with field descriptions as docstrings
9 changed in substance	property/holder (source union and three data variants), properties_directory/enum_options, and seven property modules whose pressure, force or frequency unit enums gained the new units
152 changed by renumbering only	the generator's global counter for de-duplicated class names (Units197 becomes Units221); safe to skip
58 removed, 14 refreshed	drift that predates this branch: 54 modules whose schemas no longer exist, 4 stale duplicates of modules the generator places under a package, and 14 modules behind their dev schemas (job, workflow, unit execution)

Importing every module file: 670 of 700 succeed. The 30 that do not are the versioned apse directories, whose dotted names are not importable, and the title-derived Reusable_schema_for_energy_value… modules; both fail identically on dev and are a generator-configuration question, not this PR's.

Verification

npm run lint-entity-graph     # passes; 610 schemas, 0 cycles, 0 unresolved refs
npm test                      # 39 passing
python -m unittest discover --start-directory tests/py/esse/    # 267 passing
npm run build-docs-pages -- --output site && npm run check-site-links --site site

Example coverage rises from 209/564 (37%) to 254/610 (42%). Pinned graph counts in tests/js/entityGraph.tests.ts are updated from the lint output, not by hand.

The round trip is exercised end to end in a companion PR on mat3ra/parsers, which parses a real NLR deposition record into a process config and validates it against the schema added here.

Two things reviewers should decide

• Whether the model drift lands here. The regeneration necessarily folds in the pre-existing drift listed above (54 orphaned modules removed, 14 refreshed). It is a single commit at the top of the branch, so it can be dropped from here and opened against dev on its own if that is preferred.
• Scope of the relaxations. Removing required from instrument_properties and reshaping activity.operators are the two places this PR touches judgement rather than adding files.

Not in this PR

Vendor file formats under apse/ (Nanonis, Asylum, HTEM exports), technique catalogue entries beyond the parameter block, and the HTEM crosswalk. Those are phases 4 and 5 of the plan and want real files inspected first.

🤖 Generated with Claude Code

https://claude.ai/code/session_017pSxTJQwszq7y7n4SKRQRS

---

Generated by Claude Code