Skip to content

Tracking: migrate from sciline map/reduce to stages and aggregations (sciline ADR 0003) #746

Description

@SimonHeybrock

Tracking issue for the ESS side of sciline ADR 0003, Replace map/reduce with stages composed outside the graph, proposed in scipp/sciline#245. The rollout plan in the design document has the file-level sites for every item below, from a survey of the monorepo and esslivedata on 2026-09-11. Links point at the sciline branch until the ADR PR merges. Nothing here starts before the ADR is accepted and the additive sciline release is out.

In short: a Stage is the part of a pipeline from input keys to output keys with the static part held; an Aggregation is two stages with an accumulator per accumulation key between them. Each package returns its own object in place of a map/reduced pipeline, holding the pipeline, its aggregations, and the contributions by member.

ess.reduce, additive

  • Accumulators: maybe_hist moves out of Accumulator.push into the subclasses that histogram; a Forwarder; a test that every accumulator satisfies sciline.Accumulator.
  • StreamProcessor rewritten as a driver over three stages, a forwarder, and accumulators, keeping its interface, against the streaming, visualize, and accumulator tests. Needs the sciline follow-up (provide, reporter, visualize support).
  • assign_parameter_values and parameter_mappers replaced by the package-object protocol, with or after esssans.
  • docs/user-guide/polarization/zoom.ipynb (uses get_mapped_node_names and with_sample_runs) migrates with esssans.

Reduction packages, one PR each, in this order

  • esssans: with_sample_runs/with_background_runs become one aggregation per run type with a shared finalize stage held by a package object; with_pixel_mask_filenames becomes a PixelMaskFilenames list parameter and two providers; with_banks becomes compute_members; ZoomTransmissionFractionWorkflow; parameter_mappers registrations; LoKI and zoom notebooks; tests using compute_mapped. Sets the package-object pattern.
  • essdiffraction: with_pixel_mask_filenames becomes a list parameter and one provider and loses the empty-list cyclebane workaround. All 22 test call sites pass [], so add one test with a mask file. The two DREAM advanced-notebook tables (one with two columns) become an aggregation.
  • essspectroscopy: three bank folds in the bifrost builders. Open design point for that PR: NeXusData depends on the run, so the bank aggregation sits inside per-run work when runs are members; either the package object owns a per-run bank aggregation or a DetectorNames list parameter with a looping provider. compute_mapped in one test and one notebook becomes compute_members.
  • essreflectometry: with_filenames becomes an aggregation with up to seven accumulation keys and loses the try/except around each reduce; BatchProcessor.compute loses its compute_mapped fallback; batch_processor enters the aggregation for a list-valued Filename[SampleRun]; the GUI folds; amor and estia notebooks; tests.
  • essnmx: notebooks and one test fixture only; mcstas_workflow.ipynb imports cyclebane.graph directly and must stop.
  • essimaging: drop visualize(compact=) in two notebooks.

Every package also drops visualize(compact=) from its notebooks in its own PR. constraints= (essreduce nexus workflows, offspec, amor) stays until the breaking sciline release.

esslivedata

  • The bifrost bank fold in config/instruments/bifrost/factories.py sits in the static part of the streaming pipeline; it becomes Aggregation(...).compute(...) with the result set as a parameter before the StreamProcessor is built. Consumes the rewritten StreamProcessor through the essreduce bump.
  • Drop the direct cyclebane>=26.9.0 pin once sciline drops cyclebane.

Then

  • essreduce raises its sciline minimum to the breaking release; offspec and amor drop constraints= in the same bump.
  • essapps: apply the edits listed at the end of stages.md on branch architecture-sketch, reading Fold as Aggregation and "accumulation point" as accumulation key.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions