Merging Configuration implementation into Development

[BelowBayesline]

No description provided.

[Copilot]

Warning

• Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Pull request overview

Adds configuration injection, run-history persistence, expanded examples, documentation, tests, and CI tooling.

Changes:

• Introduces global/stage configuration handling and execution-context access.
• Adds YAML run logging, historical loading, and configuration comparison.
• Expands tests, examples, documentation, typing, and CI checks.

Reviewed changes

Copilot reviewed 47 out of 53 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
tests/test_stage.py	Adds stage behavior tests.
tests/test_runner.py	Tests YAML logging and config diffs.
tests/test_pipeline_architecture.py	Expands pipeline integration coverage.
tests/test_models.py	Adds model and serialization tests.
tests/test_logger.py	Tests historical run extraction.
tests/test_loader.py	Tests historical run loading.
tests/test_execution.py	Tests configuration context helpers.
setup.cfg	Updates Python support and development dependencies.
README.md	Expands usage and project documentation.
pyproject.toml	Adds mypy and Ruff settings.
onsrap/warnings.py	Adds configuration warning classes.
onsrap/stage.py	Extends stage validation and factories.
onsrap/runner.py	Adds run persistence and config comparisons.
onsrap/run_pipeline.py	Documents the command entrypoint.
onsrap/py.typed	Marks the package as typed.
onsrap/logger.py	Adds historical run discovery.
onsrap/loader.py	Adds historical run loading.
onsrap/graph.py	Improves graph validation and typing.
onsrap/execution.py	Adds stage/global configuration access.
onsrap/errors.py	Adds configuration and history errors.
onsrap/__init__.py	Updates public configuration exports.
examples/pipeline_3/main.ipynb	Adds a minimal notebook example.
examples/pipeline_2/scripts/2_reporting.py	Adds report generation.
examples/pipeline_2/scripts/1_derive_vars.py	Adds derived-variable processing.
examples/pipeline_2/scripts/0_clean_data.py	Adds data cleaning.
examples/pipeline_2/outputs/order_analysis.md	Adds generated example output.
examples/pipeline_2/main.py	Adds configuration-driven execution.
examples/pipeline_2/data/orders.csv	Adds example source data.
examples/pipeline_2/data/orders_prepped.csv	Adds prepared example data.
examples/pipeline_2/data/orders_cleaned.csv	Adds cleaned example data.
examples/pipeline_2/conf.yaml	Defines the example configuration.
examples/pipeline_1/scripts/2_reporting.py	Uses execution-context paths.
examples/pipeline_1/scripts/1_preprocessing.py	Uses configuration path helpers.
examples/pipeline_1/scripts/0_data_validation.py	Uses execution-context directories.
examples/pipeline_1/runs/README.md	Corrects run-directory documentation.
examples/pipeline_1/main2.py	Demonstrates dependency ordering.
examples/pipeline_1/main.py	Configures run output location.
examples/pipeline_1/Example.md	Expands example purpose.
examples/pipeline_1/logs/onsrap.log	Restricted generated log update.
docs/contributor_guide/CONTRIBUTING.md	Documents new quality tools.
configuration.md	Adds configuration architecture documentation.
CHANGELOG.md	Records run-output changes.
.pre-commit-config.yaml	Adds mypy and strengthens Bandit.
.gitignore	Ignores generated example artifacts.
.github/workflows/python-package.yml	Expands CI quality and test jobs.
.github/workflows/package-build.yml	Adds package-build verification.
.github/workflows/deploy-docs.yml	Adds documentation deployment.

Files excluded by content exclusion policy (1)

• examples/pipeline_1/logs/onsrap.log

Suppressed comments (7)

configuration.md:58

• StageConfig has no datasets attribute; from_mapping only removes metadata, so a datasets key remains inside .variables. The documented accessor raises AttributeError and the preceding row also incorrectly says datasets are excluded.

| `_variables` | `.get(key)`, `.require(key)`, `.variables`, `.get_variables(...)` | Arbitrary key/value pairs — the main carrier for stage parameters. All YAML keys not named `datasets` or `metadata` end up here. |
| `datasets` | `stage_config.datasets` | Mapping of dataset identifiers to their properties (e.g. file path, format). |

configuration.md:40

• output_dir is now enforced by Pipeline._set_run_output: when present, runs are written under output_dir/runs/<run_id>. The current description gives users the wrong output-location semantics.

| `output_dir` | `Path \| None` | `None` | Conventional location for pipeline outputs. Not enforced by the runner; available to stages via `context.config.output_dir`. |

examples/pipeline_2/scripts/2_reporting.py:126

• This stage takes no execution context and hard-codes both configured paths, so changes to stage_configuration.2_reporting.input_location or report_location have no effect. Accept context, read the stage configuration, and pass those paths through to read_csv and write_report.

def main():
    orders = pd.read_csv("examples/pipeline_2/data/orders_prepped.csv")

README.md:40

• This requirement contradicts the package metadata (python_requires >=3.10) and the CI matrix, which supports Python 3.10–3.14. Requiring exactly 3.14.6 would incorrectly exclude supported installations.

- Python 3.14.6 installed

configuration.md:19

• These configuration dataclasses are not immutable: Pipeline mutates config.name, and stage code receives mutable objects through ExecutionContext. Document them as mutable (or make the dataclasses frozen and stop mutating them) so callers do not rely on a guarantee the implementation does not provide.

Both objects are constructed during `Pipeline` initialisation and are immutable
for the duration of a run. They are kept separate so that pipeline orchestration

configuration.md:338

• This diagram hard-codes project_root/runs/<run_id>, but the runner now prefers output_dir/runs/<run_id> whenever output_dir is configured. Update the path so the execution-flow documentation matches _set_run_output.

  context.run_dir                   # project_root/runs/<run_id>

configuration.md:354

• This table repeats the obsolete project_root-only run path. context.run_dir is rooted at output_dir when configured, otherwise at project_root/work_dir.

| `context.run_dir` | `Path` | `project_root / "runs" / run_id` |

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Warning

• Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Pull request overview

Copilot reviewed 47 out of 53 changed files in this pull request and generated no new comments.

Files excluded by content exclusion policy (1)

• examples/pipeline_1/logs/onsrap.log

Suppressed comments (16)

onsrap/runner.py:278

• _pipeline_run_to_dict() only encodes stage_results and stage_outputs; its manifest remains raw. Since the runner copies every result.outputs value into manifest.outputs (lines 143-144), a valid Path, set, or custom output still makes this safe_dump raise after execution. Encode the complete serialized run (or at least the manifest) before dumping it.
  onsrap/runner.py:338
• Empty mappings and lists emit no flattened entry, so adding/removing an empty configuration section (or changing {} to []) is reported as no difference. Preserve empty containers as leaf values so configuration diffs include these structural changes.
  configuration.md:355
• This path is incorrect whenever output_dir is configured. The runner sets context.run_dir to output_dir / "runs" / run_id, falling back to project_root or work_dir only when no output directory is supplied.

| `context.run_dir` | `Path` | `project_root / "runs" / run_id` |

onsrap/init.py:16

• The public RAPConfig export is removed while the package remains on the 0.1.1 patch version. Existing from onsrap import RAPConfig consumers will fail immediately; retain a compatibility alias/deprecation path or release this as an explicitly documented breaking version.
  examples/pipeline_2/scripts/2_reporting.py:126
• The reporting stage ignores its configured input_location and reads a repository-relative constant instead. This makes the configuration example fail when the input is relocated or the process runs from another working directory; accept the execution context and read the stage configuration as the earlier stages do.

def main():
    orders = pd.read_csv("examples/pipeline_2/data/orders_prepped.csv")

configuration.md:424

• This repeated “known pitfall” is also inaccurate: global configuration keys are extracted, and truly unrecognized top-level keys trigger PipelineConfigurationWarning rather than being silently discarded. Update this section to match _split_config_sections().

### 4 — Keys outside `pipeline_variables` in a composite YAML are ignored

When the composite format is detected, `_split_config_sections()` reads only
`pipeline_variables` and `stage_configuration` / `stage_config`. Any other top-level
key in the YAML file is silently discarded.

onsrap/loader.py:189

• Malformed or schema-incompatible historical YAML can raise yaml.YAMLError, TypeError, KeyError, or ValueError here, while the automatic history loaders only catch StageLoadError. Consequently, one corrupt run file can prevent a new Pipeline from being constructed. Validate the loaded mapping and wrap parse/deserialization failures in StageLoadError.
  README.md:40
• This requirement contradicts python_requires >=3.10 and the CI matrix, which supports Python 3.10–3.14. Stating that exactly Python 3.14.6 is required incorrectly excludes supported installations; document Python 3.10 or newer instead.

- Python 3.14.6 installed

configuration.md:59

• StageConfig has no datasets attribute, and from_mapping() currently leaves the datasets key inside _variables (as the new architecture test also asserts). Following this table would raise AttributeError; document datasets as a normal variable or implement the advertised attribute.

| `_variables` | `.get(key)`, `.require(key)`, `.variables`, `.get_variables(...)` | Arbitrary key/value pairs — the main carrier for stage parameters. All YAML keys not named `datasets` or `metadata` end up here. |
| `datasets` | `stage_config.datasets` | Mapping of dataset identifiers to their properties (e.g. file path, format). |

configuration.md:164

• Top-level global configuration is not ignored: _split_config_sections() recognizes global_configuration, global_config, global_variables, and global_vars, while other remaining sections emit a warning rather than being silently discarded. Include the global section in this format description and correct the behavior for unknown keys.

This issue also appears on line 420 of the same file.

In this format:
- Everything under `pipeline_variables` becomes `PipelineConfig`.
- Everything under `stage_configuration` becomes the `stage_configs` mapping.
- Any key at the top level outside these two sections is **ignored**.

configuration.md:170

• The documented flat format is not accepted by the implementation. _split_config_sections() always calls _extract_mappings() for pipeline_variables/pipeline_config, and _extract_keys() raises when neither top-level key exists. Either implement flat-format detection or remove this unsupported usage example.

### Flat format

Used when none of the composite section markers are present. The entire mapping
is treated as pipeline config; an optional nested `stage_configuration` or
`stage_config` key within it carries the stage-level config.

configuration.md:41

• output_dir is enforced by the runner: Pipeline._set_run_output() uses it as the parent of runs/<run_id>. This description can send users to the wrong location; document that output_dir/runs/<run_id> is used, with project_root/work_dir only as the fallback.

This issue also appears on line 355 of the same file.

| `output_dir` | `Path \| None` | `None` | Conventional location for pipeline outputs. Not enforced by the runner; available to stages via `context.config.output_dir`. |

configuration.md:381

• The manifest parameters do not currently contain the full configuration state: PipelineConfig.to_dict() omits at least stages_to_run and overwrite, and _manifest_parameters() is built from that result. Those values affect execution, so the claim of exact reproducibility is false unless they are serialized too.

When a pipeline run completes, `Pipeline._manifest_parameters()` serialises the
full configuration state into `RunManifest.parameters`. This includes the
`PipelineConfig` fields and a nested `stage_configuration` block containing every
`StageConfig.to_dict()`. This means the exact parameters used for any given run
are reproducible from the manifest alone.

examples/pipeline_2/scripts/2_reporting.py:120

• This hard-coded destination ignores stage_configuration.2_reporting.report_location, so changing the configuration has no effect and every run overwrites the same repository file. Pass the configured report path (or a run-scoped path from the context) into write_report.

This issue also appears on line 125 of the same file.

    report_file = Path("examples/pipeline_2/outputs/order_analysis.md")

examples/pipeline_2/scripts/0_clean_data.py:13

• The missing-variable message is printed unconditionally, so valid input reports both “All variables present” and “Missing the following variables: []”. Put the second message in an else branch (or return after the success message).

    if missing == []:
        print("All variables present")
    print(f"Missing the following variables: {missing}")

onsrap/logger.py:183

• These adjacent literals concatenate to “does not write to afilepath”, producing a malformed user-facing error. Add spaces at the literal boundary (for example, “a filepath. ”).