diff --git a/.github/workflows/signaloid-python.yaml b/.github/workflows/signaloid-python.yaml index 61c881a..7a1893a 100644 --- a/.github/workflows/signaloid-python.yaml +++ b/.github/workflows/signaloid-python.yaml @@ -47,6 +47,24 @@ jobs: echo "github.event.release.tag_name:" ${{ github.event.release.tag_name }} echo "contains(github.event.pull_request.labels.*.name, 'ci:upload-binaries'):" ${{ contains(github.event.pull_request.labels.*.name, 'ci:upload-binaries') }} + - name: Check submodules are up to date + # Independent of the matrix, so run it once. The fetch below reuses the + # credentials actions/checkout persisted into each submodule, so this + # works whether the assets remote is private or public. + if: ${{ matrix.args.host == 'ubuntu-22.04' && matrix.python-version == '3.10' }} + run: | + git submodule foreach --quiet --recursive ' + branch=$(git config -f "$toplevel/.gitmodules" "submodule.$name.branch" || echo HEAD) + git fetch --quiet origin "$branch" + pinned=$(git rev-parse HEAD) + latest=$(git rev-parse FETCH_HEAD) + if [ "$pinned" != "$latest" ]; then + echo "::error::Submodule $sm_path is behind origin/$branch. Pinned $pinned, latest $latest. Update it with: git submodule update --remote $sm_path && git add $sm_path" + exit 1 + fi + echo "Submodule $sm_path is up to date with origin/$branch ($pinned)" + ' + - name: Set up Python uses: actions/setup-python@v6 with: diff --git a/README.md b/README.md index 4b1f629..716fc89 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,6 @@ benchmarks for UxHw Core microarchitectures Athens and Jupiter for precisions 8, python -m signaloid.benchmarking.automation \ --path-to-application ./my-uxhw-app \ --path-to-uxhw-sdk ~/project-uxhw-sdk \ - --path-to-pin ~/pin-external-4.2 \ -u Athens Jupiter \ -s 8 16 32 \ -c Disabled Autocorrelation \ @@ -53,7 +52,10 @@ python -m signaloid.benchmarking.automation \ ``` The tool needs access to the Signaloid UxHw SDK to build the applications for -UxHw, and access to the Intel Pin tool for accurate benchmarking. Arguments +UxHw. The Intel Pin tool is optional and off by default. Export `PIN_ROOT` and +pass `--measure-dynamic-instructions` to also measure the dynamic instruction +count. Without that flag the run never uses Pin, even when `PIN_ROOT` is set, +and reports the count as missing. Arguments `-u/--representation-types`, `-s/--representation-sizes`, `-c/--uncertainty-correlation_types`, `-r/--reporting-methods` can also be supplied using a YAML file with `--config `. @@ -81,11 +83,13 @@ dist_value = DistributionalValue.parse(ux_binary_buffer) ### Create Distribution Plots Create plots to visualize distributional information by using the [`plot` function](./src/signaloid/distributional_information_plotting/plot_wrapper.py) -with a `DistributionalValue` object containing Ux Data. The `plot` function is a -wrapper function for the `PlotHistogramDiracDeltas` class for plotting a -distributional value as a histogram with variable bin widths. +with a `PlotData` object built from a `DistributionalValue` containing Ux Data. +The `plot` function is a wrapper function for the `PlotHistogramDiracDeltas` +class for plotting a distributional value as a histogram with variable bin +widths. ```python +from signaloid.distributional_information_plotting.plot_histogram_dirac_deltas import PlotData from signaloid.distributional_information_plotting.plot_wrapper import plot # Intermediate code which writes to ux_string @@ -93,5 +97,28 @@ from signaloid.distributional_information_plotting.plot_wrapper import plot # Create distributional value object from Ux String dist_value = DistributionalValue.parse(ux_string) -plot(dist_value) +plot(PlotData(dist_value)) ``` + +For plotting from raw samples, saving to a file, and the other `plot` options, +see the package [README.md](src/signaloid/distributional_information_plotting/README.md). + +### Sample from Ux Data +Draw random samples from a distributional value with the +[`sample_generator` function](./src/signaloid/distributional_information_plotting/sample_generator.py). +Samples of the finite part of the distribution are drawn by inverse transform +sampling of the binned distribution. Distributions that also carry non-finite +mass (`NaN`, `-Inf`, `+Inf`) are sampled as a mixture, with each sample drawn +from the finite or the non-finite part in proportion to their masses. + +```python +from signaloid.distributional_information_plotting.sample_generator import sample_generator + +# Intermediate code which writes to ux_string +# ... + +samples = sample_generator(ux_string, n_samples=1000) +``` + +To sample from a `DistributionalValue` that is already parsed, use +`sample_from_distributional_value` from the same module. diff --git a/pyproject.toml b/pyproject.toml index 3d81f49..42e76b4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -41,7 +41,7 @@ requires = ["poetry-core>=1.9.1", "poetry-dynamic-versioning>=1.0.0,<2.0.0"] build-backend = "poetry_dynamic_versioning.backend" [tool.poetry.dependencies] -python = ">=3.10" +python = ">=3.10,<4" numpy = [ { version = ">=2.0.0", python = ">=3.10,<3.13" }, { version = ">=2.1.0", python = ">=3.10,<3.14" }, diff --git a/src/signaloid/benchmarking/assets b/src/signaloid/benchmarking/assets index e59a566..15baa38 160000 --- a/src/signaloid/benchmarking/assets +++ b/src/signaloid/benchmarking/assets @@ -1 +1 @@ -Subproject commit e59a5667bc6e58e17f28458302eadeacc373b919 +Subproject commit 15baa381fbe25bc33c50fa5ee2701f9e0648efb4 diff --git a/src/signaloid/benchmarking/automation/README.md b/src/signaloid/benchmarking/automation/README.md index 1aef61e..6c81649 100644 --- a/src/signaloid/benchmarking/automation/README.md +++ b/src/signaloid/benchmarking/automation/README.md @@ -18,11 +18,10 @@ third-party dependencies. ### System dependencies - gcc or g++ - GNU Scientific Library (on Ubuntu: libgsl-dev). -- Python (3.10+; see root-level pyproject.toml) +- Python (3.10+, see root-level pyproject.toml) - GNU Make (make) - bash - lscpu -- hyperfine C and C++ files are compiled separately and linked with `c++`. Applications that link against GSL need `-lgsl -lgslcblas -lm`. @@ -34,6 +33,8 @@ virtual environment, then install the package with pip: python -m venv .venv source .venv/bin/activate pip install . +# Or, to also get the optional Google Sheets upload: +pip install ".[sheets]" ``` For development, you can install in editable mode with `pip install -e .` instead. @@ -87,11 +88,21 @@ See the [Signaloid documentation for more details about using GitHub repositorie with UxHw](https://docs.signaloid.io/docs/api/guides/builds/builds-repository/). -### Intel PIN +### Intel PIN (Optional) -Intel PIN is necessary for the dynamic instruction count on any timing run. The -tool uses it via command-line argument `--path-to-pin` or environment variable -`PIN_ROOT`. A timing run fails if neither is set. +Intel PIN adds a dynamic instruction count to a timing run. It is optional and +off by default. The tool turns it on only when you pass +`--measure-dynamic-instructions`. The kit location comes from the `PIN_ROOT` +environment variable. + +That flag is the single switch. A run without it never uses PIN, even when +`PIN_ROOT` is set in your shell. So you do not need to unset anything to turn +the measurement off. + +Passing the flag is an explicit opt-in, so an unset `PIN_ROOT`, or a kit that +is missing or not built, is a hard error. Without the flag a timing run still +completes. It logs that it skipped the count and reports `pinDynInstCount` as +missing. The dynamic instruction count (`pinDynInstCount`) is from the `inscount0` tool. @@ -100,8 +111,11 @@ Basic installation instructions: [Pin binary-instrumentation tool downloads](https://www.intel.com/content/www/us/en/developer/articles/tool/pin-a-binary-instrumentation-tool-downloads.html) (validated against PIN 4.2). 2. Extract it to a directory `` of your choice. -3. Set parameter `--path-to-pin ` (or `export PIN_ROOT=`). -3. Build the `inscount0` counter once (the kit does not ship it prebuilt): +3. Build the `inscount0` counter once. The kit does not ship it prebuilt. + ``` + make -C /source/tools/ManualExamples obj-intel64/inscount0.so + ``` +4. `export PIN_ROOT=`, then run with `--measure-dynamic-instructions`. ### Google Sheets (Optional) @@ -116,7 +130,9 @@ To enable it you need: pip install ".[sheets]" ``` A plain `pip install .` does **not** pull in the Sheets stack (`gspread`, - `google-api-python-client`, `oauth2client`). + `google-api-python-client`, `oauth2client`). Passing `--write-sheets` + without it raises `RuntimeError` at startup (before the pipeline runs), + naming the modules it could not find. 2. A Google Cloud service-account credentials JSON file, supplied via `--google-credentials ` or the `GOOGLE_APPLICATION_CREDENTIALS` environment variable (whichever resolves to an existing file). There is no @@ -188,7 +204,7 @@ signaloid-benchmarking [OPTIONS] | Flag | Default | Description | |---|---|---| | `--path-to-uxhw-sdk` | `~/project-uxhw-sdk` | Path to the UxHw SDK. | -| `--path-to-pin` | `None` (uses `PIN_ROOT` env) | Path to the Intel PIN kit, exported as `PIN_ROOT` for the timing script's dynamic instruction count. Omit to keep any existing `PIN_ROOT`. A timing run fails clearly if neither `--path-to-pin` nor `PIN_ROOT` is set. | +| `--measure-dynamic-instructions` | `False` | Also measure dynamic instruction counts with Intel PIN, using the kit that `PIN_ROOT` points at. This flag is the only way to switch the measurement on, so a run without it never invokes PIN even when `PIN_ROOT` is set. Passing it without a usable `PIN_ROOT` is an error. | | `--demo-cli-args` | `""` | Extra command-line arguments passed to both native-MC and UxHw executions. Use this when the demo application requires additional flags (e.g., `--demo-cli-args "--asc-file inputs/blink.asc"`). | | `--ground-truth-size` | `1` | Number of Monte Carlo samples for ground truth generation. | | `--ground-truth-type` | `MonteCarlo` | Type of ground truth: `MonteCarlo` or `WeightedSamples`. | @@ -306,12 +322,12 @@ output): 4. **UxHw Tracing Database** — Runs the application through the UxHw tracing pipeline to produce UxHw distributional outputs for each representation type/size/correlation combination. If a tracing database already exists, the - script will prompt before overwriting. The tracing build uses `-O0` (so the + script will prompt before overwriting. The tracing build uses `-O0`, so the `addDistValueTrace` `file:line` directives resolve against unoptimised debug - info); to guard against optimisation changing the traced values, each config - is also built at `-O2` and its Ux strings are checked (byte-for-byte) against - the `-O0` ones. Any difference is reported as a warning and does not stop the - run. If a config's `-O2` build or run fails, that config is skipped and + info. Optimisation could change the traced values. To guard against that, + each config is also built at `-O2` and its Ux strings are checked + (byte-for-byte) against the `-O0` ones. Any difference is reported as a + warning and does not stop the run. If a config's `-O2` build or run fails, that config is skipped and reported as failing in an `-O2 ux-string verification FAILED` summary (the run still continues). Set `TRACING_VERIFY_OPTFLAGS` to compare against a different level. @@ -325,8 +341,8 @@ output): validation. 9. **Equivalent Monte Carlo** — Computes the true EMCC by comparing adversary MC distances to UxHw distances. -10. **UxHw Timings** — Measures execution time and dynamic instruction counts - for each UxHw configuration. +10. **UxHw Timings**. Measures execution time for each UxHw configuration. It + also measures dynamic instruction counts when Intel PIN is enabled. 11. **Native MC Timings** — Measures execution time for native MC at each EMCC sample size. 12. **Load Measurements** — Loads all measurement data from the timing file. @@ -405,7 +421,8 @@ underlying bash scripts used by the pipeline: (not executed) by the Python tool with pre-set environment variables. It handles UxHw compilation, native MC benchmarking, UxHw tracing, and timing collection (the UxHw cores are compiled and timed via UxHw. The dynamic - instruction count comes from Intel PIN). Compilation warnings are redirected + instruction count comes from Intel PIN when PIN is enabled, and is left + unset otherwise). Compilation warnings are redirected to log files (`uxhw-build.log` and `native-mc-build.log` in the `logs/` directory). Source files are discovered recursively (excluding `build/` directories), and C++ files (`.cc`, `.cpp`) are automatically included when @@ -476,6 +493,8 @@ Notes on the schema: - Missing numeric fields (e.g. `dbTime` on native runs) are serialised as `null`. - `dbDynInstCount` for UxHw rows is `0`. +- `pinDynInstCount` is `null` when the run had no Intel PIN kit configured. + Intel PIN is optional and off by default, so this is the common case. - `uxhwTargetRepetitions` is the UxHw-loop target at session start. The actual rep count used per measurement can differ — native-MC rows use `NATIVE_MC_REPETITION` (dynamically computed per precision), and the UxHw diff --git a/src/signaloid/benchmarking/automation/arguments.py b/src/signaloid/benchmarking/automation/arguments.py index ab24ce9..366a7e7 100644 --- a/src/signaloid/benchmarking/automation/arguments.py +++ b/src/signaloid/benchmarking/automation/arguments.py @@ -273,15 +273,15 @@ def create_argument_parser() -> ArgumentParser: ) parser.add_argument( - "--path-to-pin", - dest="path_to_pin", - type=str, - default=None, + "--measure-dynamic-instructions", + dest="measure_dynamic_instructions", + action="store_true", help=( - "Path to the Intel PIN kit (sets PIN_ROOT for the timing " - "script, which uses it to count dynamic instructions). When " - "omitted, an inherited PIN_ROOT is used. If neither is set " - "the timing run errors with 'Intel Pin not found'." + "Also measure dynamic instruction counts, using the Intel PIN " + "kit that PIN_ROOT points at. Off by default. This flag is the " + "only way to switch the measurement on, so a run without it " + "never invokes PIN even when PIN_ROOT is set. Passing it " + "without a usable PIN_ROOT is an error." ), ) diff --git a/src/signaloid/benchmarking/automation/benchmark.py b/src/signaloid/benchmarking/automation/benchmark.py index e2abacb..dec77d6 100644 --- a/src/signaloid/benchmarking/automation/benchmark.py +++ b/src/signaloid/benchmarking/automation/benchmark.py @@ -78,7 +78,7 @@ def __init__( self, path_to_application: str, path_to_uxhw_sdk: str = DEFAULT_UXHW_SDK_PATH, - path_to_pin: str | None = None, + measure_dynamic_instructions: bool = False, has_analytic_ground_truth: bool = False, path_to_ground_truth_file: str = "", ground_truth_size: int = DEFAULT_GROUND_TRUTH_SIZE, @@ -116,7 +116,7 @@ def __init__( """ self.path_to_application = os.path.expanduser(path_to_application) self.path_to_uxhw_sdk = os.path.expanduser(path_to_uxhw_sdk) - self.path_to_pin = os.path.expanduser(path_to_pin) if path_to_pin else None + self.measure_dynamic_instructions = measure_dynamic_instructions self.has_analytic_ground_truth = has_analytic_ground_truth if self.has_analytic_ground_truth: self.path_to_ground_truth_file = os.path.expanduser( @@ -410,7 +410,7 @@ def get_application_info(self) -> None: # Export common timing environment variables self.tracing_db_path = export_timing_env( path_to_uxhw_sdk=self.path_to_uxhw_sdk, - path_to_pin=self.path_to_pin, + measure_dynamic_instructions=self.measure_dynamic_instructions, path_to_application=self.path_to_application, application_name=self.application_name, application_version=self.application_version, diff --git a/src/signaloid/benchmarking/automation/benchmark_application.py b/src/signaloid/benchmarking/automation/benchmark_application.py index 9aa01c5..5affa4c 100644 --- a/src/signaloid/benchmarking/automation/benchmark_application.py +++ b/src/signaloid/benchmarking/automation/benchmark_application.py @@ -19,6 +19,7 @@ # DEALINGS IN THE SOFTWARE. import datetime +import importlib.util import os import sys import traceback @@ -62,6 +63,11 @@ TOTAL_STEPS = 15 LOG_FILE_PREFIX = "benchmarking_automation_error" +# Top-level modules the Google Sheets upload imports (see report_writer's +# write_results_to_spreadsheet and _get_credentials). They ship in the +# optional `sheets` extra, so a plain `pip install .` leaves them absent. +SHEETS_MODULE_NAMES = ("gspread", "googleapiclient", "oauth2client") + def _use_color() -> bool: if os.environ.get("NO_COLOR"): @@ -91,12 +97,28 @@ def _resolve_sheets_credentials(args: Namespace) -> str | None: ``None`` when the user opted out. Raises: - RuntimeError: If ``--write-sheets`` is set but no credentials file - resolves on disk, or the Drive folder / Sheets template IDs are - unset. + RuntimeError: If ``--write-sheets`` is set but the ``sheets`` extra + is not installed, no credentials file resolves on disk, or the + Drive folder / Sheets template IDs are unset. """ if not args.write_sheets: return None + # The upload's dependencies are imported lazily inside + # write_results_to_spreadsheet, so without this check a missing `sheets` + # extra only surfaces at step 15, once the whole pipeline has run. + # find_spec locates the modules without importing them, keeping the + # opted-out path free of the extra's import cost. + missing_module_names = [ + module_name + for module_name in SHEETS_MODULE_NAMES + if importlib.util.find_spec(module_name) is None + ] + if missing_module_names: + raise RuntimeError( + "--write-sheets requires the `sheets` extra, but " + f"{', '.join(missing_module_names)} could not be found. " + 'Install it with: pip install ".[sheets]"' + ) credentials_path = resolve_google_credentials_path( google_credentials=args.google_credentials, ) @@ -248,7 +270,7 @@ def _run_pipeline(args: Namespace, *, credentials_path: str | None) -> None: benchmark = Benchmark( path_to_application=args.path_to_application, path_to_uxhw_sdk=args.path_to_uxhw_sdk, - path_to_pin=args.path_to_pin, + measure_dynamic_instructions=(args.measure_dynamic_instructions), has_analytic_ground_truth=(args.has_analytic_ground_truth), path_to_ground_truth_file=(args.path_to_ground_truth_file), ground_truth_type=args.ground_truth_type, diff --git a/src/signaloid/benchmarking/automation/build.py b/src/signaloid/benchmarking/automation/build.py index 11e560f..00ff180 100644 --- a/src/signaloid/benchmarking/automation/build.py +++ b/src/signaloid/benchmarking/automation/build.py @@ -27,15 +27,15 @@ from signaloid.benchmarking.config import ( EquivMC, TimingFormat, - get_repo_root, get_resources_dir, + get_timing_script, ) def export_timing_env( *, path_to_uxhw_sdk: str, - path_to_pin: str | None, + measure_dynamic_instructions: bool, path_to_application: str, application_name: str, application_version: str, @@ -53,9 +53,11 @@ def export_timing_env( Args: path_to_uxhw_sdk: Path to the UxHw SDK. - path_to_pin: Path to the Intel PIN kit, exported as ``PIN_ROOT``. - ``None`` leaves any inherited ``PIN_ROOT`` in place. PIN is - mandatory, so the run errors if neither is set. + measure_dynamic_instructions: Whether to measure dynamic instruction + counts with Intel PIN. Off by default. When True the kit is + taken from ``PIN_ROOT``, which must already be set. When False + ``PIN_ROOT`` is removed from the environment, so an inherited + value cannot switch the measurement on behind the caller's back. path_to_application: Path to the application source tree. application_name: Application name (e.g. ``Finance-...``). application_version: Application version string. @@ -70,14 +72,24 @@ def export_timing_env( ``TRACING_DB_ABS``). Callers should store it back to ``self.tracing_db_path``. """ - os.environ["SIGNALOID_PYTHON_DIR"] = os.fspath(get_repo_root()) os.environ["BENCHMARKING_RESOURCES_DIR"] = str(get_resources_dir()) os.environ["PATH_TO_UXHW_SDK"] = path_to_uxhw_sdk - # PIN is mandatory: when no path is given we leave any shell-set - # PIN_ROOT in place. There is no built-in default. get-timings.sh - # will error clearly if neither is set. - if path_to_pin: - os.environ["PIN_ROOT"] = path_to_pin + # The flag is the single switch for the dynamic instruction count. + # Without it we drop PIN_ROOT, so a kit exported in the caller's shell + # profile cannot silently turn the measurement on. get-timings.sh reads + # the same variable and skips the count when it is absent. + if measure_dynamic_instructions: + pin_root = os.environ.get("PIN_ROOT", "") + if not pin_root: + raise ValueError( + "--measure-dynamic-instructions needs an Intel PIN kit, but " + "PIN_ROOT is not set. Export it to the kit directory, for " + "example 'export PIN_ROOT=~/pin-external-4.2', or drop the " + "flag to skip the dynamic instruction count." + ) + os.environ["PIN_ROOT"] = os.path.expanduser(pin_root) + else: + os.environ.pop("PIN_ROOT", None) # The timing bash layer shells out to `python3 -m # signaloid.benchmarking...`. We pass our own interpreter so those # subprocesses use this venv (with the benchmarking dependencies) even when @@ -200,6 +212,8 @@ def run_timing_script( if len(native_mc_sizes) == 0: native_mc_sizes = "50" + quoted_timing_script = shlex.quote(str(get_timing_script())) + bash_cmd = f""" TRACES=( {traces_lines}) @@ -207,7 +221,7 @@ def run_timing_script( REPRESENTATION_TYPES=({representations}) REPRESENTATION_SIZES=({rep_sizes}) CORRELATION_TRACKING_TYPES=({correlations_str}) -. $SIGNALOID_PYTHON_DIR/src/signaloid/benchmarking/benchmark_timing/get-timings.sh +. {quoted_timing_script} """ stderr_log = os.path.join(logs_dir, "timing_script_stderr.log") # Use tee so stderr streams to terminal diff --git a/src/signaloid/benchmarking/automation/config_layer_test.py b/src/signaloid/benchmarking/automation/config_layer_test.py index cac17c6..c37b08b 100644 --- a/src/signaloid/benchmarking/automation/config_layer_test.py +++ b/src/signaloid/benchmarking/automation/config_layer_test.py @@ -95,6 +95,40 @@ def test_cli_overrides_config_value(self) -> None: self.assertEqual(args.reporting_methods, ["Quantile-95"]) self.assertEqual(args.representation_types, ["Athens"]) + def test_config_enables_dynamic_instructions(self) -> None: + # measure_dynamic_instructions is an argparse dest, so it is a valid + # config key. A YAML sweep can turn the measurement on without the + # flag being typed on the command line. + config = _base_config() + config["measure_dynamic_instructions"] = True + config_path = _write_config(self.tmp_path, config) + self._set_argv(["prog", "--config", config_path]) + + args = load_config(create_argument_parser()) + + self.assertTrue(args.measure_dynamic_instructions) + + def test_config_defaults_dynamic_instructions_off(self) -> None: + config_path = _write_config(self.tmp_path, _base_config()) + self._set_argv(["prog", "--config", config_path]) + + args = load_config(create_argument_parser()) + + self.assertFalse(args.measure_dynamic_instructions) + + def test_stale_path_to_pin_config_key_rejected(self) -> None: + # --path-to-pin was removed. An old config still carrying it must + # fail loudly rather than being silently ignored, which would leave + # the user thinking PIN is still configured. + config = _base_config() + config["path_to_pin"] = "~/pin-external-4.2" + config_path = _write_config(self.tmp_path, config) + self._set_argv(["prog", "--config", config_path]) + + with self.assertRaises(ValueError) as caught: + load_config(create_argument_parser()) + self.assertIn("path_to_pin", str(caught.exception)) + def test_config_sets_non_sweep_scalar(self) -> None: # Config keys are dest names: --num-adversaries has dest # n_adversaries, so the config key is n_adversaries (not the flag diff --git a/src/signaloid/benchmarking/automation/path_to_pin_test.py b/src/signaloid/benchmarking/automation/measure_dynamic_instructions_test.py similarity index 52% rename from src/signaloid/benchmarking/automation/path_to_pin_test.py rename to src/signaloid/benchmarking/automation/measure_dynamic_instructions_test.py index 9c3eb2c..8746ca0 100644 --- a/src/signaloid/benchmarking/automation/path_to_pin_test.py +++ b/src/signaloid/benchmarking/automation/measure_dynamic_instructions_test.py @@ -23,7 +23,6 @@ import unittest from pathlib import Path -from typing import Optional from signaloid.benchmarking.automation.arguments import ( create_argument_parser, @@ -50,11 +49,11 @@ def _base_argv() -> list[str]: ] -def _export_kwargs(tmp_path: Path, path_to_pin: Optional[str]) -> dict: +def _export_kwargs(tmp_path: Path, measure_dynamic_instructions: bool) -> dict: """Minimal valid kwargs for ``export_timing_env``.""" return dict( path_to_uxhw_sdk="~/project-uxhw-sdk", - path_to_pin=path_to_pin, + measure_dynamic_instructions=measure_dynamic_instructions, path_to_application=str(tmp_path), application_name="demo", application_version="abc1234", @@ -65,8 +64,13 @@ def _export_kwargs(tmp_path: Path, path_to_pin: Optional[str]) -> dict: ) -class TestPathToPin(unittest.TestCase): - """``--path-to-pin`` parsing and its ``PIN_ROOT`` export side effect.""" +class TestMeasureDynamicInstructions(unittest.TestCase): + """``--measure-dynamic-instructions`` parsing and its ``PIN_ROOT`` effect. + + The flag is the single switch for the dynamic instruction count. These + tests pin the property the design rests on, which is that a run without + the flag never measures, whatever ``PIN_ROOT`` holds. + """ def setUp(self) -> None: # Snapshot and restore ``os.environ`` so ``export_timing_env`` side @@ -83,25 +87,56 @@ def _restore_environ() -> None: self.addCleanup(tmp_dir.cleanup) self.tmp_path = Path(tmp_dir.name) - def test_parser_accepts_path_to_pin(self) -> None: + def test_parser_defaults_to_off(self) -> None: parser = create_argument_parser() - args = parser.parse_args(_base_argv() + ["--path-to-pin", "/opt/pin-test"]) - self.assertEqual(args.path_to_pin, "/opt/pin-test") + args = parser.parse_args(_base_argv()) + self.assertFalse(args.measure_dynamic_instructions) - def test_parser_path_to_pin_defaults_to_none(self) -> None: + def test_parser_accepts_flag(self) -> None: parser = create_argument_parser() - args = parser.parse_args(_base_argv()) - self.assertIsNone(args.path_to_pin) + args = parser.parse_args(_base_argv() + ["--measure-dynamic-instructions"]) + self.assertTrue(args.measure_dynamic_instructions) + + def test_parser_rejects_removed_path_to_pin(self) -> None: + # --path-to-pin is gone. Argparse must reject it rather than + # silently ignoring a stale invocation. + parser = create_argument_parser() + with self.assertRaises(SystemExit): + parser.parse_args(_base_argv() + ["--path-to-pin", "/opt/pin"]) - def test_export_timing_env_exports_pin_root(self) -> None: - export_timing_env(**_export_kwargs(self.tmp_path, "/opt/pin-test")) + def test_flag_keeps_pin_root(self) -> None: + os.environ["PIN_ROOT"] = "/opt/pin-test" + export_timing_env(**_export_kwargs(self.tmp_path, True)) self.assertEqual(os.environ["PIN_ROOT"], "/opt/pin-test") - def test_export_timing_env_omits_pin_root_when_unset(self) -> None: - # PIN is mandatory and overridable: with no --path-to-pin we must not - # touch PIN_ROOT, so get-timings.sh applies its built-in fallback. + def test_flag_expands_user_in_pin_root(self) -> None: + os.environ["PIN_ROOT"] = "~/pin-kit" + export_timing_env(**_export_kwargs(self.tmp_path, True)) + self.assertEqual(os.environ["PIN_ROOT"], os.path.expanduser("~/pin-kit")) + + def test_flag_without_pin_root_raises(self) -> None: + os.environ.pop("PIN_ROOT", None) + with self.assertRaises(ValueError) as caught: + export_timing_env(**_export_kwargs(self.tmp_path, True)) + self.assertIn("PIN_ROOT", str(caught.exception)) + + def test_flag_with_empty_pin_root_raises(self) -> None: + # Set but empty is not a usable kit. Treat it as unset rather than + # letting the shell look for a kit at "/". + os.environ["PIN_ROOT"] = "" + with self.assertRaises(ValueError): + export_timing_env(**_export_kwargs(self.tmp_path, True)) + + def test_no_flag_drops_inherited_pin_root(self) -> None: + # The whole point of the redesign. A kit exported in the caller's + # shell profile must not switch the measurement on. + os.environ["PIN_ROOT"] = "/opt/pin-test" + export_timing_env(**_export_kwargs(self.tmp_path, False)) + self.assertNotIn("PIN_ROOT", os.environ) + + def test_no_flag_with_no_pin_root_is_fine(self) -> None: os.environ.pop("PIN_ROOT", None) - export_timing_env(**_export_kwargs(self.tmp_path, None)) + export_timing_env(**_export_kwargs(self.tmp_path, False)) self.assertNotIn("PIN_ROOT", os.environ) diff --git a/src/signaloid/benchmarking/automation/measure_process_time.py b/src/signaloid/benchmarking/automation/measure_process_time.py new file mode 100644 index 0000000..0caa735 --- /dev/null +++ b/src/signaloid/benchmarking/automation/measure_process_time.py @@ -0,0 +1,246 @@ +# Copyright (c) 2026, Signaloid. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to +# deal in the Software without restriction, including without limitation the +# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +# sell copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +import argparse +import math +import os +import shlex +import subprocess +import sys +import tempfile +import time +from typing import IO + +# Longest stderr excerpt kept for a failure message. Benchmark binaries can +# be chatty. The useful part is at the end. +_STDERR_EXCERPT_LIMIT = 2000 + + +def scaled_repetitions( + *, + single_time: float, + target_total_time: float, + min_repetitions: int, + max_repetitions: int, +) -> int: + """ + Scale a repetition count so the total run time approaches a target. + + This mirrors the compute_time_scaled_repetitions shell function. + run_uxhw_benchmarks already uses that helper. Both timing paths + therefore size their repetition counts the same way. The rule is to + divide the target by one run's cost, round up, then clamp. + + A non-positive single_time means the run was too fast to measure. + That case falls back to max_repetitions, matching the shell helper. + + Args: + single_time: Measured duration of one run, in seconds. + target_total_time: Total measurement time aimed for, in seconds. + min_repetitions: Lower clamp on the returned count. + max_repetitions: Upper clamp on the returned count. + + Returns: + The repetition count to use. + """ + if single_time <= 0: + return max_repetitions + repetitions = math.ceil(target_total_time / single_time) + return max(min_repetitions, min(repetitions, max_repetitions)) + + +def _read_stderr_tail(stderr_sink: IO[bytes]) -> str: + """ + Read the last few kilobytes written to a stderr sink. + + Only the tail is read back. A chatty benchmark can write far more + than a failure message needs. + + Args: + stderr_sink: The temporary file the child wrote its stderr to. + + Returns: + The decoded tail of the file, at most _STDERR_EXCERPT_LIMIT bytes. + """ + stderr_sink.seek(0, os.SEEK_END) + written = stderr_sink.tell() + stderr_sink.seek(max(0, written - _STDERR_EXCERPT_LIMIT)) + return stderr_sink.read().decode("utf-8", errors="replace") + + +def time_once(*, command: str, use_shell: bool) -> float: + """ + Run a command once and return its wall-clock duration. + + The command's stdout is discarded. Benchmark binaries print their + results there. This helper's own stdout carries the measurement back + to the shell layer. + + Stderr goes to a temporary file rather than a pipe. The child writes + straight to that file descriptor. Nothing accumulates in this + process, so the measurement stays undisturbed. Only the tail is read + back, and only when the command fails. + + Args: + command: The command line to run. + use_shell: Run via the system shell when True. Otherwise split + the command with shlex and execute it directly. + + Returns: + The elapsed wall-clock time in seconds. + + Raises: + RuntimeError: If the command exits non-zero. Timing a failed run + would feed a meaningless number into the report. + """ + arguments: str | list[str] = command if use_shell else shlex.split(command) + with tempfile.TemporaryFile() as stderr_sink: + start = time.perf_counter() + completed = subprocess.run( + arguments, + shell=use_shell, + stdout=subprocess.DEVNULL, + stderr=stderr_sink, + ) + elapsed = time.perf_counter() - start + if completed.returncode != 0: + raise RuntimeError( + f"command exited {completed.returncode}: {command}\n" + f"stderr: {_read_stderr_tail(stderr_sink)}" + ) + return elapsed + + +def measure_mean_time( + *, + command: str, + use_shell: bool, + target_total_time: float, + min_repetitions: int, + max_repetitions: int, +) -> float: + """ + Time a command repeatedly and return the mean wall-clock duration. + + One warmup run is timed first and then discarded. It primes caches. + It also sizes the measured repetition count via scaled_repetitions. + + Args: + command: The command line to benchmark. + use_shell: Whether to run the command through the system shell. + target_total_time: Total measurement time aimed for, in seconds. + min_repetitions: Lower clamp on the repetition count. + max_repetitions: Upper clamp on the repetition count. + + Returns: + The mean elapsed wall-clock time in seconds. + + Raises: + RuntimeError: If any run of the command exits non-zero. + """ + warmup_time = time_once(command=command, use_shell=use_shell) + repetitions = scaled_repetitions( + single_time=warmup_time, + target_total_time=target_total_time, + min_repetitions=min_repetitions, + max_repetitions=max_repetitions, + ) + total = 0.0 + for _ in range(repetitions): + total += time_once(command=command, use_shell=use_shell) + return total / repetitions + + +def main() -> None: + """ + Parse command-line arguments and emit the mean wall-clock time. + + The bash layer calls this through python3 -m + signaloid.benchmarking.automation.measure_process_time. Stdout + carries a single float in seconds. + """ + parser = argparse.ArgumentParser( + description=( + "Time a command over a scaled number of repetitions and print " + "the mean wall-clock duration in seconds." + ), + ) + parser.add_argument( + "command", + help="The command line to benchmark, as a single string.", + ) + parser.add_argument( + "--target-total-time", + type=float, + default=30.0, + help=( + "Total measurement time to aim for, in seconds. Matches the " + "shell layer's TIMING_TARGET_TOTAL_TIME." + ), + ) + parser.add_argument( + "--min-repetitions", + type=int, + default=2, + help="Lower clamp on the repetition count.", + ) + parser.add_argument( + "--max-repetitions", + type=int, + default=20, + help="Upper clamp on the repetition count.", + ) + parser.add_argument( + "--no-shell", + dest="use_shell", + action="store_false", + default=True, + help=( + "Execute the command directly instead of through the system " + "shell. The command is split with shlex." + ), + ) + args = parser.parse_args() + + if args.min_repetitions < 1: + parser.error(f"--min-repetitions must be >= 1, got {args.min_repetitions}") + if args.max_repetitions < args.min_repetitions: + parser.error( + f"--max-repetitions ({args.max_repetitions}) must be >= " + f"--min-repetitions ({args.min_repetitions})" + ) + + print( + measure_mean_time( + command=args.command, + use_shell=args.use_shell, + target_total_time=args.target_total_time, + min_repetitions=args.min_repetitions, + max_repetitions=args.max_repetitions, + ) + ) + + +if __name__ == "__main__": + try: + main() + except (OSError, RuntimeError, ValueError) as exc: + print(f"measure_process_time: {exc}", file=sys.stderr) + sys.exit(1) diff --git a/src/signaloid/benchmarking/automation/measure_process_time_test.py b/src/signaloid/benchmarking/automation/measure_process_time_test.py new file mode 100644 index 0000000..2baef0b --- /dev/null +++ b/src/signaloid/benchmarking/automation/measure_process_time_test.py @@ -0,0 +1,313 @@ +# Copyright (c) 2026, Signaloid. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to +# deal in the Software without restriction, including without limitation the +# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +# sell copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +import os +import shlex +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +from signaloid.benchmarking.automation.measure_process_time import ( + measure_mean_time, + scaled_repetitions, + time_once, +) + +# A command that exits 0 and does nothing else. It is spelled via the running +# interpreter so the tests do not depend on which coreutils are present. +_NOOP_COMMAND = f"{shlex.quote(sys.executable)} -c pass" +_FAILING_COMMAND = f"{shlex.quote(sys.executable)} -c 'raise SystemExit(3)'" + +# A command whose text carries a shell redirection. Under the shell the +# redirection creates the target file. Executed directly the two tokens +# reach the interpreter as ignored extra arguments and no file appears. +_REDIRECT_TEMPLATE = f"{shlex.quote(sys.executable)} -c pass > {{target}}" + + +def _noisy_command(*, byte_count: int) -> str: + """ + Build a command that writes many bytes to stderr and then fails. + + Args: + byte_count: Number of bytes the command writes to stderr. + + Returns: + A command line suitable for time_once. + """ + script = "\n".join( + [ + "import sys", + f"sys.stderr.write('x' * {byte_count})", + "raise SystemExit(4)", + ] + ) + return f"{shlex.quote(sys.executable)} -c {shlex.quote(script)}" + + +class TestScaledRepetitions(unittest.TestCase): + """scaled_repetitions mirrors the compute_time_scaled_repetitions shell + helper. It divides, rounds up, then clamps.""" + + def test_divides_target_by_single_run_cost(self) -> None: + """A 1s run against a 10s target gives 10 repetitions.""" + self.assertEqual( + scaled_repetitions( + single_time=1.0, + target_total_time=10.0, + min_repetitions=2, + max_repetitions=20, + ), + 10, + ) + + def test_rounds_the_quotient_up(self) -> None: + """ + A non-integer quotient rounds up. The target total time is then + met rather than undershot. + """ + self.assertEqual( + scaled_repetitions( + single_time=3.0, + target_total_time=10.0, + min_repetitions=1, + max_repetitions=20, + ), + 4, + ) + + def test_clamps_to_the_maximum_for_a_fast_command(self) -> None: + """A very fast run would demand a huge count, so the max applies.""" + self.assertEqual( + scaled_repetitions( + single_time=0.001, + target_total_time=30.0, + min_repetitions=2, + max_repetitions=20, + ), + 20, + ) + + def test_clamps_to_the_minimum_for_a_slow_command(self) -> None: + """ + A run already longer than the target still repeats + min_repetitions times. A mean is then always taken over more + than one sample. + """ + self.assertEqual( + scaled_repetitions( + single_time=120.0, + target_total_time=30.0, + min_repetitions=2, + max_repetitions=20, + ), + 2, + ) + + def test_non_positive_single_time_falls_back_to_the_maximum(self) -> None: + """ + A run too fast to measure yields a non-positive time. The shell + helper returns max_reps in that case and so does this one. + """ + for single_time in (0.0, -1.0): + with self.subTest(single_time=single_time): + self.assertEqual( + scaled_repetitions( + single_time=single_time, + target_total_time=30.0, + min_repetitions=2, + max_repetitions=20, + ), + 20, + ) + + +class TestExecutionMode(unittest.TestCase): + """time_once picks direct execution or the shell as asked.""" + + def test_no_shell_passes_a_split_argument_list(self) -> None: + """ + Without a shell the command is split by shlex and handed to + subprocess as a list. Quoted whitespace survives as one argument. + """ + with patch("subprocess.run") as run_mock: + run_mock.return_value = subprocess.CompletedProcess([], 0) + time_once(command="prog --flag 'a b'", use_shell=False) + positional, keyword = run_mock.call_args + self.assertEqual(positional[0], ["prog", "--flag", "a b"]) + self.assertFalse(keyword["shell"]) + + def test_shell_passes_the_command_unsplit(self) -> None: + """With a shell the command string is handed over verbatim.""" + with patch("subprocess.run") as run_mock: + run_mock.return_value = subprocess.CompletedProcess([], 0) + time_once(command="prog --flag 'a b'", use_shell=True) + positional, keyword = run_mock.call_args + self.assertEqual(positional[0], "prog --flag 'a b'") + self.assertTrue(keyword["shell"]) + + def test_no_shell_leaves_a_redirection_uninterpreted(self) -> None: + """ + Direct execution must not honour shell metacharacters. The UxHw + call site relies on this. + """ + with tempfile.TemporaryDirectory() as directory: + target = os.path.join(directory, "redirected.txt") + command = _REDIRECT_TEMPLATE.format(target=shlex.quote(target)) + time_once(command=command, use_shell=False) + self.assertFalse(os.path.exists(target)) + + def test_shell_honours_a_redirection(self) -> None: + """ + Shell execution does honour metacharacters. This is the + counterpart that proves the two modes really differ. + """ + with tempfile.TemporaryDirectory() as directory: + target = os.path.join(directory, "redirected.txt") + command = _REDIRECT_TEMPLATE.format(target=shlex.quote(target)) + time_once(command=command, use_shell=True) + self.assertTrue(os.path.exists(target)) + + +class TestTimeOnce(unittest.TestCase): + """time_once measures a single run and rejects failed commands.""" + + def test_returns_a_positive_duration(self) -> None: + """A successful command yields a positive elapsed time.""" + self.assertGreater( + time_once(command=_NOOP_COMMAND, use_shell=False), + 0.0, + ) + + def test_non_zero_exit_raises(self) -> None: + """ + A failed command raises rather than returning a duration. Timing + a run that did not complete would feed a meaningless number into + the report. + """ + with self.assertRaises(RuntimeError) as raised: + time_once(command=_FAILING_COMMAND, use_shell=True) + self.assertIn("exited 3", str(raised.exception)) + + def test_failure_message_keeps_only_the_stderr_tail(self) -> None: + """ + A chatty failing command must not paste its whole stderr into the + message. Only the tail is reported, so the message stays bounded. + """ + noisy_bytes = 50_000 + with self.assertRaises(RuntimeError) as raised: + time_once(command=_noisy_command(byte_count=noisy_bytes), use_shell=False) + message = str(raised.exception) + self.assertIn("exited 4", message) + self.assertLess(len(message), noisy_bytes) + + +class TestMeasureMeanTime(unittest.TestCase): + """measure_mean_time discards a warmup run and averages the rest.""" + + def test_returns_a_positive_mean(self) -> None: + """The mean over the scaled repetitions is a positive duration.""" + mean = measure_mean_time( + command=_NOOP_COMMAND, + use_shell=False, + target_total_time=0.05, + min_repetitions=2, + max_repetitions=3, + ) + self.assertGreater(mean, 0.0) + + def test_runs_warmup_plus_the_scaled_repetitions(self) -> None: + """ + The warmup run is timed and then discarded. With the clamp + pinning the count at 2, that means three executions in total. + """ + with patch( + "signaloid.benchmarking.automation.measure_process_time.time_once", + return_value=1.0, + ) as time_once_mock: + mean = measure_mean_time( + command=_NOOP_COMMAND, + use_shell=False, + target_total_time=1.0, + min_repetitions=2, + max_repetitions=2, + ) + self.assertEqual(time_once_mock.call_count, 3) + self.assertEqual(mean, 1.0) + + def test_propagates_a_failing_command(self) -> None: + """A command that fails on its warmup run raises immediately.""" + with self.assertRaises(RuntimeError): + measure_mean_time( + command=_FAILING_COMMAND, + use_shell=True, + target_total_time=0.05, + min_repetitions=2, + max_repetitions=2, + ) + + +class TestCommandLineInterface(unittest.TestCase): + """The module entry point prints a single float for the shell layer.""" + + def test_prints_a_parseable_float(self) -> None: + """ + Stdout carries only the mean. The bash layer can then capture it + with a plain command substitution. + """ + completed = subprocess.run( + [ + sys.executable, + "-m", + "signaloid.benchmarking.automation.measure_process_time", + "--target-total-time", + "0.05", + "--min-repetitions", + "2", + "--max-repetitions", + "2", + "--no-shell", + _NOOP_COMMAND, + ], + capture_output=True, + text=True, + ) + self.assertEqual(completed.returncode, 0, completed.stderr) + self.assertGreater(float(completed.stdout.strip()), 0.0) + + def test_reports_a_failing_command_on_stderr(self) -> None: + """A failed benchmark exits non-zero with a named cause.""" + completed = subprocess.run( + [ + sys.executable, + "-m", + "signaloid.benchmarking.automation.measure_process_time", + _FAILING_COMMAND, + ], + capture_output=True, + text=True, + ) + self.assertEqual(completed.returncode, 1) + self.assertIn("measure_process_time:", completed.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/src/signaloid/benchmarking/automation/measurement_loader.py b/src/signaloid/benchmarking/automation/measurement_loader.py index 78d9eba..a5be0ce 100644 --- a/src/signaloid/benchmarking/automation/measurement_loader.py +++ b/src/signaloid/benchmarking/automation/measurement_loader.py @@ -217,7 +217,10 @@ def load_measurement_data( if config.startswith("Native-MC"): # Native-MC rows have no database time or database # dynamic instruction count. Ignore those fields. - if None in (time, e2e_time, pin_dyn_inst_count): + # `pin_dyn_inst_count` is not required either: it is None on + # every run made without an Intel PIN kit, which is the + # default. + if None in (time, e2e_time): continue for variable in benchmarking_variables: # Collapse internal whitespace, matching the normalization @@ -233,15 +236,21 @@ def load_measurement_data( ) native_mc_counter += 1 else: - # UxHw path: only ingest rows with all five numeric fields. - # Reference and Native rows carry `?` in some columns and are - # intentionally skipped. + # Reference and Native rows are not UxHw configurations. + # Skip them by name. They used to be filtered out by the + # `pin_dyn_inst_count is None` check below, which no longer + # identifies them now that a PIN-less run leaves that field + # None on genuine UxHw rows too. + if config.startswith("Reference") or config.startswith("Native"): + continue + # UxHw path: only ingest rows with all four numeric fields. + # `pin_dyn_inst_count` is excluded: it is None on every run + # made without an Intel PIN kit, which is the default. if None in ( time, db_time, e2e_time, db_dyn_inst_count, - pin_dyn_inst_count, ): continue for variable in benchmarking_variables: diff --git a/src/signaloid/benchmarking/automation/measurement_loader_test.py b/src/signaloid/benchmarking/automation/measurement_loader_test.py index 9e6b4d2..15dc7bb 100644 --- a/src/signaloid/benchmarking/automation/measurement_loader_test.py +++ b/src/signaloid/benchmarking/automation/measurement_loader_test.py @@ -275,6 +275,123 @@ def test_load_measurement_data_populates_timing_measurements(self) -> None: self.assertEqual(native["End-to-End Time"], 1.5) self.assertEqual(native["PIN Dyn. Inst. Count"], 300.0) + def test_load_measurement_data_ingests_rows_without_pin_counts(self) -> None: + """A run made without Intel PIN still populates every measurement. + + Intel PIN is optional and off by default, so ``pinDynInstCount`` is + None on every row. Those rows must still be ingested. Dropping them + would empty the measurement set and fail the configuration-count + check with a message that never mentions PIN. + """ + variable = _make_variable(name="x", description="x var", cla="-S 0") + variable.emcc_results.equiv_mc_list = [50] + + runs = [ + { + TimingFormat.META_KEY_COMMAND_LINE_ARGUMENTS: "-T -S 0", + TimingFormat.JSON_KEY_MEASUREMENTS: [ + { + TimingFormat.JSON_KEY_MEASUREMENT_CONFIG: ("Athens-16"), + TimingFormat.JSON_KEY_MEASUREMENT_TIME: 1.0, + TimingFormat.JSON_KEY_MEASUREMENT_DB_TIME: 2.0, + TimingFormat.JSON_KEY_MEASUREMENT_E2E_TIME: 3.0, + TimingFormat.JSON_KEY_MEASUREMENT_DB_DYN_INST_COUNT: (100.0), + TimingFormat.JSON_KEY_MEASUREMENT_PIN_DYN_INST_COUNT: (None), + }, + ], + }, + { + TimingFormat.META_KEY_COMMAND_LINE_ARGUMENTS: "-S 0", + TimingFormat.JSON_KEY_MEASUREMENTS: [ + { + TimingFormat.JSON_KEY_MEASUREMENT_CONFIG: "Native-MC-50", + TimingFormat.JSON_KEY_MEASUREMENT_TIME: 0.5, + TimingFormat.JSON_KEY_MEASUREMENT_DB_TIME: None, + TimingFormat.JSON_KEY_MEASUREMENT_E2E_TIME: 1.5, + TimingFormat.JSON_KEY_MEASUREMENT_DB_DYN_INST_COUNT: (None), + TimingFormat.JSON_KEY_MEASUREMENT_PIN_DYN_INST_COUNT: (None), + }, + ], + }, + ] + + load_measurement_data( + benchmarking_variables=[variable], + runs=runs, + demo_cli_args="", + representation_sizes=[16], + representation_types=[RepresentationTypes.ATHENS], + correlations=["Disabled"], + ) + + measurement_dict = variable.timing_measurements.measurement_dict + uxhw = measurement_dict["Athens-16"] + self.assertEqual(uxhw["In Application Time"], 1.0) + self.assertEqual(uxhw["Database Dyn. Inst. Count"], 100.0) + # Recorded as missing rather than as a plausible-looking zero. + self.assertIsNone(uxhw["PIN Dyn. Inst. Count"]) + + native = measurement_dict["Native-MC-50"] + self.assertEqual(native["In Application Time"], 0.5) + self.assertIsNone(native["PIN Dyn. Inst. Count"]) + + def test_load_measurement_data_skips_reference_and_native_rows(self) -> None: + """Reference and Native rows are not UxHw configurations. + + They used to be filtered out because their PIN count was the only + missing field. A PIN-less run leaves that field missing on genuine + UxHw rows too, so they are now skipped by name instead. + """ + variable = _make_variable(name="x", description="x var", cla="-S 0") + + runs = [ + { + TimingFormat.META_KEY_COMMAND_LINE_ARGUMENTS: "-S 0", + TimingFormat.JSON_KEY_MEASUREMENTS: [ + { + TimingFormat.JSON_KEY_MEASUREMENT_CONFIG: ("Athens-16"), + TimingFormat.JSON_KEY_MEASUREMENT_TIME: 1.0, + TimingFormat.JSON_KEY_MEASUREMENT_DB_TIME: 2.0, + TimingFormat.JSON_KEY_MEASUREMENT_E2E_TIME: 3.0, + TimingFormat.JSON_KEY_MEASUREMENT_DB_DYN_INST_COUNT: (100.0), + TimingFormat.JSON_KEY_MEASUREMENT_PIN_DYN_INST_COUNT: (None), + }, + { + TimingFormat.JSON_KEY_MEASUREMENT_CONFIG: ("Reference-50-1"), + TimingFormat.JSON_KEY_MEASUREMENT_TIME: 0.5, + TimingFormat.JSON_KEY_MEASUREMENT_DB_TIME: 1.0, + TimingFormat.JSON_KEY_MEASUREMENT_E2E_TIME: 1.5, + TimingFormat.JSON_KEY_MEASUREMENT_DB_DYN_INST_COUNT: (100.0), + TimingFormat.JSON_KEY_MEASUREMENT_PIN_DYN_INST_COUNT: (None), + }, + # Bare "Native-", not "Native-MC-", so it reaches the UxHw + # branch and exercises the Native half of the name skip. + { + TimingFormat.JSON_KEY_MEASUREMENT_CONFIG: ("Native-50-1"), + TimingFormat.JSON_KEY_MEASUREMENT_TIME: 0.5, + TimingFormat.JSON_KEY_MEASUREMENT_DB_TIME: 1.0, + TimingFormat.JSON_KEY_MEASUREMENT_E2E_TIME: 1.5, + TimingFormat.JSON_KEY_MEASUREMENT_DB_DYN_INST_COUNT: (100.0), + TimingFormat.JSON_KEY_MEASUREMENT_PIN_DYN_INST_COUNT: (None), + }, + ], + }, + ] + + load_measurement_data( + benchmarking_variables=[variable], + runs=runs, + demo_cli_args="", + representation_sizes=[16], + representation_types=[RepresentationTypes.ATHENS], + correlations=["Disabled"], + ) + + measurement_dict = variable.timing_measurements.measurement_dict + self.assertIn("Athens-16", measurement_dict) + self.assertNotIn("Reference-50-1", measurement_dict) + self.assertNotIn("Native-50-1", measurement_dict) + class TestLoadTimingDataToDfs(unittest.TestCase): """Exercise load_timing_data_to_dfs joining timings into emcc_data.""" diff --git a/src/signaloid/benchmarking/automation/sheets_preflight_test.py b/src/signaloid/benchmarking/automation/sheets_preflight_test.py new file mode 100644 index 0000000..8e5f3f2 --- /dev/null +++ b/src/signaloid/benchmarking/automation/sheets_preflight_test.py @@ -0,0 +1,153 @@ +# Copyright (c) 2026, Signaloid. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to +# deal in the Software without restriction, including without limitation the +# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +# sell copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +import importlib.machinery +import importlib.util +import tempfile +import unittest +from argparse import Namespace +from pathlib import Path +from typing import Any +from unittest.mock import patch + +from signaloid.benchmarking.automation import benchmark_application +from signaloid.benchmarking.automation.benchmark_application import ( + SHEETS_MODULE_NAMES, + _resolve_sheets_credentials, +) + + +def _make_args(*, write_sheets: bool, google_credentials: str | None) -> Namespace: + """ + Build the argument subset that _resolve_sheets_credentials reads. + + Args: + write_sheets: Whether the user opted in to the Sheets upload. + google_credentials: Explicit credentials path, or None. + + Returns: + A Namespace carrying just those two attributes. + """ + return Namespace( + write_sheets=write_sheets, + google_credentials=google_credentials, + ) + + +def _find_spec_missing(module_names: set[str]) -> Any: + """ + Build a find_spec replacement with a fixed view of the Sheets stack. + + Every name in ``SHEETS_MODULE_NAMES`` is answered from + ``module_names`` alone, so these tests behave identically whether or + not the `sheets` extra is installed in the running environment. Any + other name is delegated to the real ``importlib.util.find_spec``. + + Args: + module_names: Sheets modules to report as not installed. + + Returns: + A callable suitable for patching ``importlib.util.find_spec``. + """ + real_find_spec = importlib.util.find_spec + present_spec = importlib.machinery.ModuleSpec("present", loader=None) + + def fake_find_spec(name: str, *args: Any, **kwargs: Any) -> Any: + if name in SHEETS_MODULE_NAMES: + return None if name in module_names else present_spec + return real_find_spec(name, *args, **kwargs) + + return fake_find_spec + + +class TestSheetsPreflight(unittest.TestCase): + """_resolve_sheets_credentials validates the `sheets` extra up front.""" + + def setUp(self) -> None: + tmp_dir = tempfile.TemporaryDirectory() + self.addCleanup(tmp_dir.cleanup) + self.credentials_file = Path(tmp_dir.name) / "creds.json" + self.credentials_file.write_text("{}") + + def test_opted_out_skips_the_module_check(self) -> None: + """ + Without --write-sheets the extra is irrelevant, so a missing + module must not raise: the run does not touch the upload. + """ + args = _make_args(write_sheets=False, google_credentials=None) + with patch( + "importlib.util.find_spec", + _find_spec_missing(set(SHEETS_MODULE_NAMES)), + ): + self.assertIsNone(_resolve_sheets_credentials(args)) + + def test_missing_module_raises_naming_the_extra(self) -> None: + """ + A missing Sheets module raises at startup, and the message names + both the module and the command that installs it. + """ + args = _make_args( + write_sheets=True, + google_credentials=str(self.credentials_file), + ) + with patch( + "importlib.util.find_spec", + _find_spec_missing({"gspread"}), + ): + with self.assertRaises(RuntimeError) as raised: + _resolve_sheets_credentials(args) + message = str(raised.exception) + self.assertIn("gspread", message) + self.assertIn('pip install ".[sheets]"', message) + + def test_module_check_precedes_the_credentials_check(self) -> None: + """ + With no credentials *and* no extra, the missing extra is reported. + Fixing credentials first would only surface the import error at + step 15, which is the failure mode this check exists to prevent. + """ + args = _make_args(write_sheets=True, google_credentials=None) + with patch( + "importlib.util.find_spec", + _find_spec_missing(set(SHEETS_MODULE_NAMES)), + ): + with self.assertRaises(RuntimeError) as raised: + _resolve_sheets_credentials(args) + self.assertIn("`sheets` extra", str(raised.exception)) + + def test_installed_extra_passes_through_to_credentials(self) -> None: + """ + With every module present the check is transparent: resolution + continues and returns the credentials path. + """ + args = _make_args( + write_sheets=True, + google_credentials=str(self.credentials_file), + ) + with patch("importlib.util.find_spec", _find_spec_missing(set())): + with patch.object( + benchmark_application, "TARGET_FOLDER_ID", "folder-id" + ), patch.object(benchmark_application, "TEMPLATE_ID", "template-id"): + resolved = _resolve_sheets_credentials(args) + self.assertEqual(resolved, str(self.credentials_file)) + + +if __name__ == "__main__": + unittest.main() diff --git a/src/signaloid/benchmarking/benchmark_timing/get-timing-template.sh b/src/signaloid/benchmarking/benchmark_timing/get-timing-template.sh index f08b5f2..068e618 100644 --- a/src/signaloid/benchmarking/benchmark_timing/get-timing-template.sh +++ b/src/signaloid/benchmarking/benchmark_timing/get-timing-template.sh @@ -11,9 +11,10 @@ APPLICATION_NAME=$(basename "$APPLICATION_PATH" | sed 's/Signaloid-Demo-//') APPLICATION_VERSION=$(git -C "$APPLICATION_PATH" rev-parse --short=7 HEAD 2>/dev/null \ || date '+%Y-%m-%d-%H-%M-%S') -# Intel PIN kit location (PIN_ROOT). When invoked via the Python tool -# this is exported from --path-to-pin. For standalone use, set it here. -# Uncomment and point at your Pin kit directory: +# Intel PIN kit location (PIN_ROOT). PIN is optional and off by default. +# The Python tool passes it through only with --measure-dynamic-instructions. +# For standalone use, uncomment the line below and point it at your Pin +# kit directory. Leave it commented out to skip the instruction count. # export PIN_ROOT="/path/to/pin-external--gcc-linux" TRACES=( @@ -38,5 +39,24 @@ REPRESENTATION_SIZES=(64 128 256 512) CORRELATION_TRACKING_TYPES=(Disabled Autocorrelation) MAX_JUPITER_LIMIT=32 -# You need to source it to get the env variables to correctly work, also set the correct path to the shell script -. $SIGNALOID_PYTHON_DIR/src/signaloid/benchmarking/benchmark_timing/get-timings.sh +# You need to source get-timings.sh for the variables set above to take +# effect. Its location is resolved from the installed `signaloid` package, so +# this works for both a pip install and a source checkout. We try +# BENCHMARKING_PYTHON, then `python3`, then `python`, since PEP 394 is not +# honoured on every platform. A candidate has to both run and import the +# package, so the loop settles on one that can do the real work below. +GET_TIMINGS_SH="" +for BENCHMARKING_PYTHON in "${BENCHMARKING_PYTHON:-}" python3 python; do + [ -n "$BENCHMARKING_PYTHON" ] || continue + GET_TIMINGS_SH=$("$BENCHMARKING_PYTHON" -c 'import signaloid.benchmarking.config as config +print(config.get_timing_script())' 2>/dev/null) && [ -n "$GET_TIMINGS_SH" ] && break + GET_TIMINGS_SH="" +done + +if [ -z "$GET_TIMINGS_SH" ]; then + echo "Could not locate get-timings.sh. Set \$BENCHMARKING_PYTHON to a Python 3 interpreter that can import the signaloid package." >&2 + return 1 2>/dev/null || exit 1 +fi + +export BENCHMARKING_PYTHON +. "$GET_TIMINGS_SH" diff --git a/src/signaloid/benchmarking/benchmark_timing/get-timings.sh b/src/signaloid/benchmarking/benchmark_timing/get-timings.sh index 6d79151..63e4466 100644 --- a/src/signaloid/benchmarking/benchmark_timing/get-timings.sh +++ b/src/signaloid/benchmarking/benchmark_timing/get-timings.sh @@ -9,6 +9,11 @@ set -euo pipefail SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd) +# Directory containing the `signaloid` package, for `python3 -m signaloid...`. +# Derived from this script's own location, so it resolves to `src/` in a source +# checkout and to site-packages in an installed wheel. +PACKAGE_IMPORT_ROOT=$(cd -- "$SCRIPT_DIR/../../.." &>/dev/null && pwd) + # =========================================================================== # Utility functions # =========================================================================== @@ -22,6 +27,36 @@ warn() { echo "$*" >&2 } +# Check that a command exists and reports itself as Python 3. +# Usage: is_python3 python3 +is_python3() { + command -v "$1" &>/dev/null \ + && "$1" -c 'import sys; sys.exit(0 if sys.version_info[0] == 3 else 1)' &>/dev/null +} + +# Resolve the interpreter used for the `-m signaloid...` helpers below, into +# BENCHMARKING_PYTHON. The Python tool exports it already (its own +# sys.executable). For standalone use we try `python3` first, then a `python` +# that is Python 3, since PEP 394 is not honoured on every platform. +resolve_benchmarking_python() { + local candidate + + if [[ -n "${BENCHMARKING_PYTHON:-}" ]]; then + is_python3 "$BENCHMARKING_PYTHON" \ + || die "\$BENCHMARKING_PYTHON ($BENCHMARKING_PYTHON) is not a working Python 3 interpreter." + return + fi + + for candidate in python3 python; do + if is_python3 "$candidate"; then + BENCHMARKING_PYTHON=$candidate + return + fi + done + + die "No Python 3 interpreter found. Install python3, or set \$BENCHMARKING_PYTHON to one." +} + log_section() { echo "===============================================================" echo "=== $* ===" @@ -69,7 +104,7 @@ default_array() { # Python (see signaloid.benchmarking.automation.benchmarking_utils # `parse_timing_intermediate_stream`). compute_average() { - "${BENCHMARKING_PYTHON:-python3}" -c "values = $1; print(sum(values) / len(values))" + "$BENCHMARKING_PYTHON" -c "values = $1; print(sum(values) / len(values))" } # Scale repetitions so the total measurement time approaches $target_total, @@ -193,14 +228,20 @@ should_skip_representation() { return 1 } -# Run hyperfine and return the mean time. -# Usage: mean_time=$(run_hyperfine_mean "command to benchmark") -run_hyperfine_mean() { +# Return the mean wall-clock time of a command via the measure_process_time +# Python helper. Same PYTHONPATH-prefix shape as parse_cpu_time. +# The repetition count is scaled from a warmup run against the same +# TIMING_* knobs run_uxhw_benchmarks uses, so both timing paths agree. +# Usage: mean_time=$(measure_process_time "command to benchmark" [--no-shell]) +measure_process_time() { local cmd="$1" shift - rm -f hyperfine.json - hyperfine "$@" --warmup 2 --style none --export-json hyperfine.json "$cmd" >/dev/null 2>>"${LOGS_DIR:-/tmp}/hyperfine.log" - jq '.results[0].mean' hyperfine.json + PYTHONPATH="$PACKAGE_IMPORT_ROOT${PYTHONPATH:+:$PYTHONPATH}" \ + "$BENCHMARKING_PYTHON" -m signaloid.benchmarking.automation.measure_process_time \ + --target-total-time "$TIMING_TARGET_TOTAL_TIME" \ + --min-repetitions "$TIMING_MIN_REPETITIONS" \ + --max-repetitions "$TIMING_MAX_REPETITIONS" \ + "$@" "$cmd" } # Extract the seconds value from a `CPU time used:` line in one or @@ -213,8 +254,8 @@ run_hyperfine_mean() { # t=$(parse_cpu_time "$LOGS_DIR/$EXEC_STDOUT") # total=$(parse_cpu_time --sum "${stdout_files[@]}") parse_cpu_time() { - PYTHONPATH="$SIGNALOID_PYTHON_DIR/src${PYTHONPATH:+:$PYTHONPATH}" \ - "${BENCHMARKING_PYTHON:-python3}" -m signaloid.benchmarking.automation.parse_cpu_time "$@" + PYTHONPATH="$PACKAGE_IMPORT_ROOT${PYTHONPATH:+:$PYTHONPATH}" \ + "$BENCHMARKING_PYTHON" -m signaloid.benchmarking.automation.parse_cpu_time "$@" } # Read a single-value DB metric via the read_db_metrics Python helper. @@ -222,8 +263,8 @@ parse_cpu_time() { # Usage: t=$(read_db_metric host-wallclock "$UXHW_DB_BASE.db") read_db_metric() { local metric=$1 db=$2 - PYTHONPATH="$SIGNALOID_PYTHON_DIR/src${PYTHONPATH:+:$PYTHONPATH}" \ - "${BENCHMARKING_PYTHON:-python3}" -m signaloid.benchmarking.automation.read_db_metrics \ + PYTHONPATH="$PACKAGE_IMPORT_ROOT${PYTHONPATH:+:$PYTHONPATH}" \ + "$BENCHMARKING_PYTHON" -m signaloid.benchmarking.automation.read_db_metrics \ "$db" --metric "$metric" } @@ -272,7 +313,6 @@ parse_args() { # =========================================================================== validate_required_vars() { - require_var SIGNALOID_PYTHON_DIR require_var PATH_TO_UXHW_SDK require_var APPLICATION_PATH require_var APPLICATION_NAME @@ -439,20 +479,22 @@ run_uxhw_benchmarks() { db_timing_value=$(read_db_metric host-wallclock "$UXHW_DB_BASE.db") emit_sample "$UXHW_TESTCASE" "databaseTime" "$i" "$db_timing_value" - $INSTRUCTION_COUNT_COMMAND "$APPLICATION_PATH/src/$PROGRAM-$UXHW_TESTCASE" $CLA &>/dev/null - if [[ ! -f inscount.out ]]; then - die "inscount.out not found after PIN execution" + if [[ $PIN_ENABLED -eq 1 ]]; then + $INSTRUCTION_COUNT_COMMAND "$APPLICATION_PATH/src/$PROGRAM-$UXHW_TESTCASE" $CLA &>/dev/null + if [[ ! -f inscount.out ]]; then + die "inscount.out not found after PIN execution" + fi + local uxhw_sample_path="$LOGS_DIR/inscount-$UXHW_TESTCASE-$i-$$.out" + mv inscount.out "$uxhw_sample_path" + emit_sample "$UXHW_TESTCASE" "pinDynInstCount" "$i" "$uxhw_sample_path" fi - local uxhw_sample_path="$LOGS_DIR/inscount-$UXHW_TESTCASE-$i-$$.out" - mv inscount.out "$uxhw_sample_path" - emit_sample "$UXHW_TESTCASE" "pinDynInstCount" "$i" "$uxhw_sample_path" done echo cd "$APPLICATION_PATH/src" cd_to_input_dir - PROCESS_E2E_TIME=$(run_hyperfine_mean "$APPLICATION_PATH/src/$PROGRAM-$UXHW_TESTCASE $CLA" --shell=none) + PROCESS_E2E_TIME=$(measure_process_time "$APPLICATION_PATH/src/$PROGRAM-$UXHW_TESTCASE $CLA" --no-shell) # The emulated RISC-V dynamic instruction count (dbDynInstCount, the 5th # field) was produced by the now-removed UxHw `sf` emulator pass; emit @@ -642,8 +684,8 @@ run_uxhw_tracing() { # which consumes (deletes) the per-config baseline DBs. for i in "${!verify_dbs[@]}"; do [[ -z "${verify_dbs[$i]}" ]] && continue - PYTHONPATH="$SIGNALOID_PYTHON_DIR/src${PYTHONPATH:+:$PYTHONPATH}" \ - "${BENCHMARKING_PYTHON:-python3}" -m signaloid.benchmarking.automation.compare_tracing_ux_strings \ + PYTHONPATH="$PACKAGE_IMPORT_ROOT${PYTHONPATH:+:$PYTHONPATH}" \ + "$BENCHMARKING_PYTHON" -m signaloid.benchmarking.automation.compare_tracing_ux_strings \ "${verify_baseline_dbs[$i]}" "${verify_dbs[$i]}" \ --baseline-label O0 --candidate-label O2 --config "${verify_configs[$i]}" || true done @@ -656,8 +698,8 @@ run_uxhw_tracing() { fi # Phase 3: Merge per-config DBs into final tracing DB - PYTHONPATH="$SIGNALOID_PYTHON_DIR/src${PYTHONPATH:+:$PYTHONPATH}" \ - "${BENCHMARKING_PYTHON:-python3}" -m signaloid.benchmarking.automation.merge_tracing_dbs \ + PYTHONPATH="$PACKAGE_IMPORT_ROOT${PYTHONPATH:+:$PYTHONPATH}" \ + "$BENCHMARKING_PYTHON" -m signaloid.benchmarking.automation.merge_tracing_dbs \ "$TRACING_DB_ABS" "${tracing_dbs[@]}" # Cleanup temporary binaries and config files @@ -791,9 +833,14 @@ run_native_mc_benchmarks() { local process_e2e_time process_e2e_time=$(awk -v e2e_time="$last_measured_e2e_time" -v scale="$scaling_factor" \ 'BEGIN { printf "%.10f", e2e_time * scale }') - local pin_dynamic_instructions - pin_dynamic_instructions=$(awk -v instructions="$last_measured_instructions" -v scale="$scaling_factor" \ - 'BEGIN { printf "%.10f", instructions * scale }') + # An empty last_measured_instructions means PIN was off, so there + # is nothing to scale. Emit the `?` sentinel rather than letting + # awk turn the empty string into a plausible-looking 0. + local pin_dynamic_instructions="?" + if [[ -n "$last_measured_instructions" ]]; then + pin_dynamic_instructions=$(awk -v instructions="$last_measured_instructions" -v scale="$scaling_factor" \ + 'BEGIN { printf "%.10f", instructions * scale }') + fi emit_measurement "Native-MC-$REFERENCE_PRECISION" "$result" "?" "$process_e2e_time" "?" "$pin_dynamic_instructions" continue @@ -828,33 +875,40 @@ run_native_mc_benchmarks() { time_array+="]" echo - rm -f inscount.out - - # Use at most 20 repetitions for dynamic instruction count. - local native_mc_repetition_pin=$((NATIVE_MC_REPETITION < 20 ? NATIVE_MC_REPETITION : 20)) local native_mc_config="Native-MC-$REFERENCE_PRECISION" # Tracked locally so larger precisions in the same loop can # extrapolate from this row (Python is the source of truth for # the JSON field via SAMPLE lines; this average is bash-only). - local pin_dyn_inst_array="[" - echo "Running dynamic instruction measurements" - for ((i = 1; i <= native_mc_repetition_pin; i++)); do - [ -t 1 ] && echo -ne "\rRepetitions ($i/$native_mc_repetition_pin)" - $INSTRUCTION_COUNT_COMMAND "$APPLICATION_PATH/src/$PROGRAM-native" $CLA $CLA_FOR_MULTIPLE_EXECUTIONS "$REFERENCE_PRECISION" &>/dev/null - if [[ ! -f inscount.out ]]; then - die "inscount.out not found after PIN execution" - fi - pin_dyn_inst_array+=$(sed 's/Count //' inscount.out) - pin_dyn_inst_array+="," - local native_mc_sample_path="$LOGS_DIR/inscount-native-mc-$REFERENCE_PRECISION-$i-$$.out" - mv inscount.out "$native_mc_sample_path" - emit_sample "$native_mc_config" "pinDynInstCount" "$i" "$native_mc_sample_path" - done - echo - pin_dyn_inst_array+="]" + # Stays empty when PIN is off, which suppresses the extrapolated + # instruction count below. + local pin_dyn_inst_array="[]" + if [[ $PIN_ENABLED -eq 1 ]]; then + rm -f inscount.out + + # Use at most 20 repetitions for dynamic instruction count. This + # caps the PIN loop only, the timing loop above uses the full + # $NATIVE_MC_REPETITION. + local native_mc_repetition_pin=$((NATIVE_MC_REPETITION < 20 ? NATIVE_MC_REPETITION : 20)) + pin_dyn_inst_array="[" + echo "Running dynamic instruction measurements" + for ((i = 1; i <= native_mc_repetition_pin; i++)); do + [ -t 1 ] && echo -ne "\rRepetitions ($i/$native_mc_repetition_pin)" + $INSTRUCTION_COUNT_COMMAND "$APPLICATION_PATH/src/$PROGRAM-native" $CLA $CLA_FOR_MULTIPLE_EXECUTIONS "$REFERENCE_PRECISION" &>/dev/null + if [[ ! -f inscount.out ]]; then + die "inscount.out not found after PIN execution" + fi + pin_dyn_inst_array+=$(sed 's/Count //' inscount.out) + pin_dyn_inst_array+="," + local native_mc_sample_path="$LOGS_DIR/inscount-native-mc-$REFERENCE_PRECISION-$i-$$.out" + mv inscount.out "$native_mc_sample_path" + emit_sample "$native_mc_config" "pinDynInstCount" "$i" "$native_mc_sample_path" + done + echo + pin_dyn_inst_array+="]" + fi local process_e2e_time - process_e2e_time=$(run_hyperfine_mean "$APPLICATION_PATH/src/$PROGRAM-native $CLA $CLA_FOR_MULTIPLE_EXECUTIONS $REFERENCE_PRECISION") + process_e2e_time=$(measure_process_time "$APPLICATION_PATH/src/$PROGRAM-native $CLA $CLA_FOR_MULTIPLE_EXECUTIONS $REFERENCE_PRECISION") # Time and PIN instruction count are both resolved by Python # from the SAMPLE lines emitted above. The `?` sentinels trigger @@ -875,7 +929,14 @@ run_native_mc_benchmarks() { last_measured_time=$(compute_average "$time_array") last_measured_e2e_time=$process_e2e_time # B1 / B5 boundary: same reasoning as last_measured_time above. - last_measured_instructions=$(compute_average "$pin_dyn_inst_array") + # With PIN off there are no counts to average, and compute_average + # would divide by zero. Leave the value empty so the extrapolation + # block above emits the `?` sentinel instead of a fabricated count. + if [[ $PIN_ENABLED -eq 1 ]]; then + last_measured_instructions=$(compute_average "$pin_dyn_inst_array") + else + last_measured_instructions="" + fi done } @@ -931,10 +992,11 @@ main() { [[ -f "$SCRIPT_DIR/get_timings_local.sh" ]] && source "$SCRIPT_DIR/get_timings_local.sh" + resolve_benchmarking_python validate_required_vars ORIG_PWD=$(pwd) - RESOURCES_DIR="${BENCHMARKING_RESOURCES_DIR:-$SCRIPT_DIR/../../assets/template/coreClass}" + RESOURCES_DIR="${BENCHMARKING_RESOURCES_DIR:-$SCRIPT_DIR/../assets/template/coreClass}" CLA_HASH=$(echo -n "$CLA" | md5sum | cut -d ' ' -f1) @@ -975,20 +1037,35 @@ main() { INPUT_DIR="inputs" echo "\$INPUT_DIR is $INPUT_DIR" - # PIN_ROOT must be provided: the Python tool exports it from - # --path-to-pin, or set it in the environment for a standalone run. - # There is no built-in default (the check below fails clearly if unset). + # Intel PIN is optional and off by default. A run opts in implicitly by + # setting PIN_ROOT. The Python tool only passes it through when + # --measure-dynamic-instructions was given, and removes it otherwise. With + # PIN off we skip the dynamic instruction count entirely and emit no + # pinDynInstCount samples, so the `?` sentinel on the MEASUREMENT lines + # stays unresolved and the Python reader reports the count as missing. PIN_ROOT="${PIN_ROOT:-}" - PIN_TOOL="$PIN_ROOT/source/tools/ManualExamples/obj-intel64/inscount0.so" - INSTRUCTION_COUNT_COMMAND="$PIN_ROOT/pin -t $PIN_TOOL --" + if [[ -n "$PIN_ROOT" ]]; then + PIN_ENABLED=1 + PIN_TOOL="$PIN_ROOT/source/tools/ManualExamples/obj-intel64/inscount0.so" + INSTRUCTION_COUNT_COMMAND="$PIN_ROOT/pin -t $PIN_TOOL --" + else + PIN_ENABLED=0 + PIN_TOOL="" + INSTRUCTION_COUNT_COMMAND="" + fi - # Check Pin tool availability for benchmarks that need it + # Check Pin tool availability for benchmarks that need it. Setting PIN_ROOT + # is an explicit opt-in, so a missing or unbuilt kit stays fatal. if [[ $SKIP_UXHW -eq 0 ]] || [[ $SKIP_NATIVE_MC -eq 0 ]]; then - if [[ ! -x "$PIN_ROOT/pin" ]]; then - die "Intel Pin not found at '$PIN_ROOT/pin'. Set PIN_ROOT (or pass --path-to-pin) to a Pin kit directory containing the 'pin' binary." - fi - if [[ ! -f "$PIN_TOOL" ]]; then - die "Pin inscount0 tool not found at $PIN_TOOL. Build it with: make -C $PIN_ROOT/source/tools/ManualExamples obj-intel64/inscount0.so" + if [[ $PIN_ENABLED -eq 1 ]]; then + if [[ ! -x "$PIN_ROOT/pin" ]]; then + die "Intel Pin not found at '$PIN_ROOT/pin'. Set PIN_ROOT to a Pin kit directory containing the 'pin' binary." + fi + if [[ ! -f "$PIN_TOOL" ]]; then + die "Pin inscount0 tool not found at $PIN_TOOL. Build it with: make -C $PIN_ROOT/source/tools/ManualExamples obj-intel64/inscount0.so" + fi + else + echo "---> Skipping the dynamic instruction count. Pass --measure-dynamic-instructions with PIN_ROOT set to measure it." fi fi diff --git a/src/signaloid/benchmarking/benchmark_timing/get_timing_template_test.py b/src/signaloid/benchmarking/benchmark_timing/get_timing_template_test.py index e69ffcc..fd801b8 100644 --- a/src/signaloid/benchmarking/benchmark_timing/get_timing_template_test.py +++ b/src/signaloid/benchmarking/benchmark_timing/get_timing_template_test.py @@ -28,6 +28,7 @@ """ import os +import shutil import subprocess import tempfile import unittest @@ -38,7 +39,7 @@ _TEMPLATE = Path(__file__).resolve().parent / "get-timing-template.sh" -def _source_template(application_path: Path, signaloid_python_dir: Path) -> str: +def _source_template(application_path: Path, benchmarking_python: Path) -> str: """Source the template and echo APPLICATION_NAME / APPLICATION_VERSION. Returns combined stdout. Tests parse it with simple substring checks. @@ -51,7 +52,7 @@ def _source_template(application_path: Path, signaloid_python_dir: Path) -> str: result = subprocess.run( ["bash", "-c", cmd], env={ - "SIGNALOID_PYTHON_DIR": str(signaloid_python_dir), + "BENCHMARKING_PYTHON": str(benchmarking_python), "APPLICATION_PATH": str(application_path), "PATH": os.environ["PATH"], "HOME": os.environ.get("HOME", ""), @@ -72,33 +73,29 @@ def setUp(self) -> None: tmp_dir = tempfile.TemporaryDirectory() self.addCleanup(tmp_dir.cleanup) self.tmp_path = Path(tmp_dir.name) - self.stub_signaloid_python_dir = self._make_stub_signaloid_python_dir() + self.stub_benchmarking_python = self._make_stub_benchmarking_python() - def _make_stub_signaloid_python_dir(self) -> Path: - """Provide a stub SIGNALOID_PYTHON_DIR with a no-op get-timings.sh. + def _make_stub_benchmarking_python(self) -> Path: + """Provide a stub interpreter that resolves a no-op get-timings.sh. - The template ends by sourcing - ``$SIGNALOID_PYTHON_DIR/src/signaloid/benchmarking/benchmark_timing/get-timings.sh``. - For these tests we only care about the variable assignments at - the top of the template, so we point SIGNALOID_PYTHON_DIR at a - fake tree whose get-timings.sh is empty. + The template ends by asking ``$BENCHMARKING_PYTHON`` for the path of + the bundled ``get-timings.sh`` and sourcing the result. For these tests + we only care about the variable assignments at the top of the template, + so the stub interpreter ignores its arguments and prints the path of an + empty get-timings.sh. """ - fake_get_timings = ( - self.tmp_path - / "src" - / "signaloid" - / "benchmarking" - / "benchmark_timing" - / "get-timings.sh" - ) - fake_get_timings.parent.mkdir(parents=True) + fake_get_timings = self.tmp_path / "get-timings.sh" fake_get_timings.write_text("# no-op stub for tests\n") - return self.tmp_path + + stub_python = self.tmp_path / "stub-python" + stub_python.write_text(f'#!/usr/bin/env bash\necho "{fake_get_timings}"\n') + stub_python.chmod(0o755) + return stub_python def test_template_application_name_is_derived_from_path(self) -> None: app_path = self.tmp_path / "Signaloid-Demo-MyDemo" app_path.mkdir() - out = _source_template(app_path, self.stub_signaloid_python_dir) + out = _source_template(app_path, self.stub_benchmarking_python) self.assertIn("NAME=MyDemo", out) def test_template_application_version_is_non_empty_for_non_git_path( @@ -106,7 +103,7 @@ def test_template_application_version_is_non_empty_for_non_git_path( ) -> None: app_path = self.tmp_path / "demo" app_path.mkdir() - out = _source_template(app_path, self.stub_signaloid_python_dir) + out = _source_template(app_path, self.stub_benchmarking_python) version_line = next( line for line in out.splitlines() if line.startswith("VERSION=") ) @@ -118,5 +115,92 @@ def test_template_application_version_is_non_empty_for_non_git_path( ) +class TestGetTimingTemplateInterpreterDiscovery(unittest.TestCase): + """The template must find an interpreter without BENCHMARKING_PYTHON set. + + PEP 394 asks for a ``python3`` on the PATH, but it is not honoured + everywhere, so the template also accepts a ``python`` that is Python 3. + """ + + # The template shells out to these while populating APPLICATION_NAME and + # APPLICATION_VERSION, so a restricted PATH still has to provide them. + _REQUIRED_TOOLS = ("basename", "sed", "date") + + def setUp(self) -> None: + tmp_dir = tempfile.TemporaryDirectory() + self.addCleanup(tmp_dir.cleanup) + self.tmp_path = Path(tmp_dir.name) + + self.stub_get_timings = self.tmp_path / "get-timings.sh" + self.stub_get_timings.write_text("# no-op stub for tests\n") + + def _make_bin_dir(self, *interpreter_names: str) -> Path: + """Build a PATH directory holding only the named interpreters. + + Each named interpreter is a stub that ignores its arguments and prints + the path of an empty get-timings.sh, standing in for + ``print(config.get_timing_script())``. The tools the template itself + shells out to are symlinked in, so nothing else leaks in from the + real PATH. + + Args: + interpreter_names: Names to create, e.g. ``"python"``. Passing + none yields a directory with no interpreter at all. + + Returns: + The directory to use as the entire PATH. + """ + bin_dir = self.tmp_path / ("bin-" + "-".join(interpreter_names or ("none",))) + bin_dir.mkdir() + + for tool in self._REQUIRED_TOOLS: + resolved = shutil.which(tool) + assert resolved is not None, f"{tool} is needed to run this test" + (bin_dir / tool).symlink_to(resolved) + + for name in interpreter_names: + stub = bin_dir / name + stub.write_text(f'#!/bin/sh\necho "{self.stub_get_timings}"\n') + stub.chmod(0o755) + + return bin_dir + + def _source_with_path(self, bin_dir: Path) -> subprocess.CompletedProcess[str]: + """Source the template with `bin_dir` as the whole PATH. + + BENCHMARKING_PYTHON is deliberately absent so the template has to + discover an interpreter by name. + """ + app_path = self.tmp_path / "Signaloid-Demo-MyDemo" + app_path.mkdir(exist_ok=True) + bash = shutil.which("bash") + assert bash is not None, "bash is needed to run this test" + return subprocess.run( + [bash, "-c", f'source {_TEMPLATE} && echo "NAME=$APPLICATION_NAME"'], + env={"PATH": str(bin_dir), "APPLICATION_PATH": str(app_path)}, + capture_output=True, + text=True, + check=False, + ) + + def test_python3_on_path_is_used(self) -> None: + result = self._source_with_path(self._make_bin_dir("python3")) + self.assertEqual(result.returncode, 0, f"stderr={result.stderr!r}") + self.assertIn("NAME=MyDemo", result.stdout) + + def test_bare_python_is_used_when_python3_is_absent(self) -> None: + """The case this class exists for: only `python`, and it is Python 3.""" + result = self._source_with_path(self._make_bin_dir("python")) + self.assertEqual(result.returncode, 0, f"stderr={result.stderr!r}") + self.assertIn("NAME=MyDemo", result.stdout) + + def test_no_interpreter_fails_loudly(self) -> None: + """With no interpreter the template must error, not silently no-op.""" + result = self._source_with_path(self._make_bin_dir()) + self.assertNotEqual(result.returncode, 0, "expected a non-zero exit") + self.assertIn("Could not locate get-timings.sh", result.stderr) + self.assertIn("BENCHMARKING_PYTHON", result.stderr) + + if __name__ == "__main__": unittest.main() diff --git a/src/signaloid/benchmarking/benchmark_timing/get_timings_test.py b/src/signaloid/benchmarking/benchmark_timing/get_timings_test.py new file mode 100644 index 0000000..835a1a5 --- /dev/null +++ b/src/signaloid/benchmarking/benchmark_timing/get_timings_test.py @@ -0,0 +1,92 @@ +# Copyright (c) 2026, Signaloid. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to +# deal in the Software without restriction, including without limitation the +# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +# sell copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +# DEALINGS IN THE SOFTWARE. + +""" +Regression coverage for the build-resources fallback in get-timings.sh. + +``get-timings.sh`` resolves the bundled coreClass build templates from +``BENCHMARKING_RESOURCES_DIR`` when the Python tool exports it, and otherwise +falls back to a path relative to its own location. The Python tool always +exports it, so the fallback only runs for standalone bash use and went +unnoticed while it pointed one directory level too high. These tests pin the +fallback to the same directory ``get_resources_dir()`` returns. +""" + +import re +import unittest + +from pathlib import Path + +from signaloid.benchmarking.config import get_resources_dir + +_SCRIPT = Path(__file__).resolve().parent / "get-timings.sh" + +# Matches the default in `RESOURCES_DIR="${BENCHMARKING_RESOURCES_DIR:-...}"`. +_FALLBACK_PATTERN = re.compile( + r'RESOURCES_DIR="\$\{BENCHMARKING_RESOURCES_DIR:-(?P[^}]+)\}"' +) + + +def _fallback_resources_dir() -> Path: + """Resolve the script's own default for the build-resources directory. + + Returns: + The directory the fallback expands to, with ``$SCRIPT_DIR`` + substituted for the script's location and ``..`` segments collapsed. + """ + match = _FALLBACK_PATTERN.search(_SCRIPT.read_text()) + assert match is not None, "could not find the RESOURCES_DIR fallback" + default = match.group("default") + assert "$SCRIPT_DIR" in default, f"fallback is not script-relative: {default}" + return Path(default.replace("$SCRIPT_DIR", str(_SCRIPT.parent))).resolve() + + +class TestGetTimingsResourcesFallback(unittest.TestCase): + """The standalone fallback must find the bundled build templates.""" + + def test_fallback_directory_exists(self) -> None: + fallback = _fallback_resources_dir() + self.assertTrue( + fallback.is_dir(), + f"fallback build-resources directory does not exist: {fallback}", + ) + + def test_fallback_matches_python_resolver(self) -> None: + """The bash fallback and get_resources_dir() must not diverge.""" + self.assertEqual(_fallback_resources_dir(), get_resources_dir().resolve()) + + def test_fallback_contains_expected_templates(self) -> None: + """Guard the files copy_build_resources() reads from that directory.""" + fallback = _fallback_resources_dir() + for relative_path in ( + "C0/init.S", + "common/startup.cpp", + "C0Pro/Makefile.pro", + "C0/Makefile", + ): + with self.subTest(relative_path=relative_path): + self.assertTrue( + (fallback / relative_path).is_file(), + f"missing build template: {relative_path}", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/src/signaloid/benchmarking/config.py b/src/signaloid/benchmarking/config.py index 02722cf..cfaf5eb 100644 --- a/src/signaloid/benchmarking/config.py +++ b/src/signaloid/benchmarking/config.py @@ -18,7 +18,6 @@ # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE. -import os from enum import Enum from pathlib import Path @@ -312,34 +311,19 @@ class TimingFormat: MISSING_VALUE = "?" -def get_repo_root() -> Path: +def get_timing_script() -> Path: """ - Find the repository root. + Path to the bundled timing bash script. - Checks the SIGNALOID_PYTHON_DIR env var first, then walks upward from - this file looking for pyproject.toml. + Resolved relative to this package, so it works from both a source checkout + and an installed wheel. The script locates its own dependencies (the + ``signaloid`` import root and the bundled assets) from its own location, so + sourcing it by this path needs no environment setup. Returns: - The repository root directory. - - Raises: - RuntimeError: If the repository root cannot be located. + The ``benchmark_timing/get-timings.sh`` file inside this package. """ - env_override = os.environ.get("SIGNALOID_PYTHON_DIR") - if env_override: - p = Path(env_override).resolve() - if p.is_dir(): - return p - - current = Path(__file__).resolve().parent - while current != current.parent: - if (current / "pyproject.toml").exists(): - return current - current = current.parent - - raise RuntimeError( - "Could not find repository root. Set SIGNALOID_PYTHON_DIR or run from a repo checkout." - ) + return Path(__file__).parent / "benchmark_timing" / "get-timings.sh" def get_resources_dir() -> Path: diff --git a/src/signaloid/benchmarking/types.py b/src/signaloid/benchmarking/types.py index 934ef70..77e952b 100644 --- a/src/signaloid/benchmarking/types.py +++ b/src/signaloid/benchmarking/types.py @@ -134,7 +134,7 @@ class TimingMeasurements: pipeline phases. Owned by ``BenchmarkingVariable.timing_measurements``. """ - measurement_dict: dict[str, dict[str, float]] = field(default_factory=dict) + measurement_dict: dict[str, dict[str, float | None]] = field(default_factory=dict) def append( self, @@ -142,7 +142,7 @@ def append( config: str, time: float, e2e_time: float, - pin_dyn_inst_count: float, + pin_dyn_inst_count: float | None = None, db_time: float = 0.0, db_dyn_inst_count: float = 0.0, ) -> None: @@ -153,11 +153,13 @@ def append( config: Configuration key (e.g. representation type string). time: In-application elapsed time in seconds. e2e_time: End-to-end elapsed time in seconds. - pin_dyn_inst_count: Dynamic instruction count from PIN. + pin_dyn_inst_count: Dynamic instruction count from PIN, or + ``None`` when the run was made without an Intel PIN kit. + Intel PIN is optional and off by default. db_time: Database-access time in seconds. db_dyn_inst_count: Dynamic instruction count for DB access. """ - dictionary: dict[str, float] = {} + dictionary: dict[str, float | None] = {} dictionary["In Application Time"] = time dictionary["Database Time"] = db_time dictionary["End-to-End Time"] = e2e_time diff --git a/src/signaloid/distributional/distributional.py b/src/signaloid/distributional/distributional.py index 97bf0ea..5838cb1 100644 --- a/src/signaloid/distributional/distributional.py +++ b/src/signaloid/distributional/distributional.py @@ -59,7 +59,28 @@ }, } +# Representation type byte, as carried at offset 11 of the Ux Binary layout. +# 0x00 is not a core: it is the "not stated" value, for a value assembled in +# Python rather than read off a core (raw samples, a collapse result). Keeping +# it as the constructor default matters because `TTR_NATIVE_UR_TYPES` below +# makes the byte decide how a value is binned, and defaulting to a real core +# would silently claim a provenance the value does not have. +UR_TYPE_UNSPECIFIED = 0x00 UR_TYPE_ATHENS = 0x04 +UR_TYPE_JUPITER = 0x06 +UR_TYPE_ATLAS = 0x07 + +# Cores whose Dirac deltas are a TTR by construction. Their values should be +# binned by the TTR route even when `check_is_full_valid_TTR` says otherwise: +# curing and zero-mass dropping routinely leave such a value with a +# non-power-of-2 Dirac count, which fails the check without making the value +# any less TTR-shaped. +# +# Every other type — Jupiter, whose Dirac deltas are particles rather than a +# TTR, and `UR_TYPE_UNSPECIFIED` — is left to `check_is_full_valid_TTR` to +# route: the TTR binning when the value does happen to be a valid TTR, the +# non-uniform binning when it does not. +TTR_NATIVE_UR_TYPES: frozenset[int] = frozenset({UR_TYPE_ATHENS, UR_TYPE_ATLAS}) # First byte of the Ux Binary Data format, used to distinguish it from the legacy format UX_BINARY_FORMAT_MARKER = 0xF0 @@ -69,7 +90,7 @@ class DistributionalValue: def __init__( self, particle_value: float | None = None, - UR_type: int = UR_TYPE_ATHENS, + UR_type: int = UR_TYPE_UNSPECIFIED, dirac_deltas: list[DiracDelta] | None = None, double_precision: bool = True, ) -> None: diff --git a/src/signaloid/distributional_distance/binned_wasserstein.py b/src/signaloid/distributional_distance/binned_wasserstein.py index 819a974..1bf34f2 100644 --- a/src/signaloid/distributional_distance/binned_wasserstein.py +++ b/src/signaloid/distributional_distance/binned_wasserstein.py @@ -546,9 +546,6 @@ def binned_wasserstein_1_uxhw_wrapper( # the finite_dirac_deltas access above). plot_interface = PlotData(binned_dist) - boundary_positions, bin_widths, bin_heights = PlotData.create_binning( - binned_dist.finite_dirac_deltas, 0, False - ) if plot_interface.plotting_ttr_order is None: raise ValueError( "PlotData failed to resolve plotting_ttr_order for the input " @@ -556,22 +553,17 @@ def binned_wasserstein_1_uxhw_wrapper( "Wasserstein-1." ) - # Find the TTR of the created binning. This is always a valid TTR. - ttr = PlotData.bin_pdf_to_ttr( - boundary_positions, - bin_widths, - bin_heights, - plot_interface.plotting_ttr_order, - ) - # Rebuild the binning from the valid TTR via the TTR binning method. - bin_boundaries, bin_widths, bin_heights = PlotData.create_binning( - ttr, plot_interface.plotting_ttr_order, True - ) - + # `PlotData` has already selected the binning that suits this input: + # the TTR binning route when the value is a full valid TTR, and the + # non-uniform binning (one bin per Dirac delta, boundaries following the + # Dirac spacing) when it is not. Recomputing the TTR route here + # unconditionally would bin a non-TTR value as though it were a TTR, + # which is exactly what `_construct_plot_data` avoids, and would leave + # the measured distance disagreeing with the plotted distribution. return wasserstein_1_between_distribution_and_samples( - bin_boundaries=list(bin_boundaries), - bin_heights=list(bin_heights), - bin_widths=list(bin_widths), + bin_boundaries=list(plot_interface.positions), + bin_heights=list(plot_interface.masses), + bin_widths=list(plot_interface.widths), sample_positions=list(ground_truth_dist.positions), sample_weights=list(ground_truth_dist.masses), ) diff --git a/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas.py b/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas.py index 33008c3..f3737bb 100644 --- a/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas.py +++ b/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas.py @@ -30,7 +30,10 @@ from signaloid.circuitpython.extended_ulab_numpy import np # type: ignore[no-redef] from signaloid.distributional.dirac_delta import DiracDelta -from signaloid.distributional.distributional import DistributionalValue +from signaloid.distributional.distributional import ( + TTR_NATIVE_UR_TYPES, + DistributionalValue, +) class PlotData: @@ -56,6 +59,12 @@ def __init__( # Maximum number of bins for plotting MAX_BINS: int = 1024 + # Closest an internal bin boundary of a non-uniform binning may sit to the Dirac + # delta it is placed against, as a fraction of the interval between that Dirac + # delta and its neighbour. Only keeps the extremal bins from collapsing to zero + # width, so it is deliberately tiny: see `create_non_uniform_binning`. + MIN_INTERIOR_WEIGHT: float = 1e-6 + @classmethod def from_samples( cls, @@ -194,7 +203,8 @@ def _determine_boundary_positions( Args: finite_sorted_dirac_deltas: The input Dirac deltas with finite and - sorted positions. + sorted positions. At least two are required, since the extremal + boundaries are placed relative to an adjacent Dirac delta. exponent: The TTR order, i.e., the base-2 logarithm of the number of Dirac deltas in the TTR. The number of bins in the output binning is twice the number of Dirac deltas in the TTR. @@ -202,9 +212,17 @@ def _determine_boundary_positions( Returns: (boundary_positions, boundary_probabilities): The internal boundary positions and boundary probabilities that are intermediaries to get a binning. + Raises: + ValueError: When given fewer than two Dirac deltas. """ number_of_finite_dirac_deltas = len(finite_sorted_dirac_deltas) + if number_of_finite_dirac_deltas < 2: + raise ValueError( + "plot_histogram_dirac_deltas: _determine_boundary_positions requires " + f"at least two Dirac deltas, got {number_of_finite_dirac_deltas}" + ) + number_of_boundaries = 2 * number_of_finite_dirac_deltas + 1 boundary_positions = np.array([np.nan] * number_of_boundaries) boundary_probabilities = np.array([np.nan] * number_of_boundaries) @@ -433,7 +451,7 @@ def create_binning( Args: finite_sorted_dirac_deltas: The input Dirac deltas with finite - and sorted positions. + and sorted positions. At least two are required. exponent: The TTR order, i.e., the base-2 logarithm of the number of Dirac deltas in the TTR. The number of bins in the output binning is twice the number of Dirac deltas in the TTR. @@ -441,8 +459,20 @@ def create_binning( Returns: (boundary_positions, bin_widths, bin_heights): The boundary positions, bin widths, and bin heights that describe the output binning. + Raises: + ValueError: When given fewer than two Dirac deltas. Both bins that + surround a lone Dirac delta would be bounded on the outside by a + boundary placed relative to a Dirac delta that does not exist, so + the binning has no widths. Callers plot a lone Dirac delta as a + Dirac delta instead, which is what `_construct_plot_data` does. """ + if len(finite_sorted_dirac_deltas) < 2: + raise ValueError( + "plot_histogram_dirac_deltas: create_binning requires at least two " + f"Dirac deltas, got {len(finite_sorted_dirac_deltas)}" + ) + ( boundary_positions, boundary_probabilities, @@ -456,6 +486,212 @@ def create_binning( return (boundary_positions, bin_widths, bin_heights) + @staticmethod + def _cell_boundary_positions( + positions: np.ndarray, interior_weight: float + ) -> np.ndarray: + """ + Places one bin boundary between each pair of adjacent Dirac deltas, at the + fraction `interior_weight` of the way from the left to the right Dirac delta. + The two extremal boundaries are mirror images of their inner neighbours + about the extremal Dirac deltas, which is the (d/dx) = 0 boundary condition. + + Args: + positions: Strictly increasing Dirac delta positions. At least two are + required, since every boundary is placed relative to an adjacent + Dirac delta. + interior_weight: Position of an internal boundary within the interval + between two adjacent Dirac deltas. `0.5` places it at the midpoint. + Returns: + boundary_positions: The (N + 1) boundary positions of the N cells. + Raises: + ValueError: When given fewer than two positions. + """ + + number_of_cells = len(positions) + if number_of_cells < 2: + raise ValueError( + "plot_histogram_dirac_deltas: _cell_boundary_positions requires at " + f"least two positions, got {number_of_cells}" + ) + + boundary_positions = np.array([np.nan] * (number_of_cells + 1)) + boundary_positions[1:-1] = (1 - interior_weight) * positions[ + :-1 + ] + interior_weight * positions[1:] + boundary_positions[0] = 2 * positions[0] - boundary_positions[1] + boundary_positions[-1] = 2 * positions[-1] - boundary_positions[-2] + + return boundary_positions + + @staticmethod + def _local_mean_boundary_positions(positions: np.ndarray) -> np.ndarray | None: + """ + Places the bin boundaries so that every bin is centred on the Dirac delta it + holds, which preserves each Dirac delta's own mean and not just the mean of + the whole distribution. This is the property the TTR binning method has, and + the property `create_binning` gets from using two bins per Dirac delta. + + Centring bin `i` on Dirac delta `i` means (b[i] + b[i + 1]) / 2 == p[i], so + the boundaries follow the recurrence b[i + 1] = 2 * p[i] - b[i]: the whole + binning is determined by the first boundary alone. Writing e[i] for the + distance p[i] - b[i], the width of bin `i` is 2 * e[i] and the recurrence is + e[i + 1] = g[i] - e[i], for gaps g[i] = p[i + 1] - p[i]. Each e[i] is + therefore affine in e[0] with an alternating sign, so requiring every + boundary to stay strictly between the two Dirac deltas it separates, + 0 < e[i] < g[i], is a set of linear constraints on e[0]. + + Those constraints have no solution for arbitrary Dirac delta spacing, since + the recurrence is undamped and alternating gaps drive e[i] out of its + interval. Where there is a solution, this takes the midpoint of it, which is + the choice furthest from every constraint and so has the widest bins. The + constraints are solved in exact arithmetic, so a feasible interval that is + narrow enough can still yield boundaries that rounding pushes onto or past a + Dirac delta. The constructed boundaries are therefore re-checked and also + rejected in that case. + + Args: + positions: Strictly increasing Dirac delta positions, at least two. + Returns: + boundary_positions: The (N + 1) boundary positions of the N Dirac deltas, or + `None` when no first boundary exists that solve the constraints + or when rounding pushes a bin boundary beyond the next Dirac delta. + """ + + gaps = positions[1:] - positions[:-1] + + # Intersect the constraints 0 < e[i] < g[i] to bound e[0]. + lowest, highest = 0.0, float(np.inf) + constant, sign = 0.0, 1.0 + for gap in gaps: + if sign > 0: + lowest = max(lowest, -constant) + highest = min(highest, float(gap) - constant) + else: + lowest = max(lowest, constant - float(gap)) + highest = min(highest, constant) + constant, sign = float(gap) - constant, -sign + + if not lowest < highest: + return None + + boundary_positions = np.array([np.nan] * (len(positions) + 1)) + distance_to_dirac_delta = (lowest + highest) / 2 + boundary_positions[0] = positions[0] - distance_to_dirac_delta + for i, position in enumerate(positions): + boundary_positions[i + 1] = position + distance_to_dirac_delta + if i + 1 < len(positions): + distance_to_dirac_delta = ( + positions[i + 1] - position - distance_to_dirac_delta + ) + + # The constraints are solved in exact arithmetic, so re-check the result + # rather than trust that rounding kept every bin valid and populated. + if not ( + bool(np.all(boundary_positions[1:] > boundary_positions[:-1])) + and bool(np.all(positions > boundary_positions[:-1])) + and bool(np.all(positions < boundary_positions[1:])) + ): + return None + + return boundary_positions + + @staticmethod + def create_non_uniform_binning( + finite_sorted_dirac_deltas: list[DiracDelta], + ) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + """ + Creates a binning with one bin per Dirac delta, where the bin holding a + Dirac delta carries exactly that Dirac delta's mass and the internal bin + boundaries lie between adjacent Dirac deltas. The bins therefore have + non-uniform widths that follow the spacing of the Dirac deltas, and there + are no empty bins between Dirac deltas, i.e., the binning reads the input + as a continuous distribution with no gaps in its support. + + The total mass of the binning always equals that of the input. Where the + Dirac delta spacing allows it, every bin is also centred on the Dirac delta + it holds, so each Dirac delta's own mean is preserved and not merely the + mean of the whole distribution, which is the property the TTR binning method + has. That is not solvable for every spacing (see + `_local_mean_boundary_positions`), and where it is not, the boundaries fall + back to a single placement fraction shared by all of them, solved to match + the mean of the whole distribution. That mean is then exact, unless matching + it would collapse an extremal bin to zero width and the placement is + therefore clamped to `MIN_INTERIOR_WEIGHT`. The mean is then approximate, + but never further off than placing every boundary at a midpoint would leave + it. + + Args: + finite_sorted_dirac_deltas: The input Dirac deltas with finite, + strictly increasing positions. At least two are required. + Returns: + (boundary_positions, bin_widths, bin_heights): The boundary positions, + bin widths, and bin heights that describe the output binning. + Raises: + ValueError: When given fewer than two Dirac deltas. A lone Dirac delta + has no adjacent Dirac delta to place a bin boundary against, so it + has no bin width. Callers plot it as a Dirac delta instead, which is + what `_construct_plot_data` does. + """ + + if len(finite_sorted_dirac_deltas) < 2: + raise ValueError( + "plot_histogram_dirac_deltas: create_non_uniform_binning requires " + f"at least two Dirac deltas, got {len(finite_sorted_dirac_deltas)}" + ) + + positions = np.array([dd.position for dd in finite_sorted_dirac_deltas]) + masses = np.array([dd.mass for dd in finite_sorted_dirac_deltas]) + + # Preserving every Dirac delta's own mean also preserves the mean of the + # whole distribution, so prefer it wherever the spacing admits it. + boundary_positions = PlotData._local_mean_boundary_positions(positions) + if boundary_positions is not None: + bin_widths = boundary_positions[1:] - boundary_positions[:-1] + return (boundary_positions, bin_widths, masses / bin_widths) + + target_mean = float(np.sum(positions * masses)) + + def mean_of(interior_weight: float) -> float: + boundaries = PlotData._cell_boundary_positions(positions, interior_weight) + return float(np.sum(masses * (boundaries[:-1] + boundaries[1:]) / 2)) + + # The mean of the binning is affine in `interior_weight`, so two + # evaluations determine the value that reproduces the input mean. The mean + # can also be independent of `interior_weight` (e.g., for two Dirac + # deltas, where the mirrored boundaries centre both cells on their Dirac + # delta for any placement), in which case any placement is mean-exact. + mean_at_zero = mean_of(0.0) + mean_at_one = mean_of(1.0) + if abs(mean_at_one - mean_at_zero) < 1e-15: + interior_weight = 0.5 + else: + interior_weight = (target_mean - mean_at_zero) / ( + mean_at_one - mean_at_zero + ) + + # Keep the placement off the ends of [0, 1], where an extremal bin would + # collapse to zero width: the width of an internal bin is a convex + # combination of the two gaps adjacent to its Dirac delta, so it stays + # positive throughout, but the width of an extremal bin is twice the weight + # times the extremal gap. The bound is only there to keep the widths + # positive, so it is as small as it can be: clamping is what makes the mean + # of the binning approximate, and a placement near an end is the correct + # answer for a value whose extremal Dirac deltas are much closer together + # than the rest, where the density really is that much higher there. + interior_weight = min( + max(interior_weight, PlotData.MIN_INTERIOR_WEIGHT), + 1 - PlotData.MIN_INTERIOR_WEIGHT, + ) + + boundary_positions = PlotData._cell_boundary_positions( + positions, interior_weight + ) + bin_widths = boundary_positions[1:] - boundary_positions[:-1] + bin_heights = masses / bin_widths + + return (boundary_positions, bin_widths, bin_heights) + @staticmethod def _bin_pdf_expected_dirac_delta( boundary_positions: np.ndarray, bin_widths: np.ndarray, bin_heights: np.ndarray @@ -605,7 +841,24 @@ def _construct_plot_data(self) -> None: "plot_histogram_dirac_deltas: plotting_resolution must be a power of 2!" ) - if self.dist.check_is_full_valid_TTR(): + # Athens / Atlas values come off a TTR core, so bin them by the TTR + # route regardless of what `check_is_full_valid_TTR` reports. That check + # fails for reasons that say nothing about whether the value is + # TTR-shaped — `drop_zero_mass_positions` and `cure` leave a + # non-power-of-2 Dirac count, or the reconstructed boundaries tie rather + # than strictly increase — and demoting those values to the non-uniform + # binning measures them against a support model their core never used. + # The route is safe on an invalid TTR because `bin_pdf_to_ttr` below + # projects whatever it is handed onto a valid TTR before the TTR binning + # method sees it; the `except` still catches the cases it cannot. + # Jupiter's Dirac deltas are particles, not a TTR, so it keeps the + # valid-TTR gate and otherwise falls through to the non-uniform binning. + use_ttr_route = ( + self.dist.UR_type in TTR_NATIVE_UR_TYPES + or self.dist.check_is_full_valid_TTR() + ) + + if use_ttr_route: try: # Create the binning such that the average of two bins surrounding a Dirac delta # is the Dirac delta itself. @@ -632,21 +885,34 @@ def _construct_plot_data(self) -> None: except (ValueError, TypeError): pass - # Non-TTR input (or TTR pipeline failed): use a uniform-width histogram - # at `plotting_resolution`. The TTR binning method assumes the input - # forms a valid TTR, so applying it to arbitrary Dirac deltas (e.g. raw - # samples) produces meaningless boundaries. - positions = np.array([dd.position for dd in finite_dirac_deltas]) - masses = np.array([dd.mass for dd in finite_dirac_deltas]) - pos_min = float(positions[0]) - pos_max = float(positions[-1]) - edges = np.linspace(pos_min, pos_max, self.plotting_resolution + 1) - hist, _ = np.histogram(positions, bins=edges, weights=masses) - bin_widths = edges[1:] - edges[:-1] - self.positions = edges - self.masses = np.divide( - hist, - bin_widths, - out=np.zeros_like(hist, dtype=np.float64), - where=bin_widths != 0, + # For non-TTR input, use the non-uniform binning instead, which places one bin per Dirac delta with boundaries + # following the spacing of the Dirac deltas. This reads the input as a + # continuous distribution with no gaps in its support, whereas a + # uniform-width histogram leaves empty bins wherever the Dirac deltas are + # sparser than the bin width. + # A binning needs two bins, so it needs two Dirac deltas. The order is only + # 0 when the caller asked for a `plotting_resolution` of 2. + reduction_order = max(self.plotting_ttr_order, 1) + if len(finite_dirac_deltas) > 2**reduction_order: + # More Dirac deltas than the plot can resolve. Reduce them to + # (2 ** `reduction_order`) of them, which is the same number of Dirac + # deltas the valid-TTR path represents the value with, via the same two + # steps that path uses: bin the Dirac deltas without assuming a valid + # TTR, then take the TTR of that binning. Both steps preserve the total + # mass and the mean. + boundary_positions, bin_widths, bin_heights = PlotData.create_binning( + finite_dirac_deltas, 0, False + ) + finite_dirac_deltas = PlotData.bin_pdf_to_ttr( + boundary_positions, + bin_widths, + bin_heights, + reduction_order, + ) + + boundary_positions, bin_widths, bin_heights = ( + PlotData.create_non_uniform_binning(finite_dirac_deltas) ) + + self.positions = boundary_positions + self.masses = bin_heights diff --git a/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas_test.py b/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas_test.py index 47164f2..5b41ae9 100644 --- a/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas_test.py +++ b/src/signaloid/distributional_information_plotting/plot_histogram_dirac_deltas_test.py @@ -161,6 +161,403 @@ def test_create_binning_property_preserve_ttr(self) -> None: ) +class TestCreateBinningGuards(unittest.TestCase): + """Covers rejection of input that `PlotData.create_binning()` cannot bin.""" + + def test_create_binning_rejects_fewer_than_two_dirac_deltas(self) -> None: + """A lone Dirac delta leaves both extremal boundaries to be placed relative + to a Dirac delta that does not exist. That must fail loudly rather than + return `NaN`-valued boundaries, widths, and heights.""" + for input_ensemble in ([], [DiracDelta(2.5, mass=1.0)]): + for use_ttr_binning in (False, True): + with self.subTest( + number_of_dirac_deltas=len(input_ensemble), + use_ttr_binning=use_ttr_binning, + ): + with self.assertRaises(ValueError) as raised: + PlotData.create_binning(input_ensemble, 0, use_ttr_binning) + self.assertIn("at least two Dirac deltas", str(raised.exception)) + + with self.assertRaises(ValueError): + PlotData._determine_boundary_positions([DiracDelta(2.5, mass=1.0)], 0, True) + + def test_smallest_plotting_resolution_bins_a_valid_ttr(self) -> None: + """A `plotting_resolution` of 2 puts `plotting_ttr_order` at 0, which would + reduce a value to a single Dirac delta. The binning must still come out + finite, with the requested number of bins and unit total mass.""" + positions = np.array([-1.5, -0.5, 0.5, 1.5]) + masses = np.full(4, 0.25) + dirac_deltas = [ + DiracDelta(position, mass=mass) for position, mass in zip(positions, masses) + ] + + for plotting_resolution in (2, 4, 8): + with self.subTest(plotting_resolution=plotting_resolution): + dist = DistributionalValue( + dirac_deltas=[ + DiracDelta(dd.position, mass=dd.mass) for dd in dirac_deltas + ] + ) + pd = PlotData(dist, plotting_resolution=plotting_resolution) + + self.assertEqual(len(pd.masses), plotting_resolution) + self.assertFalse(bool(np.any(np.isnan(pd.positions)))) + self.assertFalse(bool(np.any(np.isnan(pd.masses)))) + + widths = pd.positions[1:] - pd.positions[:-1] + self.assertTrue(np.all(widths > 0)) + self.assertAlmostEqual(float(np.sum(widths * pd.masses)), 1.0, places=9) + + +class TestCreateNonUniformBinning(unittest.TestCase): + """Covers `PlotData.create_non_uniform_binning()`, the binning used for input + that does not form a full valid TTR.""" + + @staticmethod + def _random_ensembles( + mass_distribution: str, number_of_testcases: int = 500, seed: int = 20260731 + ) -> list[list[DiracDelta]]: + """Random ensembles of Dirac deltas with strictly increasing positions. The + generator is seeded, so a failure here is reproducible rather than a flake.""" + rng = np.random.default_rng(seed) + ensembles: list[list[DiracDelta]] = [] + + for _ in range(number_of_testcases): + number_of_dirac_deltas = int(rng.integers(2, 400 + 1)) + positions = np.unique( + rng.uniform( + rng.uniform(-100, 0), + rng.uniform(0, 100), + number_of_dirac_deltas, + ) + ) + if len(positions) < 2: + continue + + masses: np.ndarray + if mass_distribution == "equal": + masses = np.full(len(positions), 1.0) + elif mass_distribution == "uniform": + masses = rng.uniform(0, 1, len(positions)) + else: + # Heavily skewed masses, i.e., a few Dirac deltas hold almost all + # of the probability mass. + masses = rng.exponential(1.0, len(positions)) ** 3 + masses /= sum(masses) + + ensembles.append( + [ + DiracDelta(position, mass=mass) + for position, mass in zip(positions, masses) + ] + ) + + return ensembles + + def test_create_non_uniform_binning_property_one_bin_per_dirac_delta(self) -> None: + """ + Tests that the binning created by `PlotData.create_non_uniform_binning()` + holds exactly one bin per input Dirac delta, that the bin boundaries are + strictly increasing (so no bin is empty or inverted), that each Dirac delta + lies strictly inside its own bin, and that each bin carries exactly the + mass of the Dirac delta it holds. + """ + probability_threshold: float = 1e-15 + + for mass_distribution in ("equal", "uniform", "skewed"): + for input_ensemble in self._random_ensembles(mass_distribution): + positions = np.array([dd.position for dd in input_ensemble]) + masses = np.array([dd.mass for dd in input_ensemble]) + + ( + boundary_positions, + bin_widths, + bin_heights, + ) = PlotData.create_non_uniform_binning(input_ensemble) + + self.assertEqual(len(bin_widths), len(input_ensemble)) + self.assertEqual(len(boundary_positions), len(input_ensemble) + 1) + self.assertTrue( + np.all(boundary_positions[1:] > boundary_positions[:-1]) + ) + self.assertTrue(np.all(positions > boundary_positions[:-1])) + self.assertTrue(np.all(positions < boundary_positions[1:])) + self.assertTrue( + np.all( + np.abs(bin_widths * bin_heights - masses) + < probability_threshold + ) + ) + + # The boundary placement clamp in `PlotData.create_non_uniform_binning()`. + INTERIOR_WEIGHT_CLAMP: float = PlotData.MIN_INTERIOR_WEIGHT + + @staticmethod + def _mean_of_binning(boundary_positions: np.ndarray, masses: np.ndarray) -> float: + """Mean of the binning that holds `masses`, one mass per bin.""" + bin_centres = (boundary_positions[1:] + boundary_positions[:-1]) / 2 + return float(np.sum(masses * bin_centres)) + + @classmethod + def _solved_interior_weight( + cls, positions: np.ndarray, masses: np.ndarray + ) -> float: + """The unclamped boundary placement that makes the mean of the binning equal + the mean of the input, recomputed here so the test can tell whether + `PlotData.create_non_uniform_binning()` had to clamp. Recovering it from the + returned boundaries instead would compare a rounded quotient against the + clamp bound, which is not reliable at the bound itself.""" + target_mean = float(np.sum(positions * masses)) + mean_at_zero = cls._mean_of_binning( + PlotData._cell_boundary_positions(positions, 0.0), masses + ) + mean_at_one = cls._mean_of_binning( + PlotData._cell_boundary_positions(positions, 1.0), masses + ) + if abs(mean_at_one - mean_at_zero) < 1e-15: + return 0.5 + return (target_mean - mean_at_zero) / (mean_at_one - mean_at_zero) + + def test_create_non_uniform_binning_property_preserves_mean(self) -> None: + """ + Tests that the binning created by `PlotData.create_non_uniform_binning()` + has the same total mass and the same mean as the input Dirac deltas. + + Unevenly spaced Dirac deltas can need a boundary placement very close to one + of the two Dirac deltas the boundary separates. For a 3-Dirac-delta input, + for example, the mirrored extremal boundaries centre the outer two bins on + their Dirac deltas, which forces the placement to (p1 - p0) / (p2 - p0) + regardless of the masses. `MIN_INTERIOR_WEIGHT` therefore only has to keep + an extremal bin from collapsing to zero width, so the mean should come out + exact here. The property tested still allows for the clamp, requiring the + mean to be no worse than a plain midpoint placement when it does bind. + """ + probability_threshold: float = 1e-12 + position_threshold: float = 1e-9 + clamped_ensembles: int = 0 + total_ensembles: int = 0 + + for mass_distribution in ("equal", "uniform", "skewed"): + for input_ensemble in self._random_ensembles(mass_distribution): + positions = np.array([dd.position for dd in input_ensemble]) + masses = np.array([dd.mass for dd in input_ensemble]) + input_mean = float(np.sum(positions * masses)) + total_ensembles += 1 + + ( + boundary_positions, + bin_widths, + bin_heights, + ) = PlotData.create_non_uniform_binning(input_ensemble) + + probabilities = bin_widths * bin_heights + binning_mean = self._mean_of_binning(boundary_positions, masses) + self.assertLess( + abs(float(np.sum(probabilities)) - 1.0), probability_threshold + ) + + interior_weight = self._solved_interior_weight(positions, masses) + if ( + self.INTERIOR_WEIGHT_CLAMP + <= interior_weight + <= 1 - self.INTERIOR_WEIGHT_CLAMP + ): + self.assertLess( + abs(binning_mean - input_mean), + position_threshold * max(abs(input_mean), 1.0), + ) + continue + + clamped_ensembles += 1 + midpoint_mean = self._mean_of_binning( + PlotData._cell_boundary_positions(positions, 0.5), masses + ) + self.assertLessEqual( + abs(binning_mean - input_mean), + abs(midpoint_mean - input_mean) + + position_threshold * max(abs(input_mean), 1.0), + ) + + # Clamping should stay rare. If it becomes common, the placement solve has + # regressed rather than the inputs having become pathological. + self.assertLess(clamped_ensembles, total_ensembles // 100 + 1) + + def test_create_non_uniform_binning_preserves_local_means(self) -> None: + """Where the Dirac delta spacing admits it, every bin is centred on the + Dirac delta it holds, so the binning preserves each Dirac delta's own mean + and not just the mean of the whole distribution. This is what the TTR + binning method preserves, and what a single shared boundary placement + cannot: one scalar can only match one moment of the whole distribution.""" + rng = np.random.default_rng(3) + smoothly_spaced_positions = { + "uniform spacing": np.linspace(-5.0, 5.0, 64), + "geometric spacing": np.exp(np.linspace(-2.0, 2.5, 32)), + "quadratic spacing": np.linspace(0.0, 4.0, 24) ** 2, + "two Dirac deltas": np.array([0.0, 1.0]), + } + + for label, positions in smoothly_spaced_positions.items(): + with self.subTest(spacing=label): + masses = rng.uniform(0.5, 1.5, len(positions)) + masses /= masses.sum() + input_ensemble = [ + DiracDelta(position, mass=mass) + for position, mass in zip(positions, masses) + ] + + ( + boundary_positions, + bin_widths, + bin_heights, + ) = PlotData.create_non_uniform_binning(input_ensemble) + + bin_centres = (boundary_positions[1:] + boundary_positions[:-1]) / 2 + scale = float(positions[-1] - positions[0]) + self.assertLess( + float(np.max(np.abs(bin_centres - positions))), 1e-12 * scale + ) + self.assertTrue(np.all(bin_widths > 0)) + self.assertTrue( + np.all(np.abs(bin_widths * bin_heights - masses) < 1e-15) + ) + self.assertAlmostEqual( + float(np.sum(bin_widths * bin_heights)), 1.0, places=12 + ) + + def test_local_mean_boundary_positions_reports_when_unsolvable(self) -> None: + """The recurrence that centres every bin on its Dirac delta is undamped, so + alternating gaps drive a boundary out from between the two Dirac deltas it + separates. There is then no solution, which must be reported rather than + returned as an invalid binning; the caller falls back to a single shared + boundary placement, which still preserves mass and the mean.""" + positions = np.array([0.0, 10.0, 10.5, 20.5, 21.0, 31.0, 31.5]) + self.assertIsNone(PlotData._local_mean_boundary_positions(positions)) + + masses = np.full(len(positions), 1.0 / len(positions)) + input_ensemble = [ + DiracDelta(position, mass=mass) for position, mass in zip(positions, masses) + ] + input_mean = float(np.sum(positions * masses)) + + ( + boundary_positions, + bin_widths, + bin_heights, + ) = PlotData.create_non_uniform_binning(input_ensemble) + + self.assertTrue(np.all(bin_widths > 0)) + self.assertAlmostEqual(float(np.sum(bin_widths * bin_heights)), 1.0, places=12) + self.assertLess( + abs(self._mean_of_binning(boundary_positions, masses) - input_mean), + 1e-9 * max(abs(input_mean), 1.0), + ) + + def test_local_mean_boundary_positions_centres_every_bin(self) -> None: + """The solved boundaries must centre each bin on its Dirac delta and stay + strictly between the Dirac deltas they separate.""" + positions = np.array([0.0, 1.0, 2.5, 4.0, 4.5]) + + boundary_positions = PlotData._local_mean_boundary_positions(positions) + + self.assertIsNotNone(boundary_positions) + assert boundary_positions is not None + bin_centres = (boundary_positions[1:] + boundary_positions[:-1]) / 2 + self.assertTrue(np.allclose(bin_centres, positions, atol=1e-12)) + self.assertTrue(np.all(boundary_positions[1:] > boundary_positions[:-1])) + self.assertTrue(np.all(positions > boundary_positions[:-1])) + self.assertTrue(np.all(positions < boundary_positions[1:])) + + def test_create_non_uniform_binning_preserves_mean_when_dirac_deltas_crowd( + self, + ) -> None: + """An interior Dirac delta very close to one neighbour needs boundaries very + close to the interval ends. A single shared placement solved for the whole + distribution's mean puts it at ~0.0034 here, which a large clamp would block + and perturb the mean by percent. Centring every bin on its own Dirac delta + is solvable for this spacing, so both the local means and the mean of the + whole distribution must come out exact.""" + positions = np.array([-38.75449121, -38.44464168, 51.59049523]) + masses = np.full(3, 1.0 / 3.0) + input_ensemble = [ + DiracDelta(position, mass=mass) for position, mass in zip(positions, masses) + ] + input_mean = float(np.sum(positions * masses)) + + # A single shared placement would have to sit here, well inside the clamp + # that used to block it. + interior_weight = self._solved_interior_weight(positions, masses) + self.assertLess(interior_weight, 0.01) + self.assertGreater(interior_weight, PlotData.MIN_INTERIOR_WEIGHT) + + ( + boundary_positions, + bin_widths, + bin_heights, + ) = PlotData.create_non_uniform_binning(input_ensemble) + + self.assertTrue(np.all(bin_widths > 0)) + self.assertAlmostEqual(float(np.sum(bin_widths * bin_heights)), 1.0, places=12) + # The mean must be exact, not merely close: a clamp that blocked this + # placement left it 1.2% out. + self.assertLess( + abs(self._mean_of_binning(boundary_positions, masses) - input_mean), + 1e-12 * abs(input_mean), + ) + bin_centres = (boundary_positions[1:] + boundary_positions[:-1]) / 2 + self.assertTrue(np.allclose(bin_centres, positions, atol=1e-12)) + + def test_create_non_uniform_binning_rejects_fewer_than_two_dirac_deltas( + self, + ) -> None: + """A lone Dirac delta has no adjacent Dirac delta to place a boundary + against, so it has no bin width. That must fail loudly rather than return + `NaN`-valued boundaries, widths, and heights.""" + for input_ensemble in ([], [DiracDelta(2.5, mass=1.0)]): + with self.subTest(number_of_dirac_deltas=len(input_ensemble)): + with self.assertRaises(ValueError) as raised: + PlotData.create_non_uniform_binning(input_ensemble) + self.assertIn("at least two Dirac deltas", str(raised.exception)) + + for positions in (np.array([]), np.array([2.5])): + with self.subTest(number_of_positions=len(positions)): + with self.assertRaises(ValueError) as raised: + PlotData._cell_boundary_positions(positions, 0.5) + self.assertIn("at least two positions", str(raised.exception)) + + def test_single_dirac_delta_value_plots_without_binning(self) -> None: + """`PlotData` must not route a single-Dirac-delta value into any binning: + it plots the Dirac delta itself, with no `NaN` in the output.""" + dist = DistributionalValue(dirac_deltas=[DiracDelta(2.5, mass=1.0)]) + + pd = PlotData(dist) + + self.assertEqual(len(pd.positions), 1) + self.assertEqual(len(pd.masses), 1) + self.assertFalse(bool(np.any(np.isnan(pd.positions)))) + self.assertFalse(bool(np.any(np.isnan(pd.masses)))) + self.assertAlmostEqual(float(pd.positions[0]), 2.5, places=12) + + def test_create_non_uniform_binning_two_dirac_deltas(self) -> None: + """With two Dirac deltas the mirrored extremal boundaries centre each bin + on its Dirac delta, so the mean is exact for any boundary placement.""" + input_ensemble = [DiracDelta(0.0, mass=0.4), DiracDelta(1.0, mass=0.6)] + + ( + boundary_positions, + bin_widths, + bin_heights, + ) = PlotData.create_non_uniform_binning(input_ensemble) + + probabilities = bin_widths * bin_heights + bin_centres = (boundary_positions[1:] + boundary_positions[:-1]) / 2 + + self.assertEqual(len(bin_widths), 2) + self.assertAlmostEqual(float(np.sum(probabilities)), 1.0, places=12) + self.assertAlmostEqual( + float(np.sum(probabilities * bin_centres)), 0.6, places=12 + ) + + class TestPlotDataFromSamples(unittest.TestCase): """Tests for PlotData.from_samples() (delegates to DistributionalValue).""" @@ -285,19 +682,43 @@ def test_plot_from_samples_with_special_values_succeeds(self) -> None: self.assertTrue(result) -class TestNonTTRHistogramFallback(unittest.TestCase): - """Covers the uniform-width histogram fallback used when - `is_full_valid_TTR` is False or the TTR pipeline raises.""" +class TestNonTTRBinningFallback(unittest.TestCase): + """Covers the non-uniform binning fallback used when `is_full_valid_TTR` + is False or the TTR pipeline raises.""" @staticmethod def _fixture_path(name: str) -> str: here = os.path.realpath(os.path.dirname(__file__)) return os.path.join(here, name) - def test_invalid_ttr_ux_string_falls_back_to_uniform_histogram(self) -> None: + def _assert_gap_free_density(self, pd: PlotData, dist: DistributionalValue) -> None: + """The fallback binning has one bin per Dirac delta, holds no empty bins, + and preserves the total mass and the mean of the input exactly. Call this + after constructing the `PlotData`, which cures the `DistributionalValue`.""" + self.assertIsNotNone(pd.plotting_resolution) + assert pd.plotting_resolution is not None + assert pd.plotting_ttr_order is not None + expected_number_of_bins = min( + len(dist.finite_dirac_deltas), 2**pd.plotting_ttr_order + ) + self.assertEqual(len(pd.positions), expected_number_of_bins + 1) + self.assertEqual(len(pd.masses), expected_number_of_bins) + + widths = pd.positions[1:] - pd.positions[:-1] + self.assertTrue(np.all(widths > 0)) + self.assertTrue(np.all(pd.masses > 0)) + + total_area = float(np.sum(widths * pd.masses)) + self.assertAlmostEqual(total_area, 1.0, places=9) + + bin_centres = (pd.positions[1:] + pd.positions[:-1]) / 2 + binning_mean = float(np.sum(widths * pd.masses * bin_centres)) + assert dist.mean is not None + self.assertAlmostEqual(binning_mean, dist.mean, places=9) + + def test_invalid_ttr_ux_string_falls_back_to_non_uniform_binning(self) -> None: """The fixture is a Ux string that is not a valid TTR. The fallback - should produce a uniform-width histogram with shape matching - plotting_resolution.""" + should produce a gap-free, non-uniform-width binning.""" with open(self._fixture_path("invalid_ttr_ux_string.dat")) as f: ux_data = f.read().strip() dist = DistributionalValue.parse(ux_data) @@ -307,17 +728,12 @@ def test_invalid_ttr_ux_string_falls_back_to_uniform_histogram(self) -> None: pd = PlotData(dist) - self.assertIsNotNone(pd.plotting_resolution) - assert pd.plotting_resolution is not None - self.assertEqual(len(pd.positions), pd.plotting_resolution + 1) - self.assertEqual(len(pd.masses), pd.plotting_resolution) + self._assert_gap_free_density(pd, dist) + # The widths must genuinely vary. A uniform-width histogram is what this + # binning replaces. widths = pd.positions[1:] - pd.positions[:-1] - self.assertTrue(np.allclose(widths, widths[0])) - - # Density should integrate to ~1 over the finite-mass region. - total_area = float(np.sum(widths * pd.masses)) - self.assertAlmostEqual(total_area, 1.0, places=6) + self.assertFalse(np.allclose(widths, widths[0])) def test_clustered_repeated_positions_fallback_succeeds(self) -> None: """Heavy duplication / tight clusters must not crash the fallback. @@ -333,15 +749,38 @@ def test_clustered_repeated_positions_fallback_succeeds(self) -> None: ) pd = PlotData.from_samples(samples) - self.assertIsNotNone(pd.plotting_resolution) + self._assert_gap_free_density(pd, pd.dist) + + def test_raw_samples_are_reduced_to_plotting_ttr_order(self) -> None: + """With more Dirac deltas than the plot can resolve, the fallback reduces + them to the same number of Dirac deltas the valid-TTR path uses, and + still holds the invariants.""" + rng = np.random.default_rng(0) + pd = PlotData.from_samples(rng.normal(3.0, 1.5, 5000)) + + assert pd.plotting_ttr_order is not None + self.assertEqual(len(pd.masses), 2**pd.plotting_ttr_order) + self._assert_gap_free_density(pd, pd.dist) + + def test_non_uniform_binning_leaves_no_gaps_where_uniform_would(self) -> None: + """exp() applied to the positions of a valid normal TTR is not a valid + TTR. Its Dirac deltas span a wide range, so a uniform-width histogram + would leave most bins empty. The non-uniform binning leaves none.""" + positions = np.exp(np.linspace(-2.0, 2.5, 16)) + masses = np.full(16, 1.0 / 16) + dist = DistributionalValue.from_weighted_samples(positions, masses) + self.assertFalse(dist.is_full_valid_TTR) + + pd = PlotData(dist) + assert pd.plotting_resolution is not None - self.assertEqual(len(pd.positions), pd.plotting_resolution + 1) - self.assertEqual(len(pd.masses), pd.plotting_resolution) + uniform_edges = np.linspace( + float(positions[0]), float(positions[-1]), pd.plotting_resolution + 1 + ) + uniform_counts, _ = np.histogram(positions, bins=uniform_edges) + self.assertGreater(int(np.sum(uniform_counts == 0)), 0) - widths = pd.positions[1:] - pd.positions[:-1] - self.assertTrue(np.allclose(widths, widths[0])) - total_area = float(np.sum(widths * pd.masses)) - self.assertAlmostEqual(total_area, 1.0, places=6) + self._assert_gap_free_density(pd, dist) def dirac_deltas_to_ttr( diff --git a/src/signaloid/distributional_information_plotting/sample_generator.py b/src/signaloid/distributional_information_plotting/sample_generator.py index 895f372..edbcac9 100644 --- a/src/signaloid/distributional_information_plotting/sample_generator.py +++ b/src/signaloid/distributional_information_plotting/sample_generator.py @@ -144,8 +144,23 @@ def _sample_finite( """Sample from the finite part of a distributional value via inverse CDF.""" distributional_value.drop_zero_mass_positions() distributional_value.combine_dirac_deltas() + finite_deltas = distributional_value.finite_dirac_deltas + + if len(finite_deltas) == 0: + raise ValueError( + "sample_generator: the finite part of the distributional value carries " + "no mass, so there is nothing to sample from." + ) + + # Dropping zero-mass Dirac deltas and combining coincident ones can leave a + # single Dirac delta even when the input had many, e.g. when every Dirac + # delta shares one position. A lone Dirac delta has no binning to invert, so + # return identical copies of its position, as for a particle distribution. + if len(finite_deltas) == 1: + return np.full(n_samples, finite_deltas[0].position) + boundary_positions, bin_widths, bin_heights = PlotData.create_binning( - distributional_value.finite_dirac_deltas, 0, False + finite_deltas, 0, False ) cdf_values = generate_cdf_values(boundary_positions, bin_widths, bin_heights) return generate_samples(cdf_values, boundary_positions, n_samples) diff --git a/src/signaloid/distributional_information_plotting/sample_generator_test.py b/src/signaloid/distributional_information_plotting/sample_generator_test.py index b649489..3474817 100644 --- a/src/signaloid/distributional_information_plotting/sample_generator_test.py +++ b/src/signaloid/distributional_information_plotting/sample_generator_test.py @@ -22,13 +22,18 @@ import unittest import numpy as np +from signaloid.distributional.dirac_delta import DiracDelta from signaloid.distributional.distributional import DistributionalValue from signaloid.distributional_information_plotting.sample_generator import ( + sample_from_distributional_value, sample_generator, ) SAMPLE_UX_STRING = "-0.000000Ux040000000000000001BCB03C52D58D3CE400000020C0048D2279B8AFF701B1807E6239F600BFFFF1F7A03E82B602A7EB881EDAA6C0BFFAFF0EC92B7D6E0321FCB58BB7F440BFF763B747227C880386DDE1E09EAEC0BFF47A086FFA91B003BA2C11EF08E580BFF1FDCC06ED8E9703F77BE72797F440BFEF88C5501503BA0429DAF9A7420E80BFEB75E0A582D1610454636C25A09D40BFE7B77D572D585D0456D709533085C0BFE43A6FD58615450472E978D476CD40BFE0E4BE4A8177310489FC4176BD8E80BFDB5AB694DC2F6D049CBBFB39689F80BFD51A3EFE52F0A004AAD48A9FB27AC0BFCDF7B07C22983104B59D8153294540BFC1E9102D4ACC1304BCB0EDEE5C1900BFA7D59C0E0AD59404C0374A760BE0C03FA7D59C0E0AD5A204C0374A760BE0C03FC1E9102D4ACC0504BCB0EDEE5C1C803FCDF7B07C22983104B59D81532945403FD51A3EFE52F0A904AAD48A9FB277003FDB5AB694DC2F6D049CBBFB39689F803FE0E4BE4A8177310489FC4176BD8E803FE43A6FD586153C0472E978D476D0C03FE7B77D572D58680456D709533082403FEB75E0A582D1560454636C25A0A1003FEF88C5501503D10429DAF9A7420AC03FF1FDCC06ED8EA203F77BE72797F7E03FF47A086FFA91AE03BA2C11EF08DE603FF763B747227C810386DDE1E09EAB203FFAFF0EC92B7D710321FCB58BB7F4403FFFF1F7A03E82AE02A7EB881EDAA32040048D2279B8AFD801B1807E6239FD40" SAMPLE_UX_STRING_WITH_SPECIAL_VALUES = "nanUx0400000000000000017FF8000000000000000000063FF0000000000000155555555555550040080000000000001555555555555500000000000000000015555555555555007FF80000000000001555555555555500FFF000000000000015555555555555007FF00000000000001555555555555500" +# A well-formed Ux String whose 32 Dirac deltas all share the position 0.0, so +# the support collapses to a single point once coincident deltas are combined. +SAMPLE_UX_STRING_COINCIDENT_POSITIONS = "Ux040000000000000001000000000000000000000020000000000000000001B1807E6239F600000000000000000002A7EB881EDAA6C000000000000000000321FCB58BB7F44000000000000000000386DDE1E09EAEC0000000000000000003BA2C11EF08E580000000000000000003F77BE72797F44000000000000000000429DAF9A7420E8000000000000000000454636C25A09D4000000000000000000456D709533085C000000000000000000472E978D476CD4000000000000000000489FC4176BD8E800000000000000000049CBBFB39689F80000000000000000004AAD48A9FB27AC0000000000000000004B59D8153294540000000000000000004BCB0EDEE5C1900000000000000000004C0374A760BE0C0000000000000000004C0374A760BE0C0000000000000000004BCB0EDEE5C1C80000000000000000004B59D8153294540000000000000000004AAD48A9FB277000000000000000000049CBBFB39689F8000000000000000000489FC4176BD8E8000000000000000000472E978D476D0C000000000000000000456D7095330824000000000000000000454636C25A0A10000000000000000000429DAF9A7420AC0000000000000000003F77BE72797F7E0000000000000000003BA2C11EF08DE6000000000000000000386DDE1E09EAB2000000000000000000321FCB58BB7F440000000000000000002A7EB881EDAA320000000000000000001B1807E6239FD40" # ============================================================================ @@ -61,6 +66,43 @@ def test_samples_are_finite(self) -> None: self.assertIsInstance(samples, np.ndarray) self.assertTrue(np.all(np.isfinite(samples))) + def test_coincident_positions_returns_identical_samples(self) -> None: + """Many Dirac deltas at one position should return n identical samples.""" + n = 10 + samples = sample_generator(SAMPLE_UX_STRING_COINCIDENT_POSITIONS, n) + + self.assertEqual(len(samples), n) + self.assertTrue(np.all(samples == 0.0)) + + def test_coincident_positions_with_special_values(self) -> None: + """A collapsed finite support mixed with NaN mass should still sample.""" + # Dirac deltas that all share one position, plus mass on NaN, so both + # branches of the mixed case run against a support that collapses to a + # single position. + dirac_deltas = [DiracDelta(0.0, mass=0.9 / 32) for _ in range(32)] + dirac_deltas.append(DiracDelta(np.nan, mass=0.1)) + dist_value = DistributionalValue(dirac_deltas=dirac_deltas) + + n = 1_000 + np.random.seed(42) + samples = sample_from_distributional_value(dist_value, n) + + self.assertEqual(len(samples), n) + self.assertGreater(np.sum(np.isnan(samples)), 0) + self.assertTrue(np.all(samples[~np.isnan(samples)] == 0.0)) + + def test_zero_finite_mass_raises(self) -> None: + """A distributional value with no mass at all should raise ValueError.""" + dist_value = DistributionalValue.parse(SAMPLE_UX_STRING) + if dist_value is None: + raise ValueError("Failed to parse Ux-data into DistributionalValue.") + + for dd in dist_value.dirac_deltas: + dd.mass = 0.0 + + with self.assertRaises(ValueError): + sample_from_distributional_value(dist_value, 10) + def test_invalid_ux_data_raises(self) -> None: """Should raise ValueError for invalid Ux-data.""" with self.assertRaises(ValueError): diff --git a/src/signaloid/out.dat b/src/signaloid/out.dat deleted file mode 100644 index b691540..0000000 --- a/src/signaloid/out.dat +++ /dev/null @@ -1,100 +0,0 @@ -4.850327178256621963e+00 -6.848267583883947296e+00 -5.210944579836551682e+00 -3.233215222075699113e+00 -5.897096941853664731e+00 -7.629650956655302352e+00 --4.953898141910515474e-01 -8.159821201587657669e+00 -6.006895592063975720e+00 -6.005740168534201118e+00 -9.237675468825123914e+00 --3.927877756397357700e-03 -9.041611702607381673e+00 -1.387341217391892645e+00 -8.136739212764434459e-01 -4.172044836708691307e+00 -1.165638609758492095e+00 -3.582125401037927759e+00 -4.726803020674331002e+00 -4.089668950066265851e-01 -4.290889137496524341e+00 -4.315675385108691309e+00 -6.084072557755018984e+00 -1.501367064049112576e+00 -7.887318597398504938e-01 --4.714694690751508599e-01 -1.774107057684483735e+00 -4.412607631170569533e+00 -5.068356325948126795e+00 -6.810884052655161724e-01 -5.486939427818301240e+00 -7.690203179160623570e+00 -9.125742454449209617e+00 -7.500691358417268972e+00 -9.449057662002364744e-01 -2.320352556034750879e+00 -6.558301471772987057e+00 -3.262308075165714083e+00 --3.355407279487510053e-01 -5.479223783132843861e-01 -7.618472303592408679e+00 -3.240814646902165475e+00 --3.986416148417910588e-01 -8.608757383107379368e+00 -8.726771071390819756e+00 --2.680250878353067634e-01 -5.275396449850120462e+00 -5.563161463599430867e+00 -3.305879248055860753e+00 -6.434193623505916726e+00 -6.373809659036711928e+00 -7.257519079252347183e+00 -6.451236539484616062e-01 -9.470571935915295114e+00 -9.853678882348983592e+00 -3.502606197027343438e+00 -2.467936970214724024e+00 -1.817538028561818397e-01 --1.205341136862663198e-01 -6.737425662049654207e-01 --3.717207522914540707e-01 -9.319199936957627273e+00 -6.919068005103573560e-01 -4.554752531622908940e+00 -1.090777667429968067e+00 -1.298065098841230391e-01 --2.110047653085689312e-02 --7.104863659926397013e-02 -6.671657281880726487e+00 --2.795368917751783755e-04 -1.060418688491762129e+00 -8.014305724492746252e-01 -5.700968011507696609e+00 -3.773652623527242067e+00 -8.151733545793096170e-01 -6.855193590365169509e-01 -9.307565985993798918e+00 -6.163988210345446639e+00 -9.841344749106384349e-01 -8.033904166517496392e+00 -1.388577360696183982e+00 --1.818559471708045550e-01 --4.765806299761057296e-01 -4.523139358203554505e-01 -5.541947359624813663e-01 -6.257373189328736984e+00 -8.050802390235787698e+00 -4.296624310624639898e-01 -8.656915363669950292e-01 -3.659973670032748316e+00 -2.716808935991826157e+00 -7.468186708887329495e+00 -9.546506192431326809e+00 -9.976619302398788136e+00 -6.751524385360343494e+00 -5.080552971376557814e+00 -7.552074895585523251e-01 -6.669656002944876150e+00 -1.820740770428783684e+00 -6.798112745408566582e-01 diff --git a/src/signaloid/plot.png b/src/signaloid/plot.png deleted file mode 100644 index 6405afd..0000000 Binary files a/src/signaloid/plot.png and /dev/null differ