Skip to content

pythonTest.md

Latest commit

 

History

History
84 lines (59 loc) · 3.97 KB

pythonTest.md

File metadata and controls

84 lines (59 loc) · 3.97 KB
Raw

Python Co-Simulation Testing

spinalML uses a rigorous testing environment combining Python, Cocotb, and Verilator. The goal of this test suite is to ensure that the mathematical models generated by the hardware (ROMs, PWL, and LUT algorithms) perfectly match the mathematical expectations computed by our "Golden Models" in Python.

Prerequisites and Installation (WSL / Ubuntu)

We use uv as a package and virtual environment manager for its speed and reliability. Ensure that Python 3.11+ and uv are installed on your system.

# 1. Create the Python virtual environment (uses system default version)
uv venv

# 2. Activate the environment
source .venv/bin/activate

# 3. Install dependencies (Cocotb, Pytest, Numpy, etc.)
uv pip install -r requirements.txt

Note

SpinalHDL requires the mill build tool to compile Scala into Verilog. Ensure that the ./mill script at the repository root is executable. Icarus Verilog (sudo apt install iverilog) and Verilator (sudo apt install verilator) must also be installed on your system for co-simulation.

Testing Architecture

The test suite is located in the tests/python/ directory.

1. Golden Models (golden_models/)

Contains pure Python mathematical models that simulate the expected hardware behavior:

  • dtypes.py: Simulates the behavior of data types (e.g., FloatML, SIntML) with their strict rounding and overflow rules.
  • ops.py: Replicates the exact behavior of hardware operators (LUTs, PWL interpolations, addition of the 1e-5 bias, etc.).

2. Utilities (utils/tb_utils.py)

Contains the necessary machinery to:

  • Run mill to generate Verilog code.
  • Copy the generated ROMs (.bin files) into the Verilator build directory.
  • Clean up the workspace after each test (via the cleanup_verilog pytest fixture).

3. Test Files (e.g., test_dtypes.py, test_rsqrt.py)

Each file tests a specific hardware operator or module using pytest and cocotb.

Running a Test

To run a specific test (for example, verifying data types and quantization):

pytest tests/python/test_dtypes.py -s

To run a hardware operator test (e.g., Rsqrt) across all supported quantizations (I4, FP4, I16, BF16):

pytest tests/python/test_rsqrt.py -s

Execution Flow Example (Rsqrt):

  1. pytest detects the test function.
  2. Our Python function calls ./mill spinalML.test.testOnly spinalML.ops.RsqrtTest to generate the RsqrtTestComp.v component.
  3. Verilator compiles the .v file into C++.
  4. Cocotb injects the test stimuli (e.g., $1.0, 4.0, 10.0$) bit by bit into the module ports via the simulator.
  5. Cocotb reads the calculated hardware result and asserts it against the Python Golden Model (ops.py).

Debugging Math: --debug-math

To inspect how close hardware results are to the golden models (even when not bit-exact, e.g., for floating-point types like FP8/BF16), run with:

pytest tests/python/test_matmul.py -s --debug-math

Each op x dtype then logs a single aggregated metrics line, for example:

[MatmulBatched][BF16] Test | MAE: 0.0151 | MAPE: 1.20% | NMSE: 0.001 | Cosine: 0.999 (2x1x2 @ 2x1, lanes=2, trials=3, seed=42)
  • MAE (absolute mean error, on dequantized values) and MAPE (floats, true != 0) or MAE %FS (integers, relative to full scale 2^(bits-1)-1)
  • NMSE = MSE / Var(true) and Cosine similarity over flattened vectors
  • For softmax-like outputs (e.g., attention) the true values are tiny, so MAPE/NMSE/Cosine can look extreme even when the absolute error is small — trust MAE / MAE %FS there
  • The (...) suffix gives the tested geometry and the random seed; trials is the number of random draws (3 for random inputs, 1 for fixed values)
  • All lines are aggregated into a single file: tests/true_math_errors.log (truncated at start of each run, only written when --debug-math is passed)

Random inputs are reproducible: the seed defaults to 42 and can be overridden via the SPINALML_SEED environment variable.