5. Example usage

SAR Validation Analysis Tutorial

This guide demonstrates how to perform a SAR validation analysis on a random sample data set, compare it with classical statistical inference methods, and visualize the results.

1. Import packages and prepare your data as numpy arrays

First, we import the necessary libraries and classes required for the analysis. We will use numpy for data generation and sarlib for the validation analysis. To keep the code concise, we explicitly import the SAR, OLS, and SampleSizeAnalysis classes, along with the show_scatter function from sarlib. Moreover, we generate a synthetic dataset consisting of 1,000 samples and 4 predictors.

from sarlib import SAR, OLS, SampleSizeAnalysis, show_scatter
import numpy as np
x = np.random.randn(100, 3)  # predictors
y = np.random.randn(100)     # response

2. Visualize data

We begin by plotting the scatter matrix of the dataset for all considered predictors. In this instance, we execute the function without adding custom axis labels or blocking the code execution.

show_scatter(x, y)

3. Fit SAR model

Next, we fit the SAR model to our data. This allows us to compute the validation metrics derived from the SAR analysis, which serve two primary purposes:

1. Making an informed decision regarding model validation.
2. Comparing the results against classical statistical inference methods (such as the F-test).

model_sar = SAR(n_realiz=100, norm='epsins', alpha=0.05)
stats_sar = model_sar.fit(x, y, verbose=True)

The resulting SAR validation metrics are summarized in the table below:

4. Compare with an OLS model

To evaluate the SAR framework against traditional statistical inference, we fit an Ordinary Least Squares (OLS) model to the same dataset. This step extracts classical validation metrics, focusing primarily on the F-test p-value.

model_ols = OLS(n_realiz=100, alpha=0.05)
stats_ols = model_ols.fit(x, y, verbose=True)

The classical validation metrics are displayed in the following table:

5. Analyze sample size effect

5.1 Visualizing SAR Validation Results

Finally, we plot the comprehensive results obtained from the SAR validation method. The following figures illustrate the training/validation losses alongside the decision threshold, the p-values for the SAR hypothesis test, and the estimated model coefficients, all analyzed as a function of the sample size.

analysis = SampleSizeAnalysis(model_sar, x, y, steps=7)

Paragraph alignment of the generated figures:

analysis.plot_loss()

,

analysis.plot_pvalue()

,

analysis.plot_coef()

5.2 Visualizing Classical Validation Results (Optional)

If you wish to visualize the results corresponding to the classical approach (associated with the OLS model), you only need to implement the following code block:

analysis = SampleSizeAnalysis(model_ols, x, y, steps=7)

Executing this code generates the following plots, which display the F-test metrics for the OLS model and the behavior of the obtained coefficients across different sample sizes:

analysis.plot_pvalue()

,

analysis.plot_coef()