fastforest

Fast approximate-forest regression and multiclass classification in Rust, with Python bindings. It can quickly and accurately fit datasets with arbitrarily large row counts (millions of rows or more), and scales down to tiny datasets too.

Across nineteen numeric and mixed-data benchmarks spanning 1,030 to 20,216,100 rows and covering regression, binary classification, and multiclass classification, FastForest is always either the fastest to train and predict, or the most accurate. For more results, see the benchmarks section.

Regression

Dataset	Model	RMSE ↓	R² ↑	Fit (s) ↓	Predict (s) ↓
SGEMM GPU 241,600 rows · 14 features · numeric	fastforest	0.03	1.00	0.13	0.013
	autogrow	0.03	1.00	0.83	0.065
	sklearn RF	0.03	1.00	1.69	0.116
	sklearn HistGBM	0.20	0.97	0.62	0.012
Rossmann Store Sales 844,338 rows · 16 features · mixed	fastforest	0.13	0.90	0.81	0.020
	AutoForest	0.13	0.91	6.21	0.023
	autogrow	0.13	0.91	13.88	0.057
	sklearn RF	0.26	0.61	17.30	0.069
	sklearn HistGBM	0.30	0.46	1.60	0.041

_{Bold is best for that dataset and metric. AutoForest includes automatic sample sizing; autogrow additionally sizes the forest. All rows were measured on an Apple M5 Max; fit includes preprocessing.}

Classification

Dataset	Model	F1 acc ↑	Log loss ↓	Fit (s) ↓	Proba (s) ↓
Covertype 581,012 rows · binary features	fastforest	0.93	0.15	1.92	0.053
	AutoForest	0.94	0.10	5.02	0.046
	autogrow	0.94	0.10	13.68	0.168
	sklearn RF	0.92	0.17	3.80	0.179
	sklearn HistGBM	0.74	0.57	1.44	0.036
Adult Census Income 48,842 rows · 14 features · mixed	fastforest	0.81	0.31	0.13	0.007
	autogrow	0.81	0.31	0.71	0.007
	sklearn RF	0.80	0.37	0.70	0.025
	sklearn HistGBM	0.82	0.27	0.56	0.018

F1 acc is macro-averaged F1, giving every class equal weight. Covertype is passed with its supplied binary features; FastForest bundles exclusive indicators automatically.

Install

pip install fastforest

This installs the Python library and the native fastforest-fit, fastforest-predict, fastforest-convert, and fastforest-compile executables.

Training and Validation

rng = np.random.default_rng(42)
X = rng.random((1_000, 6))
y = 4*X[:, 0] - 2*X[:, 1] + X[:, 5]

ff = FastForest(seed=42, oob=True).fit(X, y)
preds = ff.predict(X[:5])
preds

array([3.0069957, 2.3236141, 1.0881499, 2.9502003, 3.3705614],
      dtype=float32)

labels = np.where(X[:, 0]+X[:, 1] > 1, "high", "low")
ffc = FastForestClassifier(seed=42, oob=True).fit(X, labels)
probs,classes = ffc.predict_proba(X[:5]), ffc.predict(X[:5])
classes

array(['high', 'high', 'high', 'high', 'low'], dtype='<U4')

X may contain numeric values, numeric strings, ordinary strings, and configured missing values. Regression y is converted to contiguous float32 and must be finite. Classification labels may be numeric or strings; classes_ records their probability-column order. Missing labels and single-class targets are rejected.

Automatic sizing

AutoForest and AutoForestClassifier size the samples while retaining the ordinary estimator API; autogrow=True also sizes the forest:

from fastforest.auto import AutoForest,AutoForestClassifier

model = AutoForest(seed=42).fit(X, y)
classifier = AutoForestClassifier(seed=42).fit(X, labels)
grown = AutoForest(autogrow=True, seed=42).fit(X, y)

For sufficiently large data, sizing fits eight-tree forests at larger bootstrap_max, max_node_samples, and max_features levels and scores each on a held-out tracking sample of at most 40,000 rows per output, never more than a fifth of the data. Each extra level requires another 1% reduction in tracking loss, independently on each axis. Both modes try bootstrap limits of 320k and 480k, skipping any level beyond the per-output row count; ordinary sizing tries node samples of 640 and 1280, autogrow widens these to 640, 1280, and 1920, and both try feature fractions of .8 and 1.0 above the task default. Bootstrap sizing runs only when rows exceed 1.8 * bootstrap_max * max(1, classes-1): 288,000 rows for regression and binary classification at the default bootstrap_max, scaling with the class count.

Without order=, when any column keeps a target statistic, the screen also tries target_statistics=False and disables the statistics when that trial improves tracking loss by at least 1%. Frozen level statistics are reliable yet sometimes redundant: when the other features already carry the same information, splits spent re-encoding it cost accuracy, and no fit-time gate predicts this as well as simply measuring it. This statistics trial runs even below the bootstrap sizing threshold, with the node and feature levels then probed alongside it. Ordered data keeps its rank-correlation gate instead: a random holdout would leak future rows and overrate identity-like statistics.

By default, the final model uses fastforest’s ordinary adaptive 32–64 tree rule and does not enable OOB. With autogrow=True, it instead grows in 32-tree batches. An independent random set of at most 40,000 tracking rows per output is fixed before the first batch; at every checkpoint, each row uses only trees for which it was out-of-bag. Another batch is added while cumulative regression MSE or classification Brier loss improves by at least 1%; the first batch that fails this test is discarded by default. Growth is capped at 512 trees by default; keep_last_batch, min_improvement, tree_batch_size, and max_trees control these choices.

Creating and Using Models

Models can be saved as compact, portable .ffm files containing the forest, fitted preprocessing schema, task, and class labels. A loaded model supports ordinary in-memory prediction as well as bounded file prediction:

from fastforest import load

model.save("model.ffm")
restored = load("model.ffm")
predictions = restored.predict(X)
restored.predict_file("test.csv", "predictions.csv")
restored.save_executable("model-predict")

predict_file processes CSV or Arrow IPC/Feather in bounded batches rather than loading the whole input. save_executable builds a standalone predictor for the current platform, embedding both the model and Rust prediction runtime; building it requires a Rust toolchain, but running it requires neither Python nor a separate model file.

Installing fastforest also provides four commands. Their parsing, preprocessing, fitting, persistence, and prediction run in Rust:

fastforest-fit train.csv --target price --task regression --output model.ffm
fastforest-predict model.ffm test.csv --output predictions.csv
fastforest-convert numeric.csv --output numeric.arrow
fastforest-compile model.ffm --output model-predict
./model-predict test.csv --output predictions.csv

fastforest-fit accepts mixed CSV or numeric Arrow input and supports regression and classification; classification prediction accepts --proba. fastforest-convert streams numeric CSV into standard Arrow IPC for faster repeated ingestion. Run any command with --help for its complete estimator, schema, and batching options.

The native fastforest-predict binary, using a default model trained on an 80% Concrete Strength split, predicts from Arrow end-to-end in 4.5 ms for one row and 4.9 ms for all 206 validation rows. Reproduce it with python tools/cli_bench.py.

Benchmarking

This section contains additional results; all benchmarks, including those at the top of the README, follow the approaches described here. Unless noted otherwise, results use one reproducible 80/20 split, stratified for classification. Fit timing includes model construction, schema inspection, preprocessing, and fitting, but excludes process startup and inter-process transfer. Prediction timing includes input transformation. Every model/dataset combination has a 180-second limit.

Each AutoForest row uses the ordinary adaptive tree count. Its following autogrow row uses the same sample sizer with growth capped at 192 trees. Both include the sizing screen and final fit in fit time, and appear only when training rows exceed the sample-sizer activation threshold.

Regression

Dataset	Model	RMSE ↓	R² ↑	Fit (s) ↓	Predict (s) ↓
California Housing 20,640 rows · 8 features	fastforest	0.49	0.82	0.09	0.003
	AutoForest	0.49	0.82	0.19	0.002
	autogrow	0.49	0.82	0.38	0.004
	sklearn RF	0.51	0.80	0.33	0.014
	sklearn HistGBM	0.47	0.83	0.43	0.003
Concrete Strength 1,030 rows · 8 features	fastforest	5.80	0.87	0.01	0.000
	AutoForest	5.70	0.87	0.01	0.000
	autogrow	5.66	0.88	0.02	0.001
	sklearn RF	5.46	0.88	0.04	0.014
	sklearn HistGBM	4.65	0.92	0.36	0.002
Diamonds 53,940 rows · 9 features	fastforest	553	0.98	0.17	0.007
	AutoForest	561	0.98	0.34	0.006
	autogrow	545	0.98	0.87	0.016
	sklearn RF	550	0.98	0.74	0.032
	sklearn HistGBM	541	0.98	0.49	0.012
Allstate Claims 188,318 rows · 130 features	fastforest	1,940	0.54	2.28	0.039
	AutoForest	1,930	0.54	5.54	0.038
	autogrow	1,903	0.56	10.57	0.084
	sklearn RF	timed out
	sklearn HistGBM	1,861	0.58	2.84	0.325
Diabetes 130-US Hospitals 101,766 rows · 46 features	fastforest	2.18	0.45	0.51	0.018
	autogrow	2.15	0.47	3.36	0.038
	sklearn RF	2.20	0.45	4.47	0.141
	sklearn HistGBM	2.13	0.48	1.26	0.115
Blue Book for Bulldozers 412,698 rows · 52 features	fastforest	0.23	0.90	1.37	0.013
	AutoForest	0.23	0.90	6.86	0.012
	autogrow	0.23	0.90	15.15	0.021
	sklearn RF	timed out
	sklearn HistGBM	0.25	0.89	3.74	0.076
Walmart Store Sales 421,570 rows · 15 features	fastforest	2,909	0.98	0.62	0.014
	AutoForest	2,693	0.98	3.18	0.015
	autogrow	2,676	0.99	7.94	0.032
	sklearn RF	5,028	0.95	11.02	0.090
	sklearn HistGBM	6,604	0.91	1.29	0.048
ASHRAE Great Energy Predictor III 20,216,100 rows · 15 features	fastforest	0.86	0.84	1.26	0.628
	AutoForest	0.81	0.86	14.17	0.761
	autogrow	0.80	0.86	21.05	1.504
	sklearn RF	timed out
	sklearn HistGBM	1.43	0.55	23.76	0.709

For mixed data, the sklearn benchmarks use a custom pipeline based on scikit-learn’s official preprocessing guidance and examples: wholly numeric columns are parsed and median-imputed, categorical columns use one-hot encoding through 20 levels and target encoding above that, and HistGBM uses native categoricals through its 255-level limit. This numeric parsing is needed for sensible handling of raw CSV-like tables; otherwise the pipeline uses the documented sklearn behavior. fastforest requires no custom preprocessing and takes the original datasets directly. A timed-out cell marks a model that exceeded the 180-second per-model limit. For validation, Blue Book uses its final 12,000 rows, Walmart uses a 12-week chronological holdout to match the competition’s future-period forecasting setup, Rossmann uses its final six weeks, ASHRAE uses December 2016, and SF Crime uses its final 10% of incidents chronologically. On those five datasets the FastForest models set order= to the split column. The Target statistics section under Data preparation describes what the declared order changes.

Classification

Dataset	Model	F1 acc ↑	Log loss ↓	Fit (s) ↓	Proba (s) ↓
Bank Marketing 45,211 rows · 16 mixed features	fastforest	0.76	0.21	0.10	0.006
	autogrow	0.76	0.20	0.64	0.007
	sklearn RF	0.72	0.23	0.23	0.023
	sklearn HistGBM	0.76	0.20	0.51	0.017
Click Prediction Small 39,948 rows · 11 mixed features	fastforest	0.56	0.44	0.23	0.010
	AutoForest	0.55	0.44	0.55	0.010
	autogrow	0.54	0.42	0.89	0.011
	sklearn RF	0.54	0.44	0.37	0.022
	sklearn HistGBM	0.52	0.41	0.38	0.017
Statlog Shuttle 58,000 rows · 9 numeric features	fastforest	0.85	0.00	0.02	0.002
	AutoForest	0.95	0.00	0.07	0.002
	autogrow	0.95	0.00	0.14	0.003
	sklearn RF	0.85	0.00	0.17	0.016
	sklearn HistGBM	0.58	0.24	0.33	0.007
Airlines Delay 539,383 rows · 7 mixed features	fastforest	0.64	0.66	0.80	0.056
	autogrow	0.64	0.64	6.01	0.111
	sklearn RF	0.63	0.70	113.52	0.361
	sklearn HistGBM	0.64	0.62	1.38	0.078
HIGGS 1,000,000 rows · 28 numeric features	fastforest	0.72	0.54	1.93	0.062
	AutoForest	0.73	0.53	12.89	0.068
	autogrow	0.73	0.52	24.96	0.135
	sklearn RF	0.73	0.53	24.22	0.519
	sklearn HistGBM	0.73	0.53	1.95	0.064
SF Crime 878,049 rows · 6 mixed features	fastforest	0.06	2.78	8.00	0.082
	autogrow	0.06	2.43	31.79	0.153
	sklearn RF	0.06	4.64	19.58	0.499
	sklearn HistGBM	0.05	2.64	19.19	0.467
KDD Cup 1999 4,898,431 rows · 41 mixed features	fastforest	0.54	0.00	4.07	0.155
	AutoForest	0.67	0.00	21.44	0.156
	autogrow	0.73	0.00	33.31	0.308
	sklearn RF	0.67	0.00	61.07	1.948
	sklearn HistGBM	0.37	0.68	28.03	1.901

Reproducing the benchmarks

Install the development dependencies and release build, then reproduce one dataset with:

pip install -e '.[dev]'
cargo build --release --bins
python tools/stage_binaries.py
maturin develop --release
python tools/accuracy.py --dataset california

Available regression datasets are sgemm, california, concrete, diamonds, allstate, diabetes, bluebook, bluebook_raw, walmart, walmart_raw, ashrae, and rossmann. Classification choices are covertype, adult, bank, click, shuttle, airlines, higgs, sf_police, and kddcup99. Run a subset of models with --models, for example --models FastForest or --models RandomForest,HistGBM. Reproduce all displayed results with:

python tools/accuracy.py --datasets readme

Data preparation

FastForest fits a deterministic schema for every input column:

1. Non-missing values are parsed as float32 when every value can be parsed and are otherwise treated as strings. Numeric columns sort numerically and other columns sort naturally: digit runs inside strings compare numerically, so item2 precedes item10 (natural_sort=False restores plain lexical order). Numeric columns whose values are all integral retain that metadata so analysis displays them with no decimal places.
2. A constant column is discarded. Every other column becomes one zero-based rank in its sort order; binary columns are therefore ordinary boolean features.
3. The default missing value is the empty value. Override it per column with missing_values, using column names or indexes. Missing is encoded as a separate rank, and each split learns whether it belongs in its left or right child. No imputation or indicator column is added. By default, a column containing no training missing values rejects missing values during prediction; set allow_new_missing=True to route them to the larger child seen in the split’s sampled rows. Entirely missing columns are discarded.

X = np.array([
    ["18", "red",   ""],
    ["42", "blue",  "3.5"],
    ["31", "green", "2.0"],
], dtype=object)

model = FastForest(missing_values={2: ""}).fit(X, [1, 4, 3])

Binary columns with no missing values are checked for mutual exclusivity on at most 10,000 sampled training-pool rows. Compatible indicators are collapsed into one categorical feature when their bundle is active in more than half the sample. The fitted membership and order are saved with the model; importance, explanations, and partial dependence treat the bundle as one feature and column_info_ lists its members.

Date and time columns are detected by default from at most 200 random training-pool rows using a conservative list of common formats. Every sampled non-missing value must match; ambiguous day/month forms remain candidates until a value above 12 resolves them, with month-first used if they remain ambiguous. Detected formats are saved with the model and never inferred again during prediction. Date columns are expanded natively using the same parts as fastai’s add_datepart: year, month, ISO week, day, day-of-week, day-of-year, month/quarter/year boundary flags, hour, minute, second, and Unix elapsed seconds. Constant parts are discarded automatically, while missing or unparsable date values produce ordinary missing date parts.

Set date_columns={} to disable detection, or provide explicit strftime formats to override it:

model = FastForest(date_columns={"saledate":"%m/%d/%Y %H:%M"}).fit(X, y)

Ranking is a compact training representation, not a prediction-time requirement for numeric columns. After fitting, rank cutoffs are converted back to native numeric boundaries, so seen and unseen numeric values are compared directly without a rank lookup. Nonnumeric values are mapped through their fitted lexical ordering, with unseen values receiving their insertion rank. Missing numeric values remain NaN during native prediction and follow the direction stored in each split.

Python accepts pandas data frames, NumPy arrays, and Arrow tables, selects the bounded training pool first, converts only retained rows, and performs the bounded 200-row date-format check. The native CSV path likewise builds Arrow arrays only for retained rows, while Arrow IPC keeps its existing typed buffers. Full-column schema fitting and inference transformation then run in Rust behind the Arrow boundary, including numeric and lexical interpretation, missing values, categories, date expansion, and parallel column processing. The compact ranked training matrix and native-value prediction matrix remain internal implementation details.

Generated ranks and date parts remain internal. Feature importance, explanations, and partial-dependence results aggregate them back to the original column and display its original values. Fitted interpretations are available in model.column_info_.

For reproducible sklearn comparisons on the same raw dataframe, sklearn_preprocessor implements the policy used by the benchmark: wholly numeric columns are parsed and median-imputed, categorical columns are one-hot encoded through 20 levels and target encoded above 20, and explicitly supplied missing markers are converted to nulls.

from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import make_pipeline
from fastforest import sklearn_preprocessor

preprocess = sklearn_preprocessor(X_train, missing_values={"age":"?"})
model = make_pipeline(preprocess, RandomForestRegressor(n_jobs=-1))
model.fit(X_train, y_train)

Install the optional dependencies with pip install 'fastforest[sklearn]'.

Target statistics

By default, fitting also builds one frozen target statistic per qualifying categorical column. For regression, a level’s statistic is its mean target. For classification, it is the level’s class rates projected onto their principal axis across levels. For a binary target this is the level’s positive rate. The statistic for a level with fewer than min_rows_per_level training rows, and for values unseen during training, is the pooled value. Missing is an ordinary level, with its own statistic and its own count.

FastForest keeps a column’s statistic only when its reliability score reaches min_stat_agreement. With order= naming the column your rows arrive by, the score is the weighted rank correlation between the level orderings of earlier and later rows. On a chronological split, identity columns such as a store id keep their statistics and calendar columns lose them. Without order=, the score is how far differences between level statistics exceed within-level noise.

model = FastForest(order="saledate").fit(X, y)

The fitted table, saved with the model, is identical at training, out-of-bag evaluation, and prediction. frequency=True adds each level’s exact training count as another derived feature. natural_sort=False restores plain lexical text ordering. target_statistics=False disables the statistics. Importance, explanations, and partial dependence aggregate the derived features back to the original column, as they do for ranks and date parts.

Algorithm

Each regression tree draws min(floor(bootstrap_fraction * n_rows), bootstrap_max) training rows. Classification treats bootstrap_max as a per-output cap and therefore uses bootstrap_max * max(1, n_classes-1) total rows per tree. replacement=None adaptively samples with replacement below 10,000 regression rows or 40,000 classification rows, and otherwise without it; pass True or False to override this. When bootstrap_fraction=None, it resolves to 0.8 with OOB enabled and 1 otherwise. Fractions above 1 are supported with replacement; without replacement the maximum is 1. Pass bootstrap_max=None to disable the cap. At each node, the default histogram splitter:

1. A node with fewer than min_node_size rows, or whose first max_node_samples sampled targets are equal, becomes a leaf.
2. A random contiguous window containing at most max_node_samples of the node’s shuffled rows is selected.
3. The tree randomly selects the configured fraction of encoded features, with a minimum of one.
4. For each selected feature, the sampled rows are sorted by their encoded rank and every distinct observed boundary is evaluated. Regression minimizes size-weighted sample standard deviation, shrinking each child mean toward its parent by a three-row prior. Classification uses tree-frequency-weighted entropy. Missing values occupy the final contiguous rank range: the ordinary pass leaves them right, and a second ordered pass tries them left only when that range is nonempty. These scores penalize poorly supported small children directly; the only hard requirement is that both children are nonempty.
5. Every regression leaf predicts the mean target of all tree-sampled rows that reached it. A classification leaf stores their class-probability vector. Thus leaf fitting processes each tree’s capped sample once in total; it does not route the whole dataset through every tree.

By default, forest size targets two million sampled rows across its trees: n_trees = clamp(ceil(2_000_000 / sampled_rows_per_tree), 32, 64). Set n_trees to override it. The standard regression cap resolves to 32 trees, as does Covertype’s seven-class cap. Other defaults are minimum node size 8, all rows capped at 160,000 per output, 90% feature sampling for regression or 60% for classification, at most 320 evaluated rows per node, and a three-row regression split prior. Enabling OOB changes the default sampling fraction to 0.8 so every row can receive held-out predictions. Preprocessing and trees build in parallel over columns and trees respectively. Classification prediction divides rows into roughly four blocks per Rayon worker and calculates how many fitted trees fit in a conservative 512 KiB working-set budget, including nodes and leaf probabilities. It processes those cache-sized tree batches within each row block; small trees retain row locality, while large trees automatically become tree-major. Supplying seed makes the fitted forest deterministic regardless of parallel scheduling.

max_features accepts "sqrt" or a fraction in (0, 1]; its default is 0.9 for regression and 0.6 for classification.

FastForestClassifier.predict_proba averages the leaf probabilities over trees, while predict returns the corresponding original label. With OOB enabled, oob_decision_function_, oob_counts_, and OOB accuracy oob_score_ are available; oob_indices_ maps the bounded results to original training rows. Ordinary fitting remains bounded by the shared pool, per-output row cap, and max_node_samples rows per node.

Split selection and tuning

The histogram splitter is the production default. The original random-cutoff search remains available as a simpler teaching implementation:

model = FastForest(random_splitter=True, seed=42).fit(X, y)
fixed = FastForest(max_features="sqrt", seed=42).fit(X, y)

The histogram search randomly selects max_features, builds sparse target-statistic histograms from the node evaluation window, and checks every observed boundary for those features. The random splitter instead proposes random (feature, value) cutoffs, deduplicates them, and evaluates them on the same kind of node window. Its candidate count is controlled by cutoff_divisor; max_features is ignored when random_splitter=True.

The focused sweep tool takes comma-separated levels for every tree hyperparameter. The first value is the shared baseline and each later value creates one one-axis configuration. It compares an eight-tree batched OOB screen with ordinary resolved-tree fits on the dataset’s canonical validation split, recording OOB, validation, and both training losses in one per-dataset CSV:

python tools/sweep.py --dataset california

Out-of-bag predictions

OOB calculation is opt-in with oob=True. After fitting:

• oob_prediction_ contains each training row’s mean prediction from trees that did not sample that row.
• oob_counts_ contains the number of contributing trees.
• A row with no contributing tree has count zero and prediction NaN.
• Sampling without replacement at bootstrap_fraction=1.0 leaves no OOB rows, so all counts are zero and predictions are NaN.

Both attributes are None when OOB is disabled.

Model analysis

FastForest includes analysis tools with ordinary NumPy results. Data frames are accepted and supply feature names automatically; arrays use x0, x1, and so on. Sampling happens before Arrow conversion: permutation importance and feature relations use at most 5,000 rows, PDP/ICE uses 500, feature dependence uses 5,000, and drop-column importance uses at most 40,000 training and 5,000 validation rows by default. These limits are configurable through each function’s sampling arguments. Plot methods import matplotlib only when called.

The executable examples below use the 1,030-row Concrete Compressive Strength dataset, cached under data/.

Xc,yc = fetch_openml(data_id=44959, return_X_y=True, as_frame=True, data_home="../data")
Xc_train,Xc_valid,yc_train,yc_valid = train_test_split(Xc, yc, test_size=.2, random_state=42)
concrete = FastForest(seed=42, oob=True).fit(Xc_train, yc_train)

Importance

Use validation-set permutation importance by default. It measures the drop in model score after shuffling a feature without retraining:

importance = concrete.feature_importance(Xc_valid, yc_valid)
importance.plot();

Correlated features can substitute for one another and therefore look individually unimportant. Permute them together to measure their joint importance:

importance = model.feature_importance(X_valid, y_valid,
    features={"location": ["latitude", "longitude"]})

model.drop_column_importance(X_train, y_train, X_valid, y_valid) performs the slower complementary analysis: it refits the forest without each feature. It accepts the same features groups. model.split_importance() returns the nearly free, normalized training-time split-gain measure, but permutation or grouped permutation is preferable because split importance is biased by the available cutoffs and correlated predictors.

Individual predictions and uncertainty

explanation = concrete.explain(Xc_valid[:3])
explanation.row(0)

[('age', 365, 9.071619033813477),
 ('fine_aggregate', 670.0, 5.62085485458374),
 ('water', 228.0, -2.9426684379577637),
 ('blast_furnace_slag', 114.0, 2.7738962173461914),
 ('superplasticizer', 0.0, -1.4535503387451172),
 ('coarse_aggregate', 932.0, 0.7147500514984131),
 ('cement', 266.0, -0.1426078975200653),
 ('fly_ash', 0.0, 0.08987802267074585)]

explanation.plot(0);

tree_predictions = concrete.predict_trees(Xc_valid)
prediction_std = concrete.predict_std(Xc_valid)
prediction_std[:5]

array([ 3.8229113, 10.553153 ,  8.164038 ,  6.691792 ,  5.368883 ],
      dtype=float32)

For every row, prediction = bias + contributions.sum(). Contributions telescope through each tree’s decision path and are then averaged across trees. They explain this forest’s computation, not causality; correlated features can redistribute contributions between themselves.

Partial dependence and ICE

age = concrete.partial_dependence(Xc_train, "age")
age.plot();

age.plot(clusters=5);

interaction = concrete.partial_dependence(Xc_train, ["cement", "water"])
interaction.plot();

Partial dependence repeatedly replaces the selected feature values and averages the resulting predictions. ICE retains the individual prediction lines. These plots describe the fitted model rather than a causal intervention, and highly correlated features can produce unrealistic synthetic rows.

Grouped features aggregate as one feature:

enclosure = model.partial_dependence(X_train,
    {"enclosure": ["enclosure_ac", "enclosure_erops", "enclosure_orops"]})

Collinearity and redundancy

relations = feature_relations(Xc_train)
relations.groups(threshold=0.2)

[('cement',),
 ('blast_furnace_slag',),
 ('fly_ash',),
 ('water',),
 ('superplasticizer',),
 ('coarse_aggregate',),
 ('fine_aggregate',),
 ('age',)]

relations.plot_dendrogram();

relations.plot();

feature_dependence complements correlation: it measures how predictable each feature is from the others, in any nonlinear form the forest can capture.

dependence = feature_dependence(Xc_train)
dependence.predictability

array([ 0.91981745,  0.87221277,  0.93467766,  0.91137922,  0.9297995 ,
        0.89800131,  0.92628348, -0.02972996])

dependence.plot();

feature_relations uses tie-aware Spearman correlation and average linkage implemented directly with NumPy. feature_dependence detects nonlinear redundancy by treating each feature in turn as a target, fitting a small forest from the remaining features, and measuring grouped prediction and permutation dependence.