Principle:Rapidsai Cuml Model Explainability
| Knowledge Sources | |
|---|---|
| Domains | Machine_Learning, Explainability, Model_Interpretation |
| Last Updated | 2026-02-08 12:00 GMT |
Overview
Model explainability through SHAP (SHapley Additive exPlanations) is a game-theoretic framework that assigns each feature an importance value (Shapley value) for a particular prediction, providing consistent and locally accurate explanations for any machine learning model.
Description
Modern machine learning models often function as black boxes -- they produce accurate predictions but offer no insight into why a particular prediction was made. SHAP addresses this by grounding feature importance in cooperative game theory, specifically Shapley values from 1953. The core idea is to treat each feature as a "player" in a cooperative game where the "payout" is the model prediction, and to fairly distribute credit among features based on their marginal contributions across all possible feature coalitions.
Shapley Values: For each feature, the Shapley value is the average marginal contribution of that feature across all possible orderings of features. This satisfies four desirable properties: efficiency (values sum to the difference between the prediction and the expected value), symmetry (features with equal contributions receive equal values), linearity (values are additive across models), and the null player property (features that never change the prediction receive zero value).
KernelSHAP: A model-agnostic method that approximates Shapley values using a weighted linear regression approach. It samples subsets of features, evaluates the model with each subset (replacing absent features with background data values), and fits a weighted linear model where the weights are derived from the Shapley kernel. KernelSHAP works with any model but is computationally expensive for large feature sets because the number of possible coalitions grows exponentially. A background dataset (summary of the training data) is used to marginalize over absent features; k-means sampling of the background dataset reduces computation without substantially degrading explanation quality.
PermutationSHAP: Computes exact Shapley values by evaluating the model on all permutations of features (or a sampled subset of permutations). For each permutation, features are added one at a time and the marginal contribution is recorded. Averaging over permutations gives the Shapley value. This method avoids the weighting approximation of KernelSHAP but requires many model evaluations.
TreeSHAP: An exact, polynomial-time algorithm specifically for tree-based models (decision trees, random forests, gradient boosted trees). It exploits the recursive partitioning structure of trees to compute exact Shapley values in O(T * L * D^2) time, where T is the number of trees, L is the maximum number of leaves, and D is the maximum depth. TreeSHAP can compute either interventional or tree-path-dependent explanations.
Usage
Model explainability is the right choice when:
- Stakeholders need to understand why a model made a specific prediction (loan approval, medical diagnosis, fraud detection).
- Regulatory requirements demand model transparency and auditability.
- Feature importance analysis is needed to guide feature engineering or data collection.
- Model debugging is required to identify whether the model is relying on spurious correlations.
- Comparing explanations across different model types using a unified framework.
Use KernelSHAP for arbitrary black-box models. Use TreeSHAP when the model is tree-based, as it provides exact values with dramatically lower computational cost. Use PermutationSHAP when exact values are needed for non-tree models and computational budget allows. Use k-means sampling to create a compact background dataset when the training set is large.
Theoretical Basis
Shapley Value Definition:
where is the set of all features, is a subset not containing feature , and is the model prediction when only features in are present (absent features are marginalized out).
KernelSHAP Approximation:
Given model f, instance x, background dataset D:
1. Sample M coalitions z' in {0, 1}^d (binary mask of included features)
2. For each coalition z':
- Construct input x_z' by replacing absent features with values from D
- Compute f(x_z') averaged over background samples
- Compute SHAP kernel weight:
pi(z') = (d - 1) / (C(d, |z'|) * |z'| * (d - |z'|))
3. Solve weighted linear regression:
phi = argmin sum_{z'} pi(z') * (f(x_z') - phi_0 - sum_j phi_j * z'_j)^2
phi_0 = E[f(x)] (base value)
phi_j = SHAP value for feature j
TreeSHAP (Recursive Algorithm):
function TreeSHAP(node, x, phi):
if node is leaf:
for each feature j:
phi[j] += weight * (contribution of feature j along this path)
return
feature = node.split_feature
threshold = node.split_threshold
if x[feature] <= threshold:
recurse into left child, extending the path
else:
recurse into right child, extending the path
The path-tracking data structure records:
- which features have been encountered
- the fraction of training data flowing through each branch
- the conditional contribution of each feature split
K-Means Background Sampling:
Given training set X with n samples:
D_background = KMeans(X, k=nsamples).cluster_centers_
weights = cluster_sizes / n
This reduces background dataset from n to k representative points,
making KernelSHAP O(k) instead of O(n) per coalition evaluation.