Implementation:Scikit learn Scikit learn Permutation Importance

Overview

Concrete tool for computing model-agnostic feature importance by permutation provided by scikit-learn. The Template:Code function evaluates a fitted estimator's reliance on each feature by measuring how much the model's scoring metric degrades when each feature column is randomly shuffled. This approach works with any fitted estimator and any scoring metric, making it a versatile alternative to the impurity-based Template:Code property available on tree-based models.

Function Signature

from sklearn.inspection import permutation_importance

permutation_importance(
    estimator,
    X,
    y,
    *,
    scoring=None,
    n_repeats=5,
    n_jobs=None,
    random_state=None,
    sample_weight=None,
    max_samples=1.0,
)

Parameters

estimator (object) -- An estimator that has already been fitted and is compatible with the scorer.
X (ndarray or DataFrame, shape (n_samples, n_features)) -- Data on which permutation importance will be computed. Can be the training set or a hold-out set.
y (array-like or None, shape (n_samples,) or (n_samples, n_classes)) -- Targets for supervised tasks, or None for unsupervised.
scoring (str, callable, list, tuple, or dict, default=None) -- Scorer to use. If None, the estimator's default scoring criterion is used. Can be a single string, a callable with signature Template:Code, or a collection of scorers for multi-metric evaluation.
n_repeats (int, default=5) -- Number of times to permute each feature. Higher values yield more stable importance estimates but increase computation time.
n_jobs (int or None, default=None) -- Number of jobs to run in parallel. Computation is parallelized over the feature columns. None means 1 unless in a Template:Code context. -1 means using all processors.
random_state (int or RandomState instance, default=None) -- Pseudo-random number generator to control the permutations of each feature. Pass an int for reproducible results.
sample_weight (array-like of shape (n_samples,), default=None) -- Sample weights used in scoring.
max_samples (int or float, default=1.0) -- The number of samples to draw from X to compute feature importance in each repeat (without replacement). If float, draws Template:Code samples. If 1.0 or Template:Code, all samples are used. Subsampling keeps the method tractable on large datasets.

Return Value

The function returns a Template:Code object (or a dict of Template:Code objects when multiple scorers are used) with the following attributes:

importances_mean -- Mean of feature importance over Template:Code (ndarray of shape Template:Code).
importances_std -- Standard deviation over Template:Code (ndarray of shape Template:Code).
importances -- Raw permutation importance scores (ndarray of shape Template:Code).

When multiple scoring metrics are provided, the result is a dictionary keyed by scorer name, with each value being a Template:Code as described above.

Impurity-Based Alternative

For tree-based ensemble models (such as Template:Code and Template:Code), an impurity-based feature importance measure is also available as the Template:Code property on the fitted model. This computes the (normalized) total reduction of the splitting criterion brought by each feature across all trees. While fast to compute, impurity-based importance is biased toward high-cardinality features and reflects training-set behavior rather than generalization performance. The Template:Code function is recommended as a more reliable alternative, especially when evaluated on held-out data.

Example Usage

from sklearn.linear_model import LogisticRegression
from sklearn.inspection import permutation_importance

X = [[1, 9, 9], [1, 9, 9], [1, 9, 9],
     [0, 9, 9], [0, 9, 9], [0, 9, 9]]
y = [1, 1, 1, 0, 0, 0]

clf = LogisticRegression().fit(X, y)
result = permutation_importance(clf, X, y, n_repeats=10, random_state=0)

print(result.importances_mean)
# array([0.4666, 0.    , 0.    ])
print(result.importances_std)
# array([0.2211, 0.    , 0.    ])

Source Location

Template:Code, function Template:Code (lines 138-313).

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment