Implementation:Scikit learn Scikit learn CV Results Attributes

Template:Implementation Metadata

Overview

Concrete tool for accessing hyperparameter search results and best configuration attributes provided by scikit-learn.

After calling fit on a GridSearchCV or RandomizedSearchCV instance, the results are stored in several attributes. The central attribute is cv_results_, a dictionary designed to be converted directly into a pandas DataFrame for analysis. Companion attributes (best_params_, best_score_, best_estimator_, best_index_) provide direct access to the optimal configuration.

Code Reference

Best Selection Logic (lines 1065-1106)

After all candidates are evaluated, the fit method selects the best configuration and optionally refits:

# For multi-metric evaluation, store the best_index_, best_params_ and
# best_score_ iff refit is one of the scorer names
# In single metric evaluation, refit_metric is "score"
if self.refit or not self.multimetric_:
    self.best_index_ = self._select_best_index(
        self.refit, refit_metric, results
    )
    if not callable(self.refit):
        # With a non-custom callable, we can select the best score
        # based on the best index
        self.best_score_ = results[f"mean_test_{refit_metric}"][
            self.best_index_
        ]
    self.best_params_ = results["params"][self.best_index_]

if self.refit:
    self.best_estimator_ = clone(base_estimator).set_params(
        **clone(self.best_params_, safe=False)
    )
    refit_start_time = time.time()
    if y is not None:
        self.best_estimator_.fit(X, y, **routed_params.estimator.fit)
    else:
        self.best_estimator_.fit(X, **routed_params.estimator.fit)
    refit_end_time = time.time()
    self.refit_time_ = refit_end_time - refit_start_time

self.cv_results_ = results
self.n_splits_ = n_splits

The _select_best_index Method

@staticmethod
def _select_best_index(refit, refit_metric, results):
    """Select index of the best combination of hyperparameters."""
    if callable(refit):
        # If callable, refit is expected to return the index of the best
        # parameter set.
        best_index = refit(results)
        if not isinstance(best_index, numbers.Integral):
            raise TypeError("best_index_ returned is not an integer")
        if best_index < 0 or best_index >= len(results["params"]):
            raise IndexError("best_index_ index out of range")
    else:
        best_index = results[f"rank_test_{refit_metric}"].argmin()
    return best_index

Main Result Attributes

cv_results_

A dictionary with keys as column headers and values as numpy arrays, designed for direct conversion to a pandas DataFrame.

Structure:

Key Pattern	Type	Description
params	list of dicts	The parameter dictionaries for each candidate.
param_{name}	masked array	Individual parameter values per candidate. Masked entries indicate the parameter does not apply to that candidate (relevant for multi-grid searches).
split{k}_test_score	numpy array	Test score for fold k (0-indexed), for each candidate.
mean_test_score	numpy array	Mean of test scores across all folds, for each candidate.
std_test_score	numpy array	Standard deviation of test scores across folds, for each candidate.
rank_test_score	numpy array of int32	Rank of each candidate by mean test score (1 = best). Ties receive the minimum rank.
split{k}_train_score	numpy array	Training score for fold k (only when return_train_score=True).
mean_train_score	numpy array	Mean training score (only when return_train_score=True).
std_train_score	numpy array	Standard deviation of training scores (only when return_train_score=True).
mean_fit_time	numpy array	Mean time (seconds) spent fitting across folds.
std_fit_time	numpy array	Standard deviation of fit time.
mean_score_time	numpy array	Mean time (seconds) spent scoring across folds.
std_score_time	numpy array	Standard deviation of score time.

For multi-metric evaluation, the score keys use the scorer name instead of score, e.g., mean_test_accuracy, rank_test_f1_macro.

best_params_

# Type: dict
# Example: {'C': 1, 'kernel': 'linear'}
search.best_params_

The parameter setting that gave the best results on the held-out data. This is equivalent to search.cv_results_['params'][search.best_index_].

best_score_

# Type: float
# Example: 0.9800
search.best_score_

The mean cross-validated score of the best candidate. Not available when refit is a callable (because the callable may use custom selection logic that does not correspond to a single metric).

best_estimator_

# Type: estimator object
# Example: SVC(C=1, kernel='linear')
search.best_estimator_

The estimator that was cloned, configured with best_params_, and refit on the entire dataset. Only available when refit=True (or refit is a string/callable that selects a configuration). This estimator can be used directly for prediction.

best_index_

# Type: int
# Example: 0
search.best_index_

The index into the cv_results_ arrays corresponding to the best candidate. Useful for looking up all associated statistics for the best configuration.

Usage Examples

Converting cv_results_ to a DataFrame

import pandas as pd
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

search = GridSearchCV(
    SVC(),
    param_grid={'C': [0.1, 1, 10], 'kernel': ['linear', 'rbf']},
    cv=5,
    return_train_score=True,
)
search.fit(X, y)

# Convert to DataFrame for tabular analysis
df = pd.DataFrame(search.cv_results_)

# View the most important columns, sorted by rank
cols = ['rank_test_score', 'mean_test_score', 'std_test_score',
        'mean_train_score', 'param_C', 'param_kernel', 'mean_fit_time']
print(df[cols].sort_values('rank_test_score'))

Identifying the Best Configuration

# Direct access to best results
print(f"Best parameters: {search.best_params_}")
print(f"Best CV score: {search.best_score_:.4f}")
print(f"Best estimator: {search.best_estimator_}")

# Equivalent lookup via best_index_
idx = search.best_index_
print(f"Best index: {idx}")
print(f"Params at best index: {search.cv_results_['params'][idx]}")
print(f"Score at best index: {search.cv_results_['mean_test_score'][idx]:.4f}")
print(f"Std at best index: {search.cv_results_['std_test_score'][idx]:.4f}")

Overfitting Diagnosis

import pandas as pd

df = pd.DataFrame(search.cv_results_)

# Compare train vs test scores to detect overfitting
df['overfit_gap'] = df['mean_train_score'] - df['mean_test_score']
print(df[['param_C', 'param_kernel', 'mean_train_score',
           'mean_test_score', 'overfit_gap']].sort_values('overfit_gap',
           ascending=False))

Custom Refit with One-Standard-Error Rule

import numpy as np
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

def one_se_rule(cv_results):
    """Select simplest model within 1 SE of the best score."""
    means = cv_results['mean_test_score']
    stds = cv_results['std_test_score']
    best_idx = np.argmax(means)
    threshold = means[best_idx] - stds[best_idx]
    # Among candidates above threshold, pick the one with smallest C
    candidates = np.where(means >= threshold)[0]
    c_values = [cv_results['params'][i]['C'] for i in candidates]
    return candidates[np.argmin(c_values)]

search = GridSearchCV(
    SVC(kernel='rbf'),
    param_grid={'C': [0.01, 0.1, 1, 10, 100]},
    cv=5,
    refit=one_se_rule,
)
search.fit(X, y)
print(search.best_params_)  # Likely a smaller C value

Related Pages

• Principle:Scikit_learn_Scikit_learn_Search_Results_Analysis