Principle:Scikit learn Scikit learn Model Training

Field	Value
sources	Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer; scikit-learn documentation: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
domains	Machine_Learning, Optimization, Statistics
last_updated	2026-02-08 15:00 GMT

Overview

An optimization process that adjusts model parameters to minimize a loss function on training data.

Description

Model training (also called fitting) is the core computational step in supervised learning. Given a training set of labeled examples $\{({\mathbf{𝐱}}_i,y_i){\}}_{i=1}^n$, the training process searches for parameter values that minimize a chosen loss function over the training data, subject to optional regularization constraints.

In scikit-learn, training is triggered by calling the fit(X, y) method on an instantiated estimator. This method:

1. Validates and preprocesses the input data (type checking, dtype conversion, sparse format handling).
2. Executes the optimization algorithm specified by the estimator's hyperparameters.
3. Stores the learned parameters as instance attributes with trailing underscores (e.g., coef_, intercept_).
4. Returns self, enabling method chaining.

The specific optimization strategy depends on the estimator and its configuration. Common approaches include:

• Gradient-based optimization -- Iterative methods such as L-BFGS, Newton-CG, and stochastic average gradient (SAG/SAGA) that use gradient information to find the loss minimum.
• Coordinate descent -- Used by the liblinear solver for L1-regularized problems.
• Closed-form solutions -- Some estimators (e.g., ordinary least squares) compute parameters directly via matrix algebra.

Regularization is a technique applied during training to prevent overfitting by penalizing large parameter values. Common regularization forms include L2 (ridge), L1 (lasso), and Elastic-Net (a combination of L1 and L2).

Usage

Use model training when:

• Fitting a model to labeled data -- The standard supervised learning workflow requires calling fit(X_train, y_train) on the training subset.
• Retraining after hyperparameter changes -- After modifying hyperparameters via set_params, the model must be re-fitted.
• Warm starting -- Some estimators support warm_start=True, allowing training to resume from previously learned parameters rather than starting from scratch.

Theoretical Basis

Maximum Likelihood Estimation

For classification with logistic regression, training corresponds to maximum likelihood estimation (MLE). The model assumes that the probability of class $k$ given features $\mathbf{𝐱}$ follows the softmax (multinomial) or sigmoid (binary) function of a linear combination of features.

In the binary case, the model estimates:

$P(y=1|\mathbf{𝐱})=\sigma ({\mathbf{𝐰}}^T\mathbf{𝐱}+b)=\frac{1}{1+e^{-({\mathbf{𝐰}}^T\mathbf{𝐱}+b)}}$

where $\sigma$ is the logistic sigmoid function, $\mathbf{𝐰}$ is the weight vector, and $b$ is the bias (intercept).

Loss Minimization

MLE is equivalent to minimizing the negative log-likelihood, which for logistic regression yields the logistic loss (also called cross-entropy loss or log loss):

${ℒ}(\mathbf{𝐰},b)=-\frac{1}{n}\sum _{i=1}^n[y_i\log ({\hat{p}}_i)+(1-y_i)\log (1-{\hat{p}}_i)]$

where ${\hat{p}}_i=\sigma ({\mathbf{𝐰}}^T{\mathbf{𝐱}}_i+b)$ is the predicted probability for sample $i$.

Regularized Objective

With regularization, the objective becomes:

$\underset{\mathbf{𝐰},b}{\min }\frac{1}{2C}[(1-\alpha )\Vert \mathbf{𝐰}{\Vert }_2^2+\alpha \Vert \mathbf{𝐰}{\Vert }_1]+{ℒ}(\mathbf{𝐰},b)$

where $C$ is the inverse regularization strength and $\alpha$ is the L1 ratio (Elastic-Net mixing parameter). Setting $\alpha =0$ yields pure L2 regularization; $\alpha =1$ yields pure L1 regularization.

Related Pages

• Implementation:Scikit_learn_Scikit_learn_LogisticRegression_Fit
• Heuristic:Scikit_learn_Scikit_learn_Convergence_Warning_Handling
• Heuristic:Scikit_learn_Scikit_learn_Feature_Scaling_Numerical_Stability