Principle:Scikit learn Scikit learn Support Vector Machines

Knowledge Sources	Scikit_learn Scikit-learn Docs
Domains	Supervised Learning, Classification
Last Updated	2026-02-08 15:00 GMT

Overview

Support vector machines find the optimal hyperplane that maximizes the margin between classes, optionally mapping data into higher-dimensional spaces via kernel functions.

Description

Support Vector Machines (SVMs) are powerful supervised learning models for classification and regression that seek the decision boundary with the largest possible margin of separation between classes. The key insight is that only a subset of training points (the support vectors) determine the decision boundary, making SVMs memory-efficient. The kernel trick allows SVMs to learn non-linear decision boundaries by implicitly operating in high-dimensional feature spaces without explicit transformation. SVMs occupy a central position in machine learning, known for strong theoretical guarantees based on statistical learning theory and structural risk minimization.

Usage

Use SVC (Support Vector Classification) for binary and multiclass classification tasks where high accuracy is needed and the dataset is small to moderate in size. Use SVR (Support Vector Regression) for regression tasks where an epsilon-insensitive tube around the prediction is appropriate. Use NuSVC/NuSVR when you prefer to parameterize the model using the fraction of support vectors rather than the penalty parameter C. Choose the RBF kernel as a default for non-linear problems, the linear kernel when the data is high-dimensional relative to the number of samples, and the polynomial kernel when interaction effects between features are expected. SVMs do not scale well to very large datasets; for large-scale linear problems, consider SGD-based approaches instead.

Theoretical Basis

Linear SVM (hard margin) solves:

$\underset{w,b}{\min }\frac{1}{2}\Vert w{\Vert }^2\text{s.t.}{y}_i(w^T{x}_i+b)\ge 1,\mathrm{\forall }i$

The margin is $2/\Vert w\Vert$, and maximizing the margin is equivalent to minimizing $\Vert w{\Vert }^2$.

Soft-margin SVM introduces slack variables ${\xi }_i\ge 0$ to allow misclassifications:

$\underset{w,b,\xi }{\min }\frac{1}{2}\Vert w{\Vert }^2+C\sum _{i=1}^n{\xi }_i$ $\text{s.t.}{y}_i(w^T{x}_i+b)\ge 1-{\xi }_i,{\xi }_i\ge 0$

The parameter $C$ controls the trade-off between margin width and training error.

Dual formulation:

$\underset{\alpha }{\max }\sum _{i=1}^n{\alpha }_i-\frac{1}{2}\sum _{i,j}{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i^T{x}_j$ $\text{s.t.}0\le {\alpha }_i\le C,\sum _i{\alpha }_i{y}_i=0$

Kernel trick: Replace the inner product $x_i^T{x}_j$ with a kernel function $k(x_i,x_j)=\varphi (x_i{)}^T\varphi (x_j)$:

• RBF kernel: $k(x,y)=\exp (-\gamma \Vert x-y{\Vert }^2)$
• Polynomial kernel: $k(x,y)=(\gamma x^{T}y+r{)}^d$
• Sigmoid kernel: $k(x,y)=\tanh (\gamma x^{T}y+r)$

Support Vector Regression (SVR) uses an $\epsilon$-insensitive loss:

$\underset{w,b}{\min }\frac{1}{2}\Vert w{\Vert }^2+C\sum _{i=1}^n({\xi }_i+{\xi }_i^{*})$ $\text{s.t.}|y_i-(w^T{x}_i+b)|\le \epsilon +{\xi }_i+{\xi }_i^{*}$

Errors within the $\epsilon$-tube incur no penalty; only errors exceeding it contribute to the loss.

Nu-SVM replaces $C$ with a parameter $\nu \in (0,1]$ that provides an upper bound on the fraction of training errors and a lower bound on the fraction of support vectors, offering a more interpretable parameterization.

Related Pages

• Implementation:Scikit_learn_Scikit_learn_SVC
• Implementation:Scikit_learn_Scikit_learn_BaseSVC
• Implementation:Scikit_learn_Scikit_learn_L1MinC