Principle:Scikit learn Scikit learn Decision Tree Learning

Knowledge Sources	Scikit_learn Scikit-learn Docs
Domains	Supervised Learning, Interpretable Models
Last Updated	2026-02-08 15:00 GMT

Overview

Decision tree learning constructs a tree-structured model that makes predictions by recursively partitioning the feature space along axis-aligned splits, yielding an interpretable sequence of if-then rules.

Description

Decision trees learn a hierarchy of binary decisions, each based on a single feature threshold, to partition the input space into regions with homogeneous target values. They solve both classification and regression problems while producing models that are inherently interpretable and easy to visualize. Trees handle non-linear relationships, feature interactions, and mixed feature types (numerical and categorical) without requiring feature scaling. However, individual trees are prone to overfitting and high variance, which motivates their use as base learners in ensemble methods (Random Forests, Gradient Boosting). Decision trees form one of the most fundamental and widely-used model families in machine learning.

Usage

Use decision trees when interpretability is paramount and the audience needs to understand the reasoning behind predictions. Use them as a baseline model for classification or regression before trying more complex methods. Apply pre-pruning (max depth, min samples per leaf) or post-pruning to control overfitting. Decision trees are also the foundation for ensemble methods that aggregate many trees to improve predictive performance. Use tree visualization and export capabilities to communicate model logic to stakeholders.

Theoretical Basis

Tree Construction (CART algorithm): The tree is built by recursively selecting the best split at each node:

1. For each candidate feature $j$ and threshold $t$, partition the data into left ($x_j\le t$) and right ($x_j>t$) subsets.
2. Evaluate the quality of the split using an impurity criterion.
3. Select the split $(j^{*},t^{*})$ that maximizes the reduction in impurity.
4. Repeat recursively on each subset until a stopping criterion is met.

Classification Impurity Measures:

Gini impurity: $G(m)=1-\sum _{k=1}^K{p}_{mk}^2$

where $p_{mk}$ is the proportion of class $k$ in node $m$. Gini impurity is zero when all samples belong to one class.

Entropy (Information Gain): $H(m)=-\sum _{k=1}^K{p}_{mk}{\log }_2(p_{mk})$

The information gain of a split is: $IG=H(\text{parent})-\sum _{\text{child}}\frac{n_{\text{child}}}{n_{\text{parent}}}H(\text{child})$

Regression Impurity Measure:

Mean Squared Error: $\text{MSE}(m)=\frac{1}{|m|}\sum _{i\in m}(y_i-{\overline{y}}_m{)}^2$

where ${\overline{y}}_m$ is the mean target value in node $m$.

Prediction:

• Classification: Majority class in the leaf node, or the class probability distribution.
• Regression: Mean (or median) of target values in the leaf node.

Pruning controls model complexity:

• Pre-pruning: Limit max depth, require minimum samples per split/leaf, limit max leaf nodes.
• Cost-complexity pruning (post-pruning): Minimize:

$R_{\alpha }(T)=R(T)+\alpha |T|$ where $R(T)$ is the training error, $|T|$ is the number of leaves, and $\alpha$ is the complexity parameter. The optimal $\alpha$ is chosen via cross-validation.

Feature importance is computed as the total reduction in impurity brought by each feature across all nodes: $\text{importance}(j)=\sum _{m:\text{split on }j}{n}_m\cdot \mathrm{\Delta }\text{impurity}(m)$

Related Pages

• Implementation:Scikit_learn_Scikit_learn_DecisionTreeClassifier
• Implementation:Scikit_learn_Scikit_learn_ExportTree