Principle:Haifengl Smile Data Transformation

Overview

Data Transformation is the principle of applying mathematical functions to DataFrame columns to prepare features for machine learning algorithms. In Smile, transformations are modeled as first-class objects that implement the Transform functional interface, enabling composable pipelines and invertible operations. The library supports standardization (z-score normalization), min-max scaling, robust scaling, row-wise normalization (L1, L2, L-infinity norms), max-absolute scaling, and Winsor scaling.

The core design insight is that transformations should be fit to training data and then applied uniformly to both training and test data, preventing data leakage. This fit-then-apply pattern mirrors the estimator-transformer pattern found in scikit-learn but is expressed through Java functional interfaces and composition.

Theoretical Basis

Why Transform?

Most ML algorithms assume that input features are on comparable scales. When features have widely different ranges, algorithms that rely on distance metrics (k-NN, SVM, k-means) or gradient-based optimization (neural networks, logistic regression) will be dominated by large-scale features. Feature scaling addresses this by mapping all features to a common range or distribution.

Standardization (Z-Score Normalization)

Standardization transforms each feature to have zero mean and unit variance:

$z_i=\frac{x_i-\mu }{\sigma }$

where $\mu =\frac{1}{n}\sum _{i=1}^n{x}_i$ is the sample mean and $\sigma =\sqrt{\frac{1}{n-1}\sum _{i=1}^n(x_i-\mu {)}^2}$ is the sample standard deviation.

After transformation: $\mathbb{𝔼}[z]=0$ and $\text{Var}(z)=1$.

The inverse transform recovers the original value: $x_i=z_i\cdot \sigma +\mu$.

In Smile, this is implemented by Standardizer.fit(data, columns).

Min-Max Scaling

Min-max scaling linearly maps values to the interval $[0,1]$:

$x{\text{'}}_i=\frac{x_i-x_{\min }}{x_{\max }-x_{\min }}$

where $x_{\min }$ and $x_{\max }$ are the observed minimum and maximum. Values are clipped to $[0,1]$ if new data falls outside the training range.

The inverse: $x_i=x{\text{'}}_i\cdot (x_{\max }-x_{\min })+x_{\min }$.

In Smile, this is implemented by Scaler.fit(data, columns).

Robust Standardization

Robust standardization uses the median and interquartile range (IQR) instead of mean and standard deviation, making it resistant to outliers:

$z_i=\frac{x_i-\text{median}}{Q_3-Q_1}$

where $Q_1$ and $Q_3$ are the first and third quartiles.

In Smile, this is implemented by RobustStandardizer.fit(data, columns).

Row-Wise Normalization

Normalizes each sample (row) to unit norm:

• L1 norm: $x{\text{'}}_i=\frac{x_i}{\sum _j|x_j|}$
• L2 norm: $x{\text{'}}_i=\frac{x_i}{\sqrt{\sum _j{x}_j^2}}$
• L-infinity norm: $x{\text{'}}_i=\frac{x_i}{\underset{j}{\max }|x_j|}$

In Smile, this is implemented by new Normalizer(Norm.L2, columns).

Max-Absolute Scaling

Scales each feature by its maximum absolute value, mapping to $[-1,1]$:

$x{\text{'}}_i=\frac{x_i}{\max (|x_j|)}$

In Smile, this is implemented by MaxAbsScaler.fit(data, columns).

Pipeline Composition

Transforms in Smile are composable through the Transform functional interface, which extends Function<Tuple, Tuple>. Two composition mechanisms are provided:

Sequential Composition

The andThen() method chains transforms:

$T_{\text{composed}}=T_2\circ T_1$

meaning $T_{\text{composed}}(x)=T_2(T_1(x))$.

Pipeline Construction

The static Transform.pipeline(transforms...) method creates a composed transform from multiple stages:

$T_{\text{pipeline}}=T_n\circ T_{n-1}\circ \dots \circ T_1$

Fit Pipeline

The static Transform.fit(data, trainers...) method implements the fit-then-apply pattern for multi-stage pipelines. Each trainer receives the data as transformed by all previous stages:

1. Fit $T_1$ on original data $D$
2. Apply $T_1$ to get $D_1=T_1(D)$
3. Fit $T_2$ on $D_1$
4. Compose: $T=T_2\circ T_1$
5. Continue for all trainers

This ensures that each transformer sees data in the correct scale from previous stages.

Invertibility

Transforms that implement InvertibleTransform support the inverse operation:

$T^{-1}(T(x))=x$

This is essential for:

• Interpreting predictions -- Converting model outputs back to the original scale.
• Visualization -- Displaying results in human-readable units.
• Post-processing -- Undoing normalization after prediction.

The column-level implementation stores both the forward function $f(x)$ and its inverse $f^{-1}(x)$ for each transformed column.

Relationship to the Data Loading Pipeline

Data Transformation is the fourth stage of the Smile Data Loading Pipeline:

1. File Data Loading -- Read data from files.
2. DataFrame Inspection -- Examine structure and metadata.
3. Column Selection and Filtering -- Select relevant columns.
4. Data Transformation -- Normalize and scale features. (current)
5. Numerical Conversion -- Convert to numerical arrays/matrices.

Transformation follows column selection (which determines which features to include) and precedes numerical conversion (which produces the final double[][] or DenseMatrix for algorithms).

Related Pages

• Implementation:Haifengl_Smile_Transform_Pipeline

Knowledge Sources

• Smile

Metadata

Property	Value
Domains	Data_Engineering, ETL
Workflow	Data_Loading_Pipeline
Stage	4 of 5
Last Updated	2026-02-08 22:00 GMT