Workflow:Haifengl Smile Data Loading Pipeline
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Machine_Learning |
| Last Updated | 2026-02-08 21:00 GMT |
Overview
End-to-end process for loading tabular data from multiple file formats into Smile's DataFrame for analysis, transformation, and conversion to numerical arrays or matrices.
Description
This workflow covers the standard procedure for ingesting data into the Smile framework. It uses the unified Read API to load data from CSV, JSON, Parquet, Arrow, Avro, ARFF, SAS, and libsvm formats into a DataFrame. Once loaded, the data can be inspected, filtered, selected, and transformed using DataFrame operations and the Transform pipeline. Finally, the data can be converted to double arrays or DenseMatrix objects for use with machine learning algorithms.
Usage
Execute this workflow when you have raw data files (CSV, JSON, Parquet, Arrow, Avro, ARFF, or SAS) and need to load them into Smile's DataFrame structure for exploratory analysis, feature engineering, or preparation for machine learning algorithms.
Execution Steps
Step 1: Load Data from File
Use the unified Read API to load data from a file into a DataFrame. The Read interface automatically infers the file format from the file extension (csv, json, parquet, feather, arff, sas7bdat, avro). For CSV files, you can specify format options such as delimiter, header, quote character, and escape character. For JSON files, you can choose between single-line and multi-line mode.
Key considerations:
- The Read.data() method auto-detects format from file extension
- CSV files support configurable delimiters, headers, quotes, and escape characters
- JSON supports single-line (one record per line) and multi-line modes
- Avro files require an accompanying schema
- All readers return a DataFrame with automatic type inference
Step 2: Inspect the DataFrame
Examine the structure and content of the loaded DataFrame. Check the schema (column names, data types), row count, and preview the first few rows. Use the describe() method for summary statistics and the schema() accessor for column metadata.
Key considerations:
- The toString() method shows the first 10 rows by default
- schema() returns a StructType with column names, types, and measures
- names() returns the column name array
- dtypes() returns the column data type array
- size() returns the number of rows
Step 3: Select and Filter Columns
Select relevant columns and filter rows based on conditions. Use select() to pick specific columns by name or index, and slice() or stream operations to filter rows. DataFrame supports SQL-like operations including projection, selection, and joining.
Key considerations:
- select() creates a new DataFrame with only the specified columns
- drop() removes specified columns
- merge() combines DataFrames horizontally
- union() combines DataFrames vertically
- factorize() converts string columns to categorical integer encoding
Step 4: Apply Transforms
Apply data transformations using the Transform pipeline. Transformations include normalization, standardization, and custom column-level transforms. Some transforms are invertible, allowing you to reverse the transformation after model training.
Key considerations:
- The Transform interface supports both forward and inverse operations
- ColumnTransform applies transformations to individual columns
- InvertibleColumnTransform allows reversing the transformation
- Transforms are composable and can be chained
Step 5: Convert to Numerical Representation
Convert the DataFrame to numerical arrays or matrices suitable for machine learning algorithms. Use toArray() to convert to a 2D double array, or toMatrix() to convert to a DenseMatrix. For classification tasks, extract the target variable separately.
Key considerations:
- toArray() converts all numeric columns to a double[][] array
- toMatrix() converts to a DenseMatrix backed by native BLAS memory
- For classification, separate features (X) from labels (y) before conversion
- Categorical columns should be factorized or one-hot encoded before conversion