Principle:Ggml org Ggml Model Architecture Initialization
Summary
Model Architecture Initialization is the process of defining and initializing a neural network architecture — including its weights, biases, and layer structure — before training can begin. This principle encompasses both the structural definition of the model (which layers, how they connect, their tensor shapes) and the numerical initialization of all learnable parameters.
Theory
Weight Initialization Strategies
Proper weight initialization is critical for convergence during training. Common strategies include:
- Random Normal Initialization — Sampling weights from a normal (Gaussian) distribution with a small standard deviation. This is the most straightforward approach and is suitable when combined with appropriate scaling.
- Xavier (Glorot) Initialization — Scales the variance of initial weights based on the number of input and output units in each layer. Designed to keep signal magnitudes roughly constant across layers in networks using sigmoid or tanh activations.
- He Initialization — A variant of Xavier initialization tuned for ReLU-family activations, using a larger variance to account for the fact that ReLU zeroes out roughly half of its inputs.
Architecture Definition
Before initialization, the model architecture must be defined. Two common architectures for image classification tasks are:
- Fully Connected (FC) Networks — Every neuron in one layer connects to every neuron in the next. Simple to implement but parameter-heavy for high-dimensional inputs such as images.
- Convolutional Neural Networks (CNNs) — Use learned convolutional kernels to extract spatial features with far fewer parameters through weight sharing and local connectivity.
Parameter Counting
Understanding the total number of trainable parameters is essential for estimating memory requirements, computational cost, and the risk of overfitting. Parameter counts are determined by the tensor dimensions of each layer's weights and biases.
Two Paths to Initialization
There are two fundamental paths for model initialization:
- Random Initialization (Training from Scratch) — All weights and biases are initialized using a random distribution. The model starts with no prior knowledge and must learn entirely from the training data.
- Loading Pre-trained Weights from File — Weights are deserialized from a saved model file (e.g., GGUF format). This enables transfer learning, fine-tuning, or resuming interrupted training.
Problem Solved
This principle addresses the foundational requirement of setting up the model structure with properly initialized parameters so that gradient-based optimization can converge. Poor initialization can lead to vanishing or exploding gradients, symmetry problems (where all neurons learn the same features), or extremely slow convergence. Correct initialization ensures that:
- Signal magnitudes remain stable across layers during forward and backward passes.
- Symmetry among neurons is broken so that different features can be learned.
- The model is ready for efficient optimization from the first training step.