Principle:Ggml org Ggml Neural Network Graph Building
ML_Training Model_Architecture GGML 2025-05-15 12:00 GMT
Summary
Constructing the forward computation graph for a neural network from weight tensors. The graph defines the complete forward pass for training; the corresponding backward pass (for gradient computation) is auto-generated by the framework from this graph.
Theory
Neural network graph building composes elementary tensor operations into a directed acyclic graph that maps input tensors to output predictions. The key building blocks are:
- Fully-connected (dense) layers -- matrix multiplication (
ggml_mul_mat) of input with a weight matrix, followed by bias addition (ggml_add). - Activation functions -- non-linear element-wise transforms such as ReLU (
ggml_relu) and GELU that introduce non-linearity between layers. - Convolutional layers -- 2-D convolution (
ggml_conv_2d) with learned kernels, typically followed by an activation and spatial down-sampling via pooling (ggml_pool_2d). - Skip / residual connections -- element-wise addition of an earlier layer's output to a later layer, enabling gradient flow in deep networks.
Marking every learnable weight tensor as a parameter (ggml_set_param) instructs the automatic differentiation engine to track gradients for that tensor during the backward pass.
FC Network Architecture
A minimal fully-connected network for digit classification:
input -> matmul(W1) + b1 -> ReLU -> matmul(W2) + b2 -> output (logits)
- Layer 1:
hidden = relu(fc1_weight * images + fc1_bias) - Layer 2:
logits = fc2_weight * hidden + fc2_bias
CNN Network Architecture
A convolutional network for the same task:
input -> reshape(28,28,1,batch)
-> conv2d(3x3) + ReLU -> maxpool(2x2)
-> conv2d(3x3) + ReLU -> maxpool(2x2)
-> flatten -> dense -> output (logits)
Each convolutional stage applies a learned 3x3 kernel, a ReLU activation, and 2x2 max-pooling to reduce spatial dimensions before the final dense projection.
Parameter Registration and Automatic Differentiation
Calling ggml_set_param on every weight tensor (and bias tensor) has two effects:
- It flags the tensor so the backward-pass graph generator knows to compute
d(loss)/d(param). - It allows the optimiser to enumerate all trainable parameters for the weight-update step.
The forward graph alone is sufficient; ggml_build_backward (or equivalent) derives the full backward graph automatically.