Implementation:Ggml_org_Ggml_Ggml_opt_dataset_init
Summary
The ggml_opt_dataset_init function allocates and initializes a structured dataset object for neural network training in GGML. It creates two backing tensors -- one for data points and one for labels -- sized to hold the entire training or test set, and records the shard size used for batching and shuffling during optimization.
Import
#include "ggml-opt.h"
Dependencies
- ggml-opt.h -- public header defining
ggml_opt_dataset_t, the dataset opaque type, and the initialization prototype. - ggml.h -- core GGML header providing
enum ggml_type,struct ggml_tensor, and tensor allocation functions. - ggml-backend.h -- backend abstraction for tensor memory management and data transfer.
Function Signature
ggml_opt_dataset_t ggml_opt_dataset_init(
enum ggml_type type_data,
enum ggml_type type_label,
int64_t ne_datapoint,
int64_t ne_label,
int64_t ndata,
int64_t ndata_shard);
Source: src/ggml-opt.cpp:L86-130
Parameters
| Parameter | Type | Description |
|---|---|---|
| type_data | enum ggml_type |
Element type for the data tensor. Typically GGML_TYPE_F32 for normalized floating-point features.
|
| type_label | enum ggml_type |
Element type for the labels tensor. Typically GGML_TYPE_F32 for one-hot encoded classification targets.
|
| ne_datapoint | int64_t |
Number of features per data sample. For MNIST this is 784 (28x28 pixels flattened). |
| ne_label | int64_t |
Number of elements per label vector. For MNIST this is 10 (one per digit class, one-hot encoded). |
| ndata | int64_t |
Total number of samples in the dataset. For MNIST: 60000 (training set) or 10000 (test set). |
| ndata_shard | int64_t |
Number of samples per shard (batch size for shuffling). Determines the granularity of mini-batch iteration during optimization. |
Return Value
Returns a ggml_opt_dataset_t -- an opaque handle to the initialized dataset. The dataset contains:
- Data tensor with shape
[ne_datapoint, ndata]-- a 2-D tensor where each column holds one flattened data sample. - Labels tensor with shape
[ne_label, ndata]-- a 2-D tensor where each column holds the corresponding one-hot label vector.
Both tensors are allocated with the specified element types and are ready to be populated via the loader helper functions.
Loader Helpers
After initializing the dataset, raw data is loaded from binary files using format-specific helper functions. The MNIST example provides two such loaders:
mnist_image_load
bool mnist_image_load(
const std::string & fname,
ggml_opt_dataset_t dataset);
Source: examples/mnist/mnist-common.cpp:L18-40
Reads MNIST image data from an IDX binary file, normalizes each pixel value from [0, 255] to [0.0, 1.0] by dividing by 255, and writes the result into the dataset's data tensor. Returns true on success.
mnist_label_load
bool mnist_label_load(
const std::string & fname,
ggml_opt_dataset_t dataset);
Source: examples/mnist/mnist-common.cpp:L61-83
Reads MNIST label data from an IDX binary file, converts each scalar class index (0--9) into a one-hot vector of length ne_label, and writes the result into the dataset's labels tensor. Returns true on success.
Usage Example
#include "ggml-opt.h"
#include "mnist-common.h"
// 1. Initialize a dataset for MNIST training data
// 784 features, 10 classes, 60000 samples, batch size 500
ggml_opt_dataset_t dataset = ggml_opt_dataset_init(
GGML_TYPE_F32, // type_data
GGML_TYPE_F32, // type_label
784, // ne_datapoint (28*28 pixels)
10, // ne_label (digits 0-9)
60000, // ndata (training set size)
500); // ndata_shard (batch size)
// 2. Load images and labels from IDX files
mnist_image_load("train-images-idx3-ubyte", dataset);
mnist_label_load("train-labels-idx1-ubyte", dataset);
// 3. Dataset is now ready for use with ggml_opt
// The optimizer will iterate over shards, shuffling between epochs.