Implementation:NVIDIA DALI Rec2idx
| Knowledge Sources | |
|---|---|
| Domains | Data_Loading, Indexing |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
Command-line tool that creates an index file from a MXNet RecordIO (.rec) file, enabling random access for DALI's RecordIO reader.
Description
This script defines the IndexCreator class, which extends MXNet's mx.recordio.MXRecordIO to read a RecordIO archive and produce a tab-separated index file mapping record keys to byte offsets within the archive. The index file enables DALI's fn.readers.mxnet reader to perform efficient random access into RecordIO files without sequentially scanning the entire archive.
The IndexCreator class overrides the open and close methods to manage both the RecordIO file handle and the output index file handle. It adds a tell method that queries the current read position via the MXNet C library's MXRecordIOReaderTell function. The core create_index method iterates through all records in the archive, recording the byte offset of each record before reading it, and writes lines of the format key\toffset\n to the index file. The key type is configurable (defaulting to sequential integers). Progress is reported every 1000 records.
The script includes a command-line interface via argparse that accepts a path to the .rec file and the desired output index file path. Both paths are resolved to absolute paths before processing.
Usage
Run this tool before using DALI's MXNet RecordIO reader with random access. The generated .idx file should be provided alongside the .rec file when creating the DALI reader pipeline.
Code Reference
Source Location
- Repository: NVIDIA_DALI
- File: tools/rec2idx.py
- Lines: 1-111
Signature
class IndexCreator(mx.recordio.MXRecordIO):
"""Reads RecordIO data format, and creates index file
that enables random access.
Parameters
----------
uri : str
Path to the record file.
idx_path : str
Path to the index file, that will be created/overwritten.
key_type : type
Data type for keys (optional, default = int).
"""
def __init__(self, uri, idx_path, key_type=int): ...
def open(self): ...
def close(self): ...
def tell(self) -> int: ...
def create_index(self): ...
def parse_args(): ...
def main(): ...
Import
from tools.rec2idx import IndexCreator
# or run directly:
# python tools/rec2idx.py data/train.rec data/train.idx
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| record | str (CLI arg) | Yes | Path to the MXNet RecordIO (.rec) file to index |
| index | str (CLI arg) | Yes | Path to the output index (.idx) file to create |
| key_type | type | No | Python type for record keys (default: int, producing sequential integer keys) |
Outputs
| Name | Type | Description |
|---|---|---|
| .idx file | text file | Tab-separated file with lines of key\toffset where key is the record identifier and offset is the byte position in the .rec file
|
| Console output | text | Progress messages every 1000 records showing elapsed time and count |
Usage Examples
Command-line Usage
# Create an index file for a RecordIO archive
# $ python tools/rec2idx.py data/train.rec data/train.idx
Programmatic Usage
from tools.rec2idx import IndexCreator
creator = IndexCreator("data/train.rec", "data/train.idx")
creator.create_index()
creator.close()
# The resulting index file looks like:
# 0 0
# 1 1024
# 2 3072
# ...
Using the Index with DALI
import nvidia.dali as dali
@dali.pipeline_def(batch_size=32, num_threads=4, device_id=0)
def pipeline():
images, labels = dali.fn.readers.mxnet(
path="data/train.rec",
index_path="data/train.idx",
random_shuffle=True,
)
return dali.fn.decoders.image(images, device="mixed"), labels