Implementation:NVIDIA DALI EfficientDet Dataloader
| Knowledge Sources | |
|---|---|
| Domains | Object_Detection, TensorFlow |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
Provides TensorFlow data loading and preprocessing pipeline for EfficientDet, including image normalization, multi-scale training, anchor label assignment, and dataset construction.
Description
This module implements the data loading pipeline for the EfficientDet object detection model. It contains two main processor classes: `InputProcessor` for basic image processing operations and `DetectionInputProcessor` (a subclass) for detection-specific processing including bounding box manipulation.
`InputProcessor` handles image normalization using ImageNet mean/std values (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), multi-scale training with random scale factors, scale-to-output-size for evaluation, and image resizing with padding to the target output dimensions. It maintains internal state for image scale, scaled dimensions, and crop offsets.
`DetectionInputProcessor` extends the base class with detection-specific operations: random horizontal flipping of both images and bounding boxes, box coordinate rescaling and cropping to match image transformations, box clipping to output boundaries, and filtering of degenerate boxes (zero area after cropping).
The `InputReader` class orchestrates the complete data pipeline by parsing TFRecord examples through a `TfExampleDecoder`, applying the preprocessing chain (normalization, augmentation, scaling), running anchor label assignment via an `AnchorLabeler`, and padding ground truth data to fixed sizes. It constructs a `tf.data.Dataset` with configurable batching, prefetching, sharding for distributed training, and optional fake data generation for benchmarking.
A standalone `pad_to_fixed_size` utility function pads variable-length tensors to a fixed dimension for batching compatibility.
Usage
Use the `InputReader` class to create training and evaluation datasets for EfficientDet. Provide it with model parameters, TFRecord file patterns, and training/evaluation flags to get a properly configured `tf.data.Dataset`.
Code Reference
Source Location
- Repository: NVIDIA_DALI
- File: docs/examples/use_cases/tensorflow/efficientdet/pipeline/tf/dataloader.py
- Lines: 1-414
Signature
class InputProcessor:
def __init__(self, image, output_size): ...
def normalize_image(self): ...
def set_training_random_scale_factors(self, scale_min, scale_max, target_size=None): ...
def set_scale_factors_to_output_size(self): ...
def resize_and_crop_image(self, method=tf.image.ResizeMethod.BILINEAR): ...
class DetectionInputProcessor(InputProcessor):
def __init__(self, image, output_size, boxes=None, classes=None): ...
def random_horizontal_flip(self): ...
def clip_boxes(self, boxes): ...
def resize_and_crop_boxes(self): ...
class InputReader:
def __init__(self, params, file_pattern, is_training=False, use_fake_data=False): ...
def dataset_parser(self, value, example_decoder, anchor_labeler): ...
def process_example(self, batch_size, images, cls_targets, box_targets,
num_positives, boxes, classes): ...
def __call__(self, params, input_context=None) -> tf.data.Dataset: ...
def pad_to_fixed_size(data, pad_value, output_shape) -> tf.Tensor: ...
Import
from pipeline.tf.dataloader import InputReader
reader = InputReader(params, file_pattern="train-*.tfrecord", is_training=True)
dataset = reader(params)
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| params | dict | Yes | Model parameters including image_size, jitter_min/max, data_format, max_instances_per_image, etc. |
| file_pattern | str | Yes | Glob pattern for TFRecord input files |
| is_training | bool | No | Whether to apply training augmentations (default: False) |
| use_fake_data | bool | No | Use synthetic data for benchmarking (default: False) |
| image | tf.Tensor | Yes | Raw image tensor of shape [height, width, 3] (for InputProcessor) |
| output_size | int or tuple | Yes | Target output dimensions (for InputProcessor) |
Outputs
| Name | Type | Description |
|---|---|---|
| dataset | tf.data.Dataset | Batched dataset yielding (image, cls_targets, box_targets, num_positives, boxes, classes) |
| image | tf.Tensor | Preprocessed image of shape [output_height, output_width, 3] |
| boxes | tf.Tensor | Padded ground truth boxes of shape [max_instances, 4] |
| classes | tf.Tensor | Padded ground truth classes of shape [max_instances, 1] |
Usage Examples
Create Training Dataset
from pipeline.tf.dataloader import InputReader
params = {
"image_size": 640,
"jitter_min": 0.1,
"jitter_max": 2.0,
"input_rand_hflip": True,
"max_instances_per_image": 100,
"data_format": "channels_last",
"seed": None,
}
train_reader = InputReader(
params=params,
file_pattern="/data/tfrecords/train-*",
is_training=True,
)
train_dataset = train_reader(params)
for batch in train_dataset.take(1):
images, cls_targets, box_targets, num_pos, boxes, classes = batch