Implementation:NVIDIA DALI COCO TFRecord Creator
| Knowledge Sources | |
|---|---|
| Domains | Data_Conversion, Object_Detection |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
Converts raw COCO 2017 dataset images and annotations into sharded TFRecord files suitable for object detection training with EfficientDet.
Description
This script reads COCO-format JSON annotation files and the corresponding image directory to produce TFRecord files that can be consumed by TensorFlow-based detection pipelines. It parses object bounding box annotations (converting from COCO's [x, y, width, height] format to normalized [ymin, xmin, ymax, xmax]), category labels, crowd annotations, and optionally image captions. Invalid annotations (zero or negative dimensions, boxes exceeding image boundaries) are automatically skipped and counted.
The converter supports multiprocessing via a configurable thread pool to parallelize the I/O-heavy image reading and TFRecord serialization. Output is sharded across a configurable number of files (default 32) for efficient distributed training. Each TFRecord example stores the JPEG-encoded image bytes, image metadata (height, width, filename, source ID, SHA-256 key), bounding box coordinates, class labels and names, crowd flags, and object areas.
The script is designed to work with the EfficientDet pipeline within the NVIDIA DALI examples and relies on companion modules `tfrecord_util` for feature serialization helpers and `label_map_util` for category index creation from COCO category metadata.
Usage
Use this script to prepare COCO 2017 training and validation datasets as TFRecord files before training EfficientDet models. Run it as a standalone command-line tool with flags specifying image directories, annotation file paths, output file prefix, number of shards, and optionally the number of processing threads.
Code Reference
Source Location
- Repository: NVIDIA_DALI
- File: docs/examples/use_cases/tensorflow/efficientdet/dataset/create_coco_tfrecord.py
- Lines: 1-353
Signature
def create_tf_example(
image,
image_dir,
bbox_annotations=None,
category_index=None,
caption_annotations=None,
) -> Tuple[str, tf.train.Example, int]:
...
def _load_object_annotations(object_annotations_file):
...
def _load_caption_annotations(caption_annotations_file):
...
def _create_tf_record_from_coco_annotations(
object_annotations_file,
caption_annotations_file,
image_dir,
output_path,
num_shards,
num_threads,
):
...
Import
# Standalone script; run via command line:
# python create_coco_tfrecord.py --image_dir=... --object_annotations_file=... --output_file_prefix=...
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| image_dir | str (flag) | Yes | Directory containing COCO JPEG images |
| object_annotations_file | str (flag) | No | COCO JSON annotation file with bounding boxes |
| caption_annotations_file | str (flag) | No | COCO JSON annotation file with image captions |
| image_info_file | str (flag) | No | JSON file with image metadata (used if annotations not provided) |
| output_file_prefix | str (flag) | Yes | Path prefix for output TFRecord shards |
| num_shards | int (flag) | No | Number of output TFRecord shards (default: 32) |
| num_threads | int (flag) | No | Number of parallel processing threads (default: None, auto) |
Outputs
| Name | Type | Description |
|---|---|---|
| TFRecord files | files | Sharded TFRecord files at `{output_file_prefix}-{shard_id}-of-{num_shards}.tfrecord` |
Usage Examples
Convert COCO Training Set
python create_coco_tfrecord.py \
--image_dir="/data/coco/train2017" \
--object_annotations_file="/data/coco/annotations/instances_train2017.json" \
--caption_annotations_file="/data/coco/annotations/captions_train2017.json" \
--output_file_prefix="/data/tfrecords/train" \
--num_shards=100