Implementation:NVIDIA DALI NumPy Header

Knowledge Sources	NVIDIA_DALI
Domains	Utilities, File_IO
Last Updated	2026-02-08 16:00 GMT

Overview

Declares the public API for NumPy .npy file parsing, including the HeaderData class and functions for header parsing, tensor reading, and Fortran-order transposition.

Description

The NumPy header in dali/util/numpy.h provides the public interface for loading NumPy binary files into DALI tensors. It defines the HeaderData class, which stores the parsed metadata from an NPY file header: the tensor shape, a pointer to the DALI TypeInfo descriptor, a fortran_order flag indicating column-major layout, and the data_offset byte position where the raw array data begins in the file. The class also provides convenience methods type(), size(), and nbytes() for querying the data type, total element count, and total byte size respectively.

The header declares the NUMPY_ALLOWED_TYPES macro, which lists all supported NumPy data types: bool, unsigned and signed integers (8 to 64 bit), float16, float, and double. This macro is used with DALI's TYPE_SWITCH mechanism to dispatch type-specific operations at compile time.

Five public functions are declared: ParseHeader for reading headers from standard input streams, ParseODirectHeader for O_DIRECT aligned file access, ParseHeaderContents for parsing a raw header dictionary string, FromFortranOrder for transposing Fortran-order arrays to C-order, and ReadTensor for a one-shot read of a complete NPY file into a CPU tensor.

Usage

Include this header to access the NumPy file parsing API. Use ParseHeader or ParseODirectHeader to extract metadata from an NPY file stream, then read the raw data starting at data_offset. For a simpler workflow, use ReadTensor to load the entire file into a DALI tensor in a single call.

Code Reference

Source Location

• Repository: NVIDIA_DALI
• File: dali/util/numpy.h
• Lines: 1-65

Signature

#define NUMPY_ALLOWED_TYPES \
  (bool, uint8_t, uint16_t, uint32_t, uint64_t, int8_t, int16_t, int32_t, int64_t, \
   float, float16, double)

namespace dali {
namespace numpy {

class DLL_PUBLIC HeaderData {
 public:
  TensorShape<> shape;
  const TypeInfo *type_info = nullptr;
  bool fortran_order        = false;
  int64_t data_offset       = 0;

  DALIDataType type() const;
  size_t size() const;
  size_t nbytes() const;
};

DLL_PUBLIC void ParseHeader(HeaderData &parsed_header, InputStream *src);

DLL_PUBLIC void ParseODirectHeader(HeaderData &parsed_header, InputStream *src,
                                   size_t o_direct_alignm, size_t o_direct_read_len_alignm);

DLL_PUBLIC void FromFortranOrder(SampleView<CPUBackend> output, ConstSampleView<CPUBackend> input);

DLL_PUBLIC void ParseHeaderContents(HeaderData& target, const std::string_view header);

DLL_PUBLIC Tensor<CPUBackend> ReadTensor(InputStream *src, bool pinned);

}  // namespace numpy
}  // namespace dali

Import

#include "dali/util/numpy.h"

I/O Contract

Inputs

Name	Type	Required	Description
src	InputStream*	Yes (ParseHeader, ParseODirectHeader, ReadTensor)	Input stream positioned at the start of the NPY file
parsed_header / target	HeaderData&	Yes (ParseHeader, ParseODirectHeader, ParseHeaderContents)	Output reference to receive parsed metadata
header	std::string_view	Yes (ParseHeaderContents)	Raw NPY header dictionary string
o_direct_alignm	size_t	Yes (ParseODirectHeader)	Memory alignment for O_DIRECT reads
o_direct_read_len_alignm	size_t	Yes (ParseODirectHeader)	Read length alignment for O_DIRECT
pinned	bool	Yes (ReadTensor)	Whether to use pinned (page-locked) memory for the output tensor
output	SampleView<CPUBackend>	Yes (FromFortranOrder)	Pre-allocated output sample view for transposed data
input	ConstSampleView<CPUBackend>	Yes (FromFortranOrder)	Input sample view in Fortran order

Outputs

Name	Type	Description
HeaderData.shape	TensorShape<>	Parsed tensor shape (reversed if Fortran-order)
HeaderData.type_info	const TypeInfo*	Pointer to DALI type information
HeaderData.fortran_order	bool	True if the array was stored in Fortran (column-major) order
HeaderData.data_offset	int64_t	Byte offset where array data begins in the file
return value (ReadTensor)	Tensor<CPUBackend>	CPU tensor containing the loaded and optionally transposed array

Usage Examples

Loading a NumPy Array

#include "dali/util/numpy.h"

auto stream = FileStream::Open("array.npy", false, false);
auto tensor = dali::numpy::ReadTensor(stream.get(), /*pinned=*/true);

// tensor now contains the array data in C-order
// tensor.shape() and tensor.type() provide metadata

Parsing Header Only

#include "dali/util/numpy.h"

auto stream = FileStream::Open("data.npy", false, false);

dali::numpy::HeaderData header;
dali::numpy::ParseHeader(header, stream.get());

// header.shape, header.type(), header.fortran_order, header.data_offset
size_t total_bytes = header.nbytes();

Related Pages

• Environment:NVIDIA_DALI_CUDA_GPU_Environment