Implementation:NVIDIA DALI C API V2 Pipeline Tests

Knowledge Sources	NVIDIA_DALI
Domains	Data_Pipeline, C_API
Last Updated	2026-02-08 16:00 GMT

Overview

The C API v2 pipeline test suite validates pipeline deserialization, execution across CPU/GPU configurations, operator trace retrieval, external input feeding, input descriptor introspection, and executor compatibility checking.

Description

This file implements Google Tests for the DALI C API v2 pipeline functions. It belongs to the dali::c_api::test namespace and tests the full pipeline lifecycle through the v2 C API. The tests define several helper functions that construct protobuf-serialized pipelines using internal DALI test operators (CounterOp, TestOp, TestOpWithTrace) and an ExternalSource operator.

Pipeline construction helpers create four types of test pipelines: CPU-only (GetCPUOnlyPipelineProto), CPU-to-GPU (GetCPU2GPUPipelineProto), GPU-to-CPU (GetGPU2CPUPipelineProto), and trace-emitting (GetPipelineWithTraces). A custom TestOpWithTrace operator is registered inline that copies input to output while setting a trace string containing a configurable prefix and iteration counter.

The test suite covers: basic deserialization and build verification, running pipelines across three configurations (CPU-only, CPU-to-GPU, GPU-to-CPU) with output value verification using scalar counter sequences, operator trace querying (both individual and bulk), incompatible executor detection (non-dynamic executors with GPU-to-CPU transfers), external input feeding on CPU and GPU backends (with sync and async variants), and input descriptor introspection for both simple and extended external sources (verifying name, device, ndim, dtype, and layout fields).

Usage

Run these tests to validate the C API v2 pipeline implementation, particularly when making changes to pipeline execution, output handling, trace propagation, or input feeding. The tests exercise the dynamic executor and various queue depth configurations.

Code Reference

Source Location

• Repository: NVIDIA_DALI
• File: dali/c_api_2/pipeline_test.cc
• Lines: 1-552

Signature

// Helper pipeline construction
std::string GetCPUOnlyPipelineProto(int max_batch_size, int num_threads, int device_id);
std::string GetCPU2GPUPipelineProto(int max_batch_size, int num_threads, int device_id);
std::string GetGPU2CPUPipelineProto(int max_batch_size, int num_threads, int device_id);
std::string GetPipelineWithTraces(int max_batch_size, int num_threads, int device_id);
std::string GetPipelineWithExternalSource(StorageDevice dev_type, int max_batch_size,
                                           int num_threads, int device_id,
                                           bool extended_input_desc = false);

// Custom test operator
template <typename Backend>
class TestOpWithTrace : public Operator<Backend> { ... };

// Tests
TEST(CAPI2_PipelineTest, Deserialize);
TEST(CAPI2_PipelineTest, RunCPUOnly);
TEST(CAPI2_PipelineTest, RunCPU2GPU);
TEST(CAPI2_PipelineTest, RunGPU2CPU);
TEST(CAPI2_PipelineTest, IncompatibleExec);
TEST(CAPI2_PipelineTest, Traces);
TEST(CAPI2_PipelineTest, FeedInputCPU);
TEST(CAPI2_PipelineTest, FeedInputGPUSync);
TEST(CAPI2_PipelineTest, FeedInputGPUAsync);
TEST(CAPI2_PipelineTest, InputDescSimple);
TEST(CAPI2_PipelineTest, InputDescExtended);

Import

#include <gtest/gtest.h>
#include "dali/c_api_2/pipeline.h"
#include "dali/pipeline/pipeline.h"
#include "dali/pipeline/executor/executor2/exec2_ops_for_test.h"
#include "dali/c_api_2/pipeline_test_utils.h"

I/O Contract

Inputs

Name	Type	Required	Description
max_batch_size	int	Yes	Maximum batch size for test pipelines (typically 1 or 4)
num_threads	int	Yes	Number of worker threads (typically 4)
device_id	int	Yes	CUDA device ID (0 or CPU_ONLY_DEVICE_ID)
PipelineType	enum	Yes	CPUOnly, CPU2GPU, or GPU2CPU pipeline configuration
daliFeedInputFlags_t	flags	No	Feed input flags for copy/sync behavior testing

Outputs

Name	Type	Description
Test assertions	GTest EXPECT/ASSERT	Validates pipeline outputs, trace values, error codes
Scalar sequences	int values	Counter-based output values verified against expected sequences
Trace strings	const char *	Operator trace values in format "prefix_iteration"

Usage Examples

Testing Pipeline Run with Output Verification

std::string proto = GetCPUOnlyPipelineProto(1, 4, CPU_ONLY_DEVICE_ID);

daliPipelineParams_t params{};
params.max_batch_size_present = true;
params.max_batch_size = 4;
params.prefetch_queue_depths_present = true;
params.prefetch_queue_depths = {3, 3};

auto h = Deserialize(proto, params);
CHECK_DALI(daliPipelineBuild(h));

for (int iter = 0; iter < 5; iter++) {
    if (iter == 0)
        CHECK_DALI(daliPipelinePrefetch(h));
    else
        CHECK_DALI(daliPipelineRun(h));

    auto out_h = PopOutputs(h);
    auto o1 = GetOutput(out_h, 0);
    // Verify counter-based scalar output sequence
    CheckScalarSequence(o1, params.max_batch_size,
                        1000 + iter * params.max_batch_size, 2);
}

Testing Operator Traces

auto proto = GetPipelineWithTraces(4, 4, 0);
daliPipelineParams_t params{};
params.exec_type_present = true;
params.exec_type = DALI_EXEC_DYNAMIC;

auto h = Deserialize(proto, params);
CHECK_DALI(daliPipelineBuild(h));

for (int i = 0; i < 3; i++) {
    CHECK_DALI(daliPipelineRun(h));
    auto out_h = PopOutputs(h);

    const char *trace = nullptr;
    CHECK_DALI(daliPipelineOutputsGetTrace(out_h, &trace, "op1", "trace"));
    EXPECT_EQ(std::string(trace), "t1_" + std::to_string(i));
}

Related Pages

• Environment:NVIDIA_DALI_CUDA_GPU_Environment