Implementation:NVIDIA DALI S3 Filesystem

Knowledge Sources	NVIDIA_DALI
Domains	Utilities, File_IO, Cloud_Storage
Last Updated	2026-02-08 16:00 GMT

Overview

Declares the S3 filesystem utility API for parsing S3 URIs, querying object metadata, reading object contents, and listing objects from Amazon S3 buckets.

Description

The S3 filesystem header in dali/util/s3_filesystem.h provides a lightweight abstraction layer over the AWS SDK for C++ to enable DALI to read data directly from Amazon S3 storage. The API is organized within the dali::s3_filesystem namespace and revolves around two simple data structures: S3ObjectLocation, which holds the parsed bucket name and object key, and S3ObjectStats, which stores whether an object exists and its size in bytes.

The module exposes four primary functions. parse_uri takes a standard S3 URI string (e.g., s3://bucket/key) and returns an S3ObjectLocation with the bucket and object components separated. get_stats queries the S3 service for the existence and size of an object or bucket. read_object_contents performs a ranged read of an S3 object into a pre-allocated buffer, supporting an optional byte offset for partial reads. list_objects_f iterates over all objects under a given S3 prefix, invoking a user-provided callable (PerObjectCallable) for each object with its key and size.

All functions require an open Aws::S3::S3Client pointer, which must be configured and authenticated externally. The functions are exported via DLL_PUBLIC for use from shared libraries.

Usage

Use this API when DALI pipelines need to read training data or other assets directly from Amazon S3 without downloading to local disk first. Parse the S3 URI to extract the bucket and key, check object stats if needed, then read the contents or enumerate objects in a prefix.

Code Reference

Source Location

• Repository: NVIDIA_DALI
• File: dali/util/s3_filesystem.h
• Lines: 1-88

Signature

namespace dali {
namespace s3_filesystem {

struct DLL_PUBLIC S3ObjectLocation {
  std::string bucket;
  std::string object;
};

struct DLL_PUBLIC S3ObjectStats {
  bool exists = false;
  size_t size = 0;
};

DLL_PUBLIC S3ObjectLocation parse_uri(const std::string& uri);

DLL_PUBLIC S3ObjectStats get_stats(Aws::S3::S3Client* s3_client,
                                   const S3ObjectLocation& object_location);

DLL_PUBLIC size_t read_object_contents(Aws::S3::S3Client* s3_client,
                                       const S3ObjectLocation& object_location,
                                       void* buf, size_t n, size_t offset = 0);

using PerObjectCallable = std::function<void(const std::string&, size_t)>;

DLL_PUBLIC void list_objects_f(Aws::S3::S3Client* s3_client,
                               const S3ObjectLocation& object_location,
                               PerObjectCallable per_object_call);

}  // namespace s3_filesystem
}  // namespace dali

Import

#include "dali/util/s3_filesystem.h"

I/O Contract

Inputs

Name	Type	Required	Description
uri	const std::string&	Yes (parse_uri)	S3 URI string (e.g., s3://my-bucket/path/to/object)
s3_client	Aws::S3::S3Client*	Yes (get_stats, read_object_contents, list_objects_f)	Pre-configured and authenticated AWS S3 client
object_location	const S3ObjectLocation&	Yes (get_stats, read_object_contents, list_objects_f)	Parsed bucket and object key pair
buf	void*	Yes (read_object_contents)	Pre-allocated buffer to receive the object data
n	size_t	Yes (read_object_contents)	Number of bytes to read
offset	size_t	No (read_object_contents)	Byte offset to start reading from (default 0)
per_object_call	PerObjectCallable	Yes (list_objects_f)	Callback invoked for each listed object with its key and size

Outputs

Name	Type	Description
return value (parse_uri)	S3ObjectLocation	Parsed bucket name and object key
return value (get_stats)	S3ObjectStats	Existence flag and size in bytes
return value (read_object_contents)	size_t	Number of bytes actually read into the buffer

Usage Examples

Parsing an S3 URI and Reading Contents

#include "dali/util/s3_filesystem.h"

// Parse the S3 URI
auto location = dali::s3_filesystem::parse_uri("s3://my-bucket/datasets/train/image001.jpg");
// location.bucket == "my-bucket"
// location.object == "datasets/train/image001.jpg"

// Check if object exists and get its size
auto stats = dali::s3_filesystem::get_stats(s3_client, location);
if (stats.exists) {
  std::vector<uint8_t> buffer(stats.size);
  size_t bytes_read = dali::s3_filesystem::read_object_contents(
      s3_client, location, buffer.data(), stats.size);
}

Listing Objects in an S3 Prefix

#include "dali/util/s3_filesystem.h"

auto prefix = dali::s3_filesystem::parse_uri("s3://my-bucket/datasets/train/");

dali::s3_filesystem::list_objects_f(s3_client, prefix,
    [](const std::string& key, size_t size) {
      // Process each object: key is the full object key, size is in bytes
    });

Related Pages

• Environment:NVIDIA_DALI_CUDA_GPU_Environment