Implementation:Facebookresearch Habitat lab EqaDataset

Knowledge Sources	Facebookresearch_Habitat_lab
Domains	Embodied_AI, Embodied_Question_Answering
Last Updated	2026-02-15 00:00 GMT

Overview

EQADataset is a PyTorch-compatible WebDataset for Embodied Question Answering (EQA) that caches rendered episode frames to disk as tar archives and streams them during training for both VQA and PACMAN models.

Description

EQADataset extends webdataset.Dataset to support the Embodied Q&A task. On initialization, it creates a Habitat environment, loads the episode dataset, sorts episodes by ID, computes maximum question and action lengths for padding, and restructures the answer vocabulary for contiguous indexing. If a cached tar archive of rendered frames does not exist, the dataset iterates over all scenes and episodes, renders frames along the shortest path, saves them as JPEG images, and archives them into a tar file. A custom group_by_keys_ pipeline function groups the tar entries by episode and attaches question tokens and answer labels. The dataset supports both VQA-only mode (using only the last N frames) and full PACMAN mode (using all shortest path frames).

Usage

Use EQADataset when training EQA models (VQA or PACMAN). Provide a configuration specifying the Habitat environment, dataset split, and frame storage paths. The dataset handles caching automatically.

Code Reference

Source Location

• Repository: Facebookresearch_Habitat_lab
• File: habitat-baselines/habitat_baselines/il/data/data.py
• Lines: 28-262

Signature

class EQADataset(wds.Dataset):
    def __init__(
        self,
        config,
        input_type,
        num_frames=5,
        max_controller_actions=5,
    ):

Import

from habitat_baselines.il.data.data import EQADataset

I/O Contract

Inputs

Name	Type	Required	Description
config	DictConfig	Yes	Configuration object containing habitat environment and baselines settings
input_type	str	Yes	Type of model being trained: "vqa" or "pacman"
num_frames	int	No	Number of frames used as input to VQA model (default: 5)
max_controller_actions	int	No	Maximum number of controller actions (default: 5)

Outputs

Name	Type	Description
(iterable samples)	dict	Each sample contains episode_id, question (LongTensor), answer (int), and frame images keyed by index suffix

Key Methods

get_vocab_dicts

def get_vocab_dicts(self) -> Tuple[VocabDict, VocabDict]

Returns the question and answer VocabDict objects.

calc_max_length

def calc_max_length(self) -> None

Computes maximum question token length and maximum action sequence length across all episodes.

save_frame_queue

def save_frame_queue(self, pos_queue: List[ShortestPathPoint], episode_id) -> None

Renders and saves JPEG frames for an episode's position queue.

Usage Examples

Basic Usage

from habitat_baselines.il.data.data import EQADataset
from torch.utils.data import DataLoader

dataset = EQADataset(
    config=my_config,
    input_type="vqa",
    num_frames=5,
)

q_vocab, ans_vocab = dataset.get_vocab_dicts()

dataloader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=False,
)

for batch in dataloader:
    questions = batch["question"]
    answers = batch["answer"]
    # Process frames and questions through VQA model