Implementation:Facebookresearch Habitat lab VQA Trainer

Knowledge Sources	Facebookresearch_Habitat_lab
Domains	Embodied_AI, Embodied_Question_Answering, Visual_Question_Answering
Last Updated	2026-02-15 00:00 GMT

Overview

VQATrainer is a trainer class for the Visual Question Answering model used in the Embodied Question Answering pipeline, training a VqaLstmCnnAttentionModel to answer questions about visual observations.

Description

The VQATrainer class extends BaseILTrainer and is registered under the name "vqa" in the baseline registry. It trains the VqaLstmCnnAttentionModel, which combines a pretrained MultitaskCNN encoder, an LSTM question encoder, and an attention mechanism to answer questions about sequences of image frames.

Training: The trainer loads the EQADataset configured for VQA input type with a configurable number of frames (default 5). The dataset is shuffled and converted to tuples of (episode_id, question, answer, frame_0, ..., frame_4). The trainer uses CrossEntropyLoss and the Adam optimizer. It tracks four metrics via VqaMetric: loss, accuracy, mean rank, and mean reciprocal rank. The CNN encoder can optionally be frozen during VQA training to only update the question encoder and classifier. Checkpoints are saved every epoch.

Evaluation: The evaluation loads a checkpoint, runs the model on the validation split, and computes the same four metrics. It optionally saves visual results showing the input images along with predicted and ground-truth answers at configurable intervals.

The trainer also supports optional freezing of the pretrained CNN encoder via the freeze_encoder configuration option, which keeps the encoder in eval mode and prevents gradient updates to its parameters.

Usage

Use this trainer to train and evaluate the VQA answering module in the EQA pipeline. It requires a pretrained MultitaskCNN checkpoint (produced by EQACNNPretrainTrainer) for visual feature extraction. The VQA model learns to attend to the most relevant frame in a sequence and combine visual and question features to predict an answer.

Code Reference

Source Location

• Repository: Facebookresearch_Habitat_lab
• File: habitat-baselines/habitat_baselines/il/trainers/vqa_trainer.py
• Lines: 1-433

Signature

@baseline_registry.register_trainer(name="vqa")
class VQATrainer(BaseILTrainer):
    supported_tasks = ["VQA-v0"]

    def __init__(self, config=None): ...
    def _make_results_dir(self) -> None: ...
    def _save_vqa_results(
        self,
        ckpt_idx: int,
        episode_ids: torch.Tensor,
        questions: torch.Tensor,
        images: torch.Tensor,
        pred_scores: torch.Tensor,
        gt_answers: torch.Tensor,
        q_vocab_dict: VocabDict,
        ans_vocab_dict: VocabDict,
    ) -> None: ...
    def train(self) -> None: ...
    def _eval_checkpoint(
        self,
        checkpoint_path: str,
        writer: TensorboardWriter,
        checkpoint_index: int = 0,
    ) -> None: ...

Import

from habitat_baselines.il.trainers.vqa_trainer import VQATrainer

I/O Contract

Inputs

Name	Type	Required	Description
config	DictConfig	Yes	Habitat baselines configuration containing VQA parameters (batch_size, lr, max_epochs, num_frames, freeze_encoder), dataset paths, and the pretrained CNN checkpoint path

Outputs

Name	Type	Description
checkpoint files	file	Model state dict saved as epoch_{N}.ckpt at every epoch
TensorBoard logs	file	Training and evaluation metrics: loss, accuracy, mean_rank, mean_reciprocal_rank
VQA result images	file	Optional JPEG images showing the input frames with predicted and ground-truth answers during evaluation

Usage Examples

Basic Usage

from habitat_baselines.common.baseline_registry import baseline_registry

# The trainer is registered and typically invoked via the run script:
# python -m habitat_baselines.run --config-name=eqa/il_vqa.yaml

# Programmatic usage:
config = ...  # load Hydra config
trainer_cls = baseline_registry.get_trainer("vqa")
trainer = trainer_cls(config=config)

# Training
trainer.train()

# Evaluation (called internally by eval loop)
# trainer.eval()