Implementation:Mit han lab Llm awq LlavaConfig

Knowledge Sources	Mit_han_lab_Llm_awq
Domains	Model_Configuration, Multimodal
Last Updated	2026-02-15 00:00 GMT

Overview

Defines the LlavaConfig configuration class for LLaVA and NVILA multimodal models, encapsulating all sub-component paths and multimodal processing parameters.

Description

LlavaConfig extends HuggingFace's PretrainedConfig with model_type set to "llava". It serves as the top-level configuration for NVILA-family models, holding references to three sub-component configurations and a comprehensive set of multimodal processing parameters.

The three core sub-component fields are llm_cfg (language model configuration or path), vision_tower_cfg (vision encoder configuration or path), and mm_projector_cfg (multimodal projector configuration or path). These can be either config objects or string paths that are resolved during model construction.

Model dimension parameters include hidden_size (language model dimension) and mm_hidden_size (vision feature dimension). Vision processing parameters include image_aspect_ratio (handling strategy such as "dynamic" or "dynamic_s2"), num_video_frames, fps, mm_vision_select_layer (which hidden layer to extract features from), and mm_vision_select_feature (feature type selection). Token-related flags include mm_use_im_start_end and mm_use_im_patch_token for controlling special image token behavior.

Learning rate overrides mm_projector_lr and vision_tower_lr allow per-component learning rate scheduling during fine-tuning. The S2 (Scale-Square) dynamic resolution system is configured via s2, dynamic_s2, s2_scales, s2_max_split_size, and s2_resize_output_to_scale_idx. Tiling parameters min_tiles and max_tiles control dynamic resolution splitting bounds. JSON-encoded image_encoder and video_encoder strings specify encoder instantiation targets via Hydra.

Usage

Import LlavaConfig when loading or constructing NVILA models. It is typically loaded via AutoConfig.from_pretrained from a model checkpoint directory containing a config.json with model_type "llava".

Code Reference

Source Location

• Repository: Mit_han_lab_Llm_awq
• File: tinychat/models/nvila/configuration_llava.py
• Lines: 1-89

Signature

class LlavaConfig(PretrainedConfig):
    model_type = "llava"

    def __init__(
        self,
        llm_cfg=None,
        vision_tower_cfg=None,
        mm_projector_cfg=None,
        architectures=None,
        resume_path=None,
        hidden_size=None,
        mm_hidden_size=None,
        image_aspect_ratio=None,
        num_video_frames=None,
        fps=None,
        mm_vision_select_layer=None,
        mm_vision_select_feature=None,
        mm_use_im_start_end=False,
        mm_use_im_patch_token=False,
        mm_projector_lr=None,
        vision_tower_lr=None,
        vision_resolution=None,
        interpolate_mode=None,
        s2=None,
        dynamic_s2=None,
        s2_scales=None,
        s2_max_split_size=None,
        s2_resize_output_to_scale_idx=0,
        min_tiles: Optional[int] = 1,
        max_tiles: Optional[int] = 12,
        num_time_tokens=None,
        time_token_format=None,
        image_encoder: str = '...',
        video_encoder: str = '...',
        **kwargs,
    ): ...

Import

from tinychat.models.nvila.configuration_llava import LlavaConfig

I/O Contract

Inputs

Name	Type	Required	Description
llm_cfg	str or dict	No	Language model config path or object
vision_tower_cfg	str or dict	No	Vision tower config path or object
mm_projector_cfg	str or dict	No	Multimodal projector config path or object
hidden_size	int	No	Language model hidden dimension
mm_hidden_size	int	No	Vision feature hidden dimension
image_aspect_ratio	str	No	Image aspect ratio handling strategy (e.g., "dynamic", "dynamic_s2")
num_video_frames	int	No	Number of frames to sample from video inputs
s2	bool	No	Enable S2 (Scale-Square) multi-scale processing
dynamic_s2	bool	No	Enable dynamic S2 with variable tile counts
s2_scales	str or list	No	Comma-separated scales or list of scale values
min_tiles	int	No	Minimum number of tiles for dynamic resolution (default 1)
max_tiles	int	No	Maximum number of tiles for dynamic resolution (default 12)

Outputs

Name	Type	Description
config	LlavaConfig	Configuration object with all multimodal parameters as attributes

Usage Examples

Creating a config manually

from tinychat.models.nvila.configuration_llava import LlavaConfig

config = LlavaConfig(
    llm_cfg="meta-llama/Llama-2-7b-hf",
    vision_tower_cfg="openai/clip-vit-large-patch14",
    mm_projector_cfg="mlp2x_gelu",
    hidden_size=4096,
    mm_hidden_size=1024,
    image_aspect_ratio="dynamic_s2",
    dynamic_s2=True,
    s2_scales="336,672,1008",
)

Loading from a pretrained checkpoint

from transformers import AutoConfig

config = AutoConfig.from_pretrained("path/to/nvila-model")
# config is an instance of LlavaConfig
print(config.llm_cfg, config.vision_tower_cfg)

Related Pages

• Principle:Mit_han_lab_Llm_awq_Multimodal_Model_Configuration