Implementation:Open compass VLMEvalKit Use Custom Prompt

Source	Domain
VLMEvalKit	Vision, NLP, Software_Design

Overview

Interface specification for model-specific prompt dispatch in VLMEvalKit's inference pipeline.

Description

use_custom_prompt(dataset) is defined in BaseModel (vlmeval/vlm/base.py:L14-24) returning False by default. build_prompt(line, dataset) is an abstract method at L26-37. Subclasses override use_custom_prompt() to return True for specific datasets, then provide corresponding build_prompt() implementation that formats the raw data row into a model-specific message list. The inference pipeline checks model.use_custom_prompt(dataset_name) and calls model.build_prompt(item, dataset=dataset_name) if True, otherwise calls dataset.build_prompt(item).

Usage

Override in your VLM adapter for model-specific prompt formatting.

Code Reference

• Source: vlmeval/vlm/base.py, Lines: L14-37
• Signature:

def use_custom_prompt(self, dataset: str) -> bool:
    """Whether to use custom prompt for the given dataset.
    Default: returns False.
    Override to return True for datasets needing model-specific prompts.
    """
    return False

@abstractmethod
def build_prompt(self, line: pd.Series, dataset: str) -> Union[str, List[Dict]]:
    """Build custom prompt for a specific dataset.
    Called only if use_custom_prompt() returns True.
    Args:
        line: Raw data row from dataset DataFrame.
        dataset: Dataset name string.
    Returns:
        Message list or prompt string.
    """

• Import: (method on BaseModel) from vlmeval.vlm.base import BaseModel

I/O Contract

Method	Inputs	Outputs
use_custom_prompt	dataset (str)	bool
build_prompt	line (pd.Series), dataset (str)	str or List[Dict]

Usage Examples

class MyCPMModel(BaseModel):
    def use_custom_prompt(self, dataset):
        # Custom prompts for MCQ benchmarks
        return dataset in ['MMBench', 'MMStar', 'AI2D_TEST']

    def build_prompt(self, line, dataset):
        # Format prompt with special tokens
        msgs = []
        if 'image' in line:
            msgs.append(dict(type='image', value=self.dump_image(line, dataset)[0]))
        question = line['question']
        if dataset in ['MMBench', 'MMStar']:
            # Add options with special formatting
            options = ""
            for c in 'ABCD':
                if c in line and not pd.isna(line[c]):
                    options += f"({c}) {line[c]}\n"
            question = f"{question}\n{options}Answer:"
        msgs.append(dict(type='text', value=question))
        return msgs

Related Pages

• Principle:Open_compass_VLMEvalKit_Custom_Prompt_Building