Implementation:Open compass VLMEvalKit MMIF Function And Compare
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Instruction Following, Constraint Checking |
Overview
Implements constraint checking functions for the MMIF (MultiModal Instruction Following) benchmark evaluation, validating structural and content constraints in model responses.
Description
This module provides a collection of constraint-checking functions for evaluating whether model-generated text satisfies specified formatting and content requirements. Functions include check_whether_response_paragraph_number_in_range for paragraph count validation, check_whether_response_sentence_number_in_range for sentence count validation using NLTK tokenization, and check_whether_each_paragraph_sentence_number_in_range for per-paragraph sentence bounds. The functions handle text cleaning, paragraph splitting via double-newline detection, and NLTK sentence tokenization for accurate structural analysis.
Usage
Called internally by the MMIF dataset class during instruction-following evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/mmif/function_and_compare.py, Lines: L1-429 - Import:
from vlmeval.dataset.utils.mmif.function_and_compare import check_whether_response_paragraph_number_in_range
Key Functions:
def check_whether_response_paragraph_number_in_range(response, lower_bound, upper_bound): ...
def check_whether_response_sentence_number_in_range(response, lower_bound, upper_bound): ...
def check_whether_each_paragraph_sentence_number_in_range(response, lower_bound, upper_bound): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model response text string; integer lower and upper bounds for the constraint being checked |
| Outputs | Boolean indicating whether the response satisfies the specified structural constraint |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.mmif.function_and_compare import check_whether_response_paragraph_number_in_range
is_valid = check_whether_response_paragraph_number_in_range(response_text, 2, 5)