Implementation:Open compass VLMEvalKit MEGABench Constrained Generation
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, NLP, Constrained Generation, Poetry |
Overview
Implements scoring functions for constrained text generation tasks in MEGA-Bench, including poetry evaluation with rhyme, meter, and lemmatization checks.
Description
This module provides specialized scoring for constrained generation tasks such as poetry creation and structured text output. It includes custom_lemmatize and lemmatize_phrase for WordNet-based lemmatization, phonetic analysis using the pronouncing library with custom pronunciation dictionaries for rare words, and syllable counting for meter verification. The module evaluates constraints like paragraph count, sentence count, rhyme schemes, and vocabulary restrictions. NLTK tokenization is used for sentence/word segmentation, and the evaluator supports multiple constraint types through a modular checking system.
Usage
Called internally by the MEGA-Bench evaluator for constrained generation task scoring.
Code Reference
- Source:
vlmeval/dataset/utils/megabench/scoring/constrained_generation.py, Lines: L1-518 - Import:
from vlmeval.dataset.utils.megabench.scoring.constrained_generation import custom_lemmatize, lemmatize_phrase
Key Functions:
def custom_lemmatize(word, lemmatizer): ...
def lemmatize_phrase(phrase, lemmatizer): ...
def check_whether_response_paragraph_number_in_range(response, lower_bound, upper_bound): ...
def check_whether_response_sentence_number_in_range(response, lower_bound, upper_bound): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model-generated text responses; constraint parameters (bounds for paragraph/sentence counts, rhyme patterns, etc.) |
| Outputs | Boolean values indicating whether each constraint is satisfied; aggregate constraint satisfaction scores |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.megabench.scoring.constrained_generation import check_whether_response_paragraph_number_in_range
is_valid = check_whether_response_paragraph_number_in_range(response, 2, 4)