Implementation:Open compass VLMEvalKit MEGABench Program Judge

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Code Generation, Automated Testing

Overview

Implements automated code evaluation through test case execution for programming tasks in the MEGA-Bench evaluation framework.

Description

The `ProgramJudge` class evaluates model-generated code by running it against predefined test cases. The `CodeTester` class takes user code and test cases, executing them in isolated processes with a 2-second timeout per test. The `match` static method orchestrates the evaluation pipeline: loading test cases from the evaluation context, creating a CodeTester instance, running tests, and returning an aggregate score. Results can be saved to JSON files for debugging via `save_test_results`.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/megabench/scoring/program_judge.py, Lines: L1-141
• Import: from vlmeval.dataset.utils.megabench.scoring.program_judge import ProgramJudge

Key Functions:

class ProgramJudge:
    @staticmethod
    def match(response, eval_context, task_info=None) -> int: ...
    @classmethod
    def save_test_results(cls, task_name, results, query_file): ...

class CodeTester:
    def run_tests(self): ...

I/O Contract

Direction	Description
Inputs	Model-generated code string and evaluation context dict with 'test_case' key containing test cases
Outputs	Integer score representing the fraction of test cases passed

Usage Examples

from vlmeval.dataset.utils.megabench.scoring.program_judge import ProgramJudge

score = ProgramJudge.match(code_response, {"test_case": test_cases})

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction