Environment:Open compass VLMEvalKit Python Runtime Environment
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, VLM_Evaluation |
| Last Updated | 2026-02-14 01:30 GMT |
Overview
Python 3.7+ runtime environment with 60+ Python packages for multimodal VLM evaluation, including PyTorch, Transformers, OpenAI, and various benchmark-specific dependencies.
Description
This environment provides the core Python runtime and package dependencies required to run VLMEvalKit. The package is distributed as a pip-installable Python package (vlmeval) with a comprehensive requirements file covering deep learning frameworks (PyTorch, Transformers), API clients (OpenAI, Google GenAI), data processing libraries (Pandas, NumPy, Pillow), and benchmark-specific evaluation tools (NLTK, SymPy, apted, TEDS). The package version is 0.2rc1 as declared in `vlmeval/__init__.py:21`.
Usage
Use this environment as the base prerequisite for all VLMEvalKit workflows. Every workflow (Image Benchmark Evaluation, Video Benchmark Evaluation, API Model Evaluation, Adding Custom VLM, Adding Custom Benchmark) requires this Python runtime to be properly configured.
System Requirements
| Category | Requirement | Notes |
|---|---|---|
| OS | Linux (Ubuntu recommended) | Windows/macOS not officially tested |
| Python | >= 3.7.0 | Declared in `setup.py:101` via `python_requires='>=3.7.0'` |
| Disk | 2GB+ for packages | Additional space needed for datasets (see Data_Storage_Environment) |
Dependencies
System Packages
- `git` (for git hash tracking in `vlmeval/smp/misc.py:176`)
- `nvidia-smi` (optional, for GPU detection in `run.py:14`)
- `ffmpeg` (optional, for video processing benchmarks)
Python Packages (Core)
- `torch` (no version pinned)
- `torchvision` (no version pinned)
- `transformers` (no version pinned)
- `accelerate`
- `huggingface_hub`
- `openai`
- `google-genai`
Python Packages (Data Processing)
- `pandas`
- `numpy`
- `pillow`
- `opencv-python` >= 4.7.0.72
- `omegaconf` >= 2.4.0.dev4
- `datasets`
Python Packages (Evaluation)
- `nltk`
- `sympy`
- `scikit-learn`
- `math-verify`
- `pylatexenc` == 2.10
- `antlr4-python3-runtime` == 4.11.1
- `apted` >= 1.0.3
- `levenshtein` >= 0.27.1
- `editdistance` >= 0.8.1
- `anls` >= 0.0.2
Python Packages (Video)
- `decord` >= 0.6.0
- `imageio`
Python Packages (Utilities)
- `python-dotenv` (for `.env` file loading)
- `rich` (for progress bars)
- `tqdm`
- `tabulate`
- `tiktoken`
- `portalocker`
- `sty` (for colored terminal output)
- `json_repair`
- `requests`
- `validators`
Python Packages (Optional Benchmarks)
- `torchmetrics`, `scikit-image`, `lpips`, `openai-clip`, `cairosvg` (for SArena benchmark)
- `sentence_transformers`, `bert_score` (for UniSVG benchmark)
- `colormath` >= 3.0.0 (for color evaluation in ChartMimic)
- `lxml` >= 6.0.2 (for TEDS metric)
- `pdf2image` >= 1.17.0 (for document benchmarks)
Credentials
No credentials are required for the base Python runtime itself. See Environment:Open_compass_VLMEvalKit_API_Keys_And_Credentials for API key requirements.
Quick Install
# Clone the repository
git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit
# Install the package and all dependencies
pip install -e .
# Or install from requirements directly
pip install -r requirements.txt
Code Evidence
Python version requirement from `setup.py:101`:
python_requires='>=3.7.0',
Package version from `vlmeval/__init__.py:21`:
__version__ = '0.2rc1'
SSL bypass at module import from `vlmeval/__init__.py:1-3`:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
# Temporarily bypass SSL certificate verification to download files from oss.
Optional torch import from `vlmeval/__init__.py:5-8`:
try:
import torch
except ImportError:
pass
Common Errors
| Error Message | Cause | Solution |
|---|---|---|
| `ImportError: No module named 'vlmeval'` | Package not installed | Run `pip install -e .` from repo root |
| `ModuleNotFoundError: No module named 'dotenv'` | python-dotenv not installed | `pip install python-dotenv` |
| `AssertionError: Unsupported PRED_FORMAT xxx` | Invalid PRED_FORMAT env var | Set to one of: `tsv`, `xlsx`, `json` (or unset for default `xlsx`) |
| `Failed to dump to {suffix} format, falling back to pkl` | Output format serialization failed | Check write permissions; pickle fallback is automatic (`vlmeval/smp/file.py:169`) |
Compatibility Notes
- SSL Verification: VLMEvalKit disables SSL certificate verification globally at import time (`vlmeval/__init__.py:1-2`) to allow downloading from OSS. This is a known security trade-off.
- PyTorch Optional: PyTorch is wrapped in a try/except at import time, meaning the package can be imported on CPU-only systems for API-only workflows.
- Python 3.7 Minimum: While `setup.py` declares Python 3.7+, classifiers list 3.7-3.10. Newer Python versions (3.11, 3.12) are likely compatible but not officially listed.