Implementation:Hpcaitech ColossalAI Eval JSON Config

Knowledge Sources	ColossalAI
Domains	Evaluation, Infrastructure
Last Updated	2026-02-09 00:00 GMT

Overview

JSON configuration files for specifying evaluation pipeline parameters, provided by ColossalEval.

Description

The evaluation system uses two JSON configuration files: inference/config.json defines model wrapper classes, model kwargs, dataset classes, and dataset paths; evaluation/config.json defines which metrics to compute for each dataset.

Usage

Edit the JSON config files before running inference and evaluation scripts.

Code Reference

Source Location

• Repository: ColossalAI
• File (inference config): applications/ColossalEval/examples/dataset_evaluation/config/inference/config.json
• File (eval config): applications/ColossalEval/examples/dataset_evaluation/config/evaluation/config.json

Signature

// inference/config.json structure
{
    "model": [
        {
            "name": "model_name",
            "model_class": "HuggingFaceModel",
            "model_kwargs": {"path": "/path/to/model", "max_new_tokens": 256}
        }
    ],
    "dataset": [
        {
            "name": "mmlu",
            "dataset_class": "MMLUDataset",
            "path": "/path/to/mmlu",
            "save_path": "/results/mmlu.json",
            "few_shot": true
        }
    ]
}

Import

# Config files are loaded via json.load() in the scripts
import json
with open("config/inference/config.json") as f:
    config = json.load(f)

I/O Contract

Inputs

Name	Type	Required	Description
model[].name	str	Yes	Model identifier
model[].model_class	str	Yes	HuggingFaceModel or vLLMModel
model[].model_kwargs	Dict	Yes	Path, max_new_tokens, etc.
dataset[].name	str	Yes	Dataset identifier
dataset[].dataset_class	str	Yes	MMLUDataset, GSMDataset, etc.
dataset[].path	str	Yes	Raw dataset directory
dataset[].save_path	str	Yes	Output JSON path

Outputs

Name	Type	Description
Parsed config	Dict	Configuration consumed by inference and evaluation scripts

Usage Examples

{
    "model": [
        {
            "name": "llama-7b",
            "model_class": "HuggingFaceModel",
            "model_kwargs": {
                "path": "meta-llama/Llama-2-7b-hf",
                "max_new_tokens": 256,
                "batch_size": 8
            }
        }
    ],
    "dataset": [
        {
            "name": "mmlu",
            "dataset_class": "MMLUDataset",
            "path": "/benchmarks/mmlu",
            "save_path": "/results/mmlu.json",
            "few_shot": true
        },
        {
            "name": "gsm8k",
            "dataset_class": "GSMDataset",
            "path": "/benchmarks/gsm8k",
            "save_path": "/results/gsm8k.json",
            "few_shot": true
        }
    ]
}

Related Pages

Implements Principle

• Principle:Hpcaitech_ColossalAI_Evaluation_Configuration