Implementation:Hpcaitech ColossalAI SafetyBenchZHDataset
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Benchmarking |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
SafetyBenchZHDataset is a dataset wrapper class that loads and converts the Chinese portion of the SafetyBench safety evaluation benchmark into the ColossalEval inference format, assessing model safety via Chinese-language multiple-choice questions.
Description
The class extends BaseDataset and provides a static load method that reads Chinese dev and test JSON files ("dev_zh.json" and "test_zh.json") from the SafetyBench dataset. It mirrors the structure of SafetyBenchENDataset but with Chinese language settings: the instruction prompt reads "以下是关于安全考试的单项选择题" and the input format uses Chinese question/answer formatting. Questions are formatted as multiple-choice with four options (A-D), and options are padded with "NULL" when PAD_CHOICES is enabled. The module reuses the same helper functions (process_test, process_dev, get_few_shot_data, add_few_shot_to_test) as the English variant.
Usage
Use this class when you need to evaluate a language model on Chinese safety-related multiple-choice questions within the ColossalEval framework. It supports few-shot evaluation using dev-split examples.
Code Reference
Source Location
- Repository: Hpcaitech_ColossalAI
- File: applications/ColossalEval/colossal_eval/dataset/safetybench_zh.py
- Lines: 1-152
Signature
class SafetyBenchZHDataset(BaseDataset):
@staticmethod
def load(path: str, logger: DistributedLogger, few_shot: bool, *args, **kwargs) -> List[Dict]:
Import
from colossal_eval.dataset.safetybench_zh import SafetyBenchZHDataset
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| path | str | Yes | Path to the directory containing "dev_zh.json" and "test_zh.json" files |
| logger | DistributedLogger | Yes | Logger instance for distributed logging |
| few_shot | bool | Yes | Whether to prepend dev-split examples as few-shot demonstrations for the test split |
Outputs
| Name | Type | Description |
|---|---|---|
| dataset | Dict[str, Dict] | A nested dictionary with "dev" and "test" splits, each containing per-category entries with "data" (list of data samples with fields dataset, split, category, instruction, input, output, target, id) and "inference_kwargs" (calculate_loss=False, all_classes=["A","B","C","D"], language="Chinese", max_new_tokens=32) |
Usage Examples
from colossal_eval.dataset.safetybench_zh import SafetyBenchZHDataset
from colossalai.logging import DistributedLogger
logger = DistributedLogger("safetybench_zh")
dataset = SafetyBenchZHDataset(path="/path/to/safetybench/data", logger=logger, few_shot=True)
dataset.save("/path/to/output.json")