Implementation:Microsoft LoRA DART Test Dataset

Knowledge Sources	Microsoft_LoRA DART
Domains	NLG, Data_to_Text, Benchmarking
Last Updated	2026-02-10 06:00 GMT

Overview

Test split of the DART (Data-Record to Text) benchmark dataset, containing structured tripleset-to-text examples for final evaluation of data-to-text generation models.

Description

The DART test set provides 214,471 lines of JSON data containing structured tripleset records paired with human-written text annotations. Each example consists of a set of subject-relation-object triples (derived from sources such as WikiSQL, WikiTableQuestions, and other structured data) along with one or more reference natural language descriptions. The triples use uppercase relation names (e.g., NOTES, ROLE, YEAR, TITLE) and may include special context markers like [TABLECONTEXT] and [TITLE] for table-derived examples. Multiple annotations per example enable multi-reference evaluation using metrics such as BLEU, METEOR, chrF++, TER, and BERT-Score. This dataset is used for the final held-out evaluation of LoRA-adapted GPT-2 models.

Usage

Use this dataset as the held-out test split for final evaluation of data-to-text generation models fine-tuned with LoRA. After training and checkpoint selection on the dev set, generate predictions on this test set and evaluate using the provided eval/eval.py script with --ref_type dart. The gpt2_decode.py script processes test set outputs into reference/hypothesis files for multi-reference evaluation.

Code Reference

Source Location

• Repository: Microsoft_LoRA
• File: examples/NLG/data/dart/dart-v1.1.1-full-test.json
• Converter: examples/NLG/src/format_converting_dart.py
• Decoder: examples/NLG/src/gpt2_decode.py
• Evaluator: examples/NLG/eval/eval.py

Data Schema

[
  {
    "tripleset": [
      ["Hawaii Five-O", "NOTES", "Episode: The Flight of the Jewels"],
      ["[TABLECONTEXT]", "[TITLE]", "Jeff Daniels"],
      ["[TABLECONTEXT]", "TITLE", "Hawaii Five-O"]
    ],
    "subtree_was_extended": true,
    "annotations": [
      {
        "source": "WikiTableQuestions_lily",
        "text": "Jeff Daniels played in the Hawaii Five-O episode The Flight of the Jewels"
      }
    ]
  }
]

Format Conversion

The format_converting_dart.py script transforms each example into a context/completion pair by joining triples with " | " separators and using " : " between subject, relation, and object:

# Conversion produces lines like:
# {"context": "Hawaii Five-O : notes : Episode: The Flight of the Jewels | [TABLECONTEXT] : [title] : Jeff Daniels | [TABLECONTEXT] : title : Hawaii Five-O",
#  "completion": "Jeff Daniels played in the Hawaii Five-O episode The Flight of the Jewels"}

Loading

import json

with open("examples/NLG/data/dart/dart-v1.1.1-full-test.json", "r") as f:
    data = json.load(f)

I/O Contract

Inputs

Name	Type	Required	Description
file_path	str	Yes	Path to the JSON file (e.g., examples/NLG/data/dart/dart-v1.1.1-full-test.json)

Outputs

Name	Type	Description
data	List[Dict]	Top-level JSON array of examples, each containing tripleset, subtree_was_extended, and annotations
tripleset	List[List[str]]	List of subject-relation-object triples, each a 3-element string list
subtree_was_extended	bool	Whether the data subtree was augmented during dataset construction
annotations	List[Dict]	Human-written reference text descriptions with source attribution (keys: source, text)

Usage Examples

Loading and Inspecting DART Test Data

import json

# Load the test set
with open("examples/NLG/data/dart/dart-v1.1.1-full-test.json", "r") as f:
    test_data = json.load(f)

# Inspect first example
example = test_data[0]
print(f"Triples: {example['tripleset']}")
print(f"Extended: {example['subtree_was_extended']}")
print(f"Annotations: {len(example['annotations'])} reference(s)")
for ann in example['annotations']:
    print(f"  [{ann['source']}] {ann['text']}")
print(f"Total examples: {len(test_data)}")

End-to-End Evaluation Pipeline

# Step 1: Convert test JSON to line-delimited format
python examples/NLG/src/format_converting_dart.py \
    examples/NLG/data/dart/dart-v1.1.1-full-test.json \
    examples/NLG/data/dart/dart-v1.1.1-full-test.jsonl

# Step 2: Decode model predictions into reference/hypothesis files
python examples/NLG/src/gpt2_decode.py \
    --vocab examples/NLG/vocab \
    --sample_file output/dart_sample.jsonl \
    --input_file examples/NLG/data/dart/dart-v1.1.1-full-test.jsonl \
    --output_ref_file output/dart_ref \
    --output_pred_file output/dart_pred.txt \
    --ref_type dart

# Step 3: Run evaluation metrics
python examples/NLG/eval/eval.py \
    -R output/dart_ref/reference \
    -H output/dart_pred.txt \
    -nr 4 \
    -m bleu,meteor,ter,chrf++,bert

Related Pages

• Environment:Microsoft_LoRA_PyTorch_CUDA_Environment