Implementation:Microsoft LoRA WebNLG Test Dataset

Knowledge Sources	Microsoft_LoRA WebNLG_Challenge
Domains	NLG, Data_to_Text, Benchmarking, RDF_to_Text
Last Updated	2026-02-10 06:00 GMT

Overview

Test split of the WebNLG Challenge 2017 dataset, containing RDF triple-to-text examples organized by DBpedia categories for final evaluation of natural language generation from structured data.

Description

The WebNLG test set provides 135,056 lines of JSON data containing RDF triples from DBpedia paired with human-written lexicalisations. This is the held-out evaluation split used for reporting final benchmark results. Each entry includes a category label, modified RDF triples with subject/property/object fields, and multiple reference lexicalisations. A critical feature of the WebNLG test set is its inclusion of both "seen" categories (those present in training data, such as Airport, Astronaut, Building) and "unseen" categories (not present in training), which enables evaluation of a model's ability to generalize to novel domains. The gpt2_decode.py script supports filtering test results by --filter seen, --filter unseen, or --filter all to separately measure performance on each category subset.

Usage

Use this dataset as the held-out test split for final evaluation of LoRA-adapted GPT-2 models on RDF-to-text generation. After training and checkpoint selection on the dev set, generate predictions on this test set and evaluate using eval/eval.py with --ref_type webnlg. The evaluation supports multi-reference scoring with metrics including BLEU, METEOR, chrF++, TER, BERT-Score, and BLEURT. Results can be stratified by seen/unseen categories for comprehensive generalization analysis.

Code Reference

Source Location

• Repository: Microsoft_LoRA
• File: examples/NLG/data/webnlg_challenge_2017/test.json
• Converter: examples/NLG/src/format_converting_webnlg.py
• Decoder: examples/NLG/src/gpt2_decode.py
• Evaluator: examples/NLG/eval/eval.py

Data Schema

{
  "entries": [
    {
      "1": {
        "category": "Airport",
        "dbpedialinks": [],
        "lexicalisations": [
          {
            "comment": "good",
            "lang": "",
            "lex": "Abilene, Texas is served by the Abilene regional airport.",
            "xml_id": "Id1"
          },
          {
            "comment": "good",
            "lang": "",
            "lex": "Abilene Regional Airport serves the city of Abilene in Texas.",
            "xml_id": "Id2"
          }
        ],
        "links": [],
        "modifiedtripleset": [
          {
            "subject": "Abilene_Regional_Airport",
            "property": "cityServed",
            "object": "Abilene,_Texas"
          }
        ],
        "originaltriplesets": {
          "originaltripleset": [
            [
              {
                "subject": "Abilene_Regional_Airport",
                "property": "cityServed",
                "object": "Abilene,_Texas"
              }
            ],
            [
              {
                "subject": "Abilene_Regional_Airport",
                "property": "city",
                "object": "Abilene,_Texas"
              }
            ]
          ]
        },
        "shape": "NA",
        "shape_type": "NA",
        "size": "1",
        "xml_id": "Id1"
      }
    }
  ]
}

Format Conversion

The format_converting_webnlg.py script processes entries, filters for "good" lexicalisations, and marks categories as seen or unseen:

# Seen categories used during training
seen = ['Airport', 'Astronaut', 'Building', 'City', 'ComicsCharacter',
        'Food', 'Monument', 'SportsTeam', 'University', 'WrittenWork']

# Conversion produces lines like:
# {"context": "Abilene_Regional_Airport : cityServed : Abilene,_Texas",
#  "completion": "Abilene, Texas is served by the Abilene regional airport.",
#  "cate": true}

Loading

import json

with open("examples/NLG/data/webnlg_challenge_2017/test.json", "r") as f:
    data = json.load(f)

I/O Contract

Inputs

Name	Type	Required	Description
file_path	str	Yes	Path to the JSON file (e.g., examples/NLG/data/webnlg_challenge_2017/test.json)

Outputs

Name	Type	Description
data	Dict	Top-level dictionary with an entries key
entries	List[Dict]	List of numbered entry dictionaries
category	str	DBpedia category (e.g., Airport, Astronaut, Building); may include unseen categories
modifiedtripleset	List[Dict]	RDF triples with subject, property, object keys
lexicalisations	List[Dict]	Human-written reference texts with comment (quality), lex (text), lang, and xml_id
originaltriplesets	Dict	Original DBpedia triplesets preserving raw RDF formatting and alternate property mappings

Usage Examples

Loading and Inspecting WebNLG Test Data

import json

# Load the test set
with open("examples/NLG/data/webnlg_challenge_2017/test.json", "r") as f:
    test_data = json.load(f)

entries = test_data['entries']
print(f"Total entries: {len(entries)}")

# Inspect first example
first_entry = entries[0]["1"]
print(f"Category: {first_entry['category']}")
print(f"Triples: {first_entry['modifiedtripleset']}")
print(f"Lexicalisations ({len(first_entry['lexicalisations'])} references):")
for lex in first_entry['lexicalisations']:
    print(f"  [{lex['comment']}] {lex['lex']}")

End-to-End Evaluation Pipeline with Seen/Unseen Filtering

# Step 1: Convert test JSON to line-delimited format
python examples/NLG/src/format_converting_webnlg.py \
    examples/NLG/data/webnlg_challenge_2017/test.json \
    examples/NLG/data/webnlg_challenge_2017/test.jsonl

# Step 2: Decode model predictions (filter by seen/unseen/all)
python examples/NLG/src/gpt2_decode.py \
    --vocab examples/NLG/vocab \
    --sample_file output/webnlg_sample.jsonl \
    --input_file examples/NLG/data/webnlg_challenge_2017/test.jsonl \
    --output_ref_file output/webnlg_ref \
    --output_pred_file output/webnlg_pred.txt \
    --ref_type webnlg \
    --filter all

# Step 3: Run evaluation metrics
python examples/NLG/eval/eval.py \
    -R output/webnlg_ref/reference \
    -H output/webnlg_pred.txt \
    -nr 4 \
    -m bleu,meteor,ter,chrf++,bert

Related Pages

• Environment:Microsoft_LoRA_PyTorch_CUDA_Environment