Implementation:Microsoft LoRA Legacy Run SWAG
Template:Implementation metadata
Overview
Fine-tuning script for transformer models on the SWAG (Situations With Adversarial Generations) multiple choice dataset for grounded commonsense inference.
Description
run_swag.py is a legacy HuggingFace Transformers example script included in the Microsoft LoRA NLU example directory. It fine-tunes any model compatible with AutoModelForMultipleChoice on the SWAG dataset, where the task is to select the most plausible continuation of a given context sentence from four adversarially generated options.
The script implements the formatting approach proposed by Radford et al. ("Improving Language Understanding by Generative Pre-Training") where each of the four candidate endings is paired with the context as a separate input sequence formatted as [CLS] context [SEP] choice_i [SEP], and a softmax over the four outputs determines the final prediction.
It includes a complete pipeline with SwagExample and InputFeatures data classes, feature conversion with caching, a training loop with AdamW optimization, linear warmup, gradient accumulation, multi-GPU (DataParallel), distributed training (DDP), optional FP16 (Apex), TensorBoard logging, and periodic checkpoint saving. Evaluation computes loss and accuracy.
This script is part of the HuggingFace Transformers library (legacy examples) bundled in the Microsoft LoRA repository.
⚠️ DEPRECATED: This file resides in the legacy/ directory and is not actively maintained. Prefer modern equivalents where available.
Usage
Use this script to fine-tune BERT or other multiple-choice-capable transformers on the SWAG commonsense inference benchmark. The SWAG dataset tests whether a model can select the most plausible physical grounding continuation of a real-world scenario.
Code Reference
Source Location
| Property | Value |
|---|---|
| File path | examples/NLU/examples/legacy/run_swag.py
|
| Lines | 720 |
| Module | run_swag
|
Key Classes and Functions
| Name | Type | Signature / Description |
|---|---|---|
SwagExample |
class | __init__(self, swag_id, context_sentence, start_ending, ending_0, ending_1, ending_2, ending_3, label=None)
|
InputFeatures |
class | __init__(self, example_id, choices_features, label) -- stores per-choice input_ids, input_mask, segment_ids
|
read_swag_examples |
function | read_swag_examples(input_file, is_training=True) -- reads SWAG CSV into list of SwagExample
|
convert_examples_to_features |
function | convert_examples_to_features(examples, tokenizer, max_seq_length, is_training) -- tokenizes and encodes examples into InputFeatures
|
load_and_cache_examples |
function | load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False) -- loads with caching and distributed barriers
|
train |
function | train(args, train_dataset, model, tokenizer) -- full training loop; returns (global_step, avg_loss)
|
evaluate |
function | evaluate(args, model, tokenizer, prefix="") -- computes eval loss and accuracy
|
select_field |
function | select_field(features, field) -- extracts a field across all choices for all features
|
_truncate_seq_pair |
function | _truncate_seq_pair(tokens_a, tokens_b, max_length) -- truncates longer sequence first
|
main |
function | Entry point: parses args, initializes model, runs train/eval |
CLI Usage
python run_swag.py \ --model_name_or_path bert-base-uncased \ --do_train \ --do_eval \ --train_file /path/to/swag/train.csv \ --predict_file /path/to/swag/val.csv \ --output_dir /path/to/output \ --per_gpu_train_batch_size 8 \ --learning_rate 5e-5 \ --num_train_epochs 3.0 \ --max_seq_length 80
I/O Contract
Inputs
| Input | Type | Description |
|---|---|---|
--train_file |
str (required) |
Path to SWAG training CSV (e.g., train.csv)
|
--predict_file |
str (required) |
Path to SWAG evaluation CSV (e.g., val.csv)
|
--model_name_or_path |
str (required) |
Pretrained model name or path |
--output_dir |
str (required) |
Directory for checkpoints and results |
--max_seq_length |
int (default 384) |
Maximum total input sequence length after tokenization |
--per_gpu_train_batch_size |
int (default 8) |
Batch size per GPU for training |
--per_gpu_eval_batch_size |
int (default 8) |
Batch size per GPU for evaluation |
--learning_rate |
float (default 5e-5) |
Initial learning rate for AdamW |
--num_train_epochs |
float (default 3.0) |
Total training epochs |
--logging_steps |
int (default 50) |
Log metrics every N update steps |
--save_steps |
int (default 50) |
Save checkpoint every N update steps |
SWAG CSV Format
| Column Index | Content |
|---|---|
| 2 | SWAG ID |
| 4 | Context sentence |
| 5 | Start of the ending (common prefix) |
| 7-10 | Four candidate ending completions |
| 11 | Correct label (0-3) |
Outputs
| Output | Type | Description |
|---|---|---|
eval_results.txt |
text file | Evaluation loss and accuracy |
checkpoint-{step}/ |
directory | Saved model, tokenizer vocabulary, and training args |
training_args.bin |
binary | Serialized training arguments in output directory |
| Return value | Dict[str, float] |
Dict with eval_loss and eval_accuracy keys
|
Usage Examples
Fine-tuning BERT on SWAG
python run_swag.py \ --model_name_or_path bert-base-uncased \ --do_train \ --do_eval \ --train_file /data/swag/train.csv \ --predict_file /data/swag/val.csv \ --output_dir /output/swag_bert/ \ --per_gpu_train_batch_size 16 \ --learning_rate 5e-5 \ --num_train_epochs 3.0 \ --max_seq_length 80 \ --logging_steps 200 \ --save_steps 1000 \ --overwrite_output_dir
Distributed Training
python -m torch.distributed.launch --nproc_per_node=4 run_swag.py \ --model_name_or_path bert-large-uncased \ --do_train \ --do_eval \ --train_file /data/swag/train.csv \ --predict_file /data/swag/val.csv \ --output_dir /output/swag_bert_large/ \ --per_gpu_train_batch_size 8 \ --learning_rate 2e-5 \ --num_train_epochs 3.0 \ --max_seq_length 80 \ --overwrite_output_dir