Implementation:OpenBMB UltraFeedback GPT4 Preference Annotator

Knowledge Sources	UltraFeedback OpenAI API
Domains	NLP, Evaluation
Last Updated	2023-10-02 00:00 GMT

Overview

Concrete tool for generating multi-aspect GPT-4 preference annotations using rubric templates and randomized completion ordering.

Description

The annotate_preference.py module provides the core annotation pipeline with three main components:

process(responses, aspect): Parses GPT-4's structured response into annotation dictionaries. Uses regex patterns specific to each aspect type:

instruction_following/honesty: r"Rating: (.+?)\nRationale: (.+)" → {Rating, Rationale}
truthfulness/helpfulness: r"Type: (.+?)\nRationale: (.+?)\nRating: (.+?)\nRationale: (.+)" → {Type, Rationale, Rating, Rationale For Rating}

get_eval(sys_prompt, user_prompt, max_tokens=500): Calls openai.ChatCompletion.create with model="gpt-4", temperature=0, max_tokens=500, top_p=0.6. Includes retry logic (10 attempts).

annotate(example): Orchestrates the full annotation process: for each of 4 aspects, generates random completion orderings, formats the appropriate template with instruction and completion texts, calls GPT-4, parses the response, and maps ratings back to original completion positions.

The preference_templates.py module provides 6 rubric templates: instruction_following_template, honesty_template, truthfulness_template_with_answer, truthfulness_template_without_answer, helpfulness_template_with_answer, helpfulness_template_without_answer. Templates are selected based on whether world knowledge is available for the subset.

Usage

Run as a standalone script. Configure the TEMPLATE dict to map aspect names to template strings, set the subsets list, and iterate over examples calling annotate() for each.

Code Reference

Source Location

Repository: UltraFeedback
File: src/data_annotation/annotate_preference.py (Lines 18-162)
File: src/data_annotation/preference_templates.py (Lines 1-477)

Signature

def process(responses: str, aspect: str) -> List[Dict]:
    """Parses GPT-4 response into structured annotations.
    Args:
        responses: Raw GPT-4 output, split by double newlines into 4 blocks
        aspect: One of "instruction_following", "honesty", "truthfulness", "helpfulness"
    Returns:
        List of 4 annotation dicts
    Raises:
        ValueError, AttributeError: If response doesn't match expected format
    """
    ...

def get_eval(sys_prompt: str, user_prompt: str, max_tokens: int = 500) -> str:
    """Calls GPT-4 via openai.ChatCompletion.create.
    Args:
        sys_prompt: System prompt from preference_templates
        user_prompt: Formatted rubric template with instruction and completions
        max_tokens: Maximum response length (default 500)
    Returns:
        GPT-4 response content string
    """
    ...

SHUFLLE_NUM = 1  # Number of random orderings per aspect

def annotate(example: Dict) -> Dict:
    """Annotates all completions across 4 aspects with randomized ordering.
    Args:
        example: Dict with 'instruction', 'completions' (4 completions with responses)
    Returns:
        example: Same dict with completions enriched with 'annotations' dict
                 containing per-aspect ratings and rationales
    """
    ...

Import

import openai
import re
import random
from copy import deepcopy
from preference_templates import (
    system_prompt,
    instruction_following_template,
    truthfulness_template,
    honesty_template,
    harmlessness_template,
    helpfulness_template
)

I/O Contract

Inputs

Name	Type	Required	Description
example	Dict	Yes	Dataset example with 'instruction' and 'completions' (4 entries, each with 'response')
subset	str	Yes	Global variable: determines world knowledge source
TEMPLATE	Dict[str, str]	Yes	Global variable: maps aspect names to template strings
example["correct_answers"]	List[str]	No	TruthfulQA/FLAN: ground-truth correct answers
example["incorrect_answers"]	List[str]	No	TruthfulQA: known incorrect answers

Outputs

Name	Type	Description
example["completions"][i]["annotations"]	Dict[str, List[Dict]]	Per-aspect annotation lists. instruction_following/honesty: [{Rating, Rationale}]. truthfulness/helpfulness: [{Type, Rationale, Rating, Rationale For Rating}]

Usage Examples

Running Preference Annotation

from preference_templates import (
    system_prompt, instruction_following_template,
    honesty_template, truthfulness_template,
    harmlessness_template, helpfulness_template
)

TEMPLATE = {
    "instruction_following": instruction_following_template,
    "honesty": honesty_template,
    "truthfulness": truthfulness_template,
    "harmlessness": harmlessness_template,
    "helpfulness": helpfulness_template,
}

subset = "truthful_qa"
example = {
    "instruction": "What is the capital of France?",
    "correct_answers": ["Paris"],
    "incorrect_answers": ["London", "Berlin"],
    "completions": [
        {"response": "The capital of France is Paris.", ...},
        {"response": "France's capital is London.", ...},
        {"response": "I believe it's Paris.", ...},
        {"response": "The capital is Berlin.", ...},
    ]
}

result = annotate(example)
# result["completions"][0]["annotations"]["honesty"][0]["Rating"] -> "5"
# result["completions"][1]["annotations"]["truthfulness"][0]["Type"] -> ["1"]

Related Pages

Implements Principle

Principle:OpenBMB_UltraFeedback_Fine_Grained_Preference_Annotation

Requires Environment

Environment:OpenBMB_UltraFeedback_OpenAI_API_Environment

Uses Heuristic

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment