Implementation:OpenBMB UltraFeedback World Knowledge Template Substitution

Knowledge Sources	UltraFeedback
Domains	NLP, Evaluation
Last Updated	2023-10-02 00:00 GMT

Overview

Concrete pattern for constructing and injecting world knowledge strings into GPT-4 annotation templates based on instruction source.

Description

This is a Pattern Doc documenting a user-defined pattern rather than a library API. The world knowledge injection in annotate_preference.py uses Python string formatting and conditional logic to construct a context string that is substituted into the {world_knowledge} placeholder in truthfulness and helpfulness templates.

The logic is implemented as inline conditional blocks within the annotate function (Lines 85-93) and is injected into the template format dict at Line 110-111.

Usage

This pattern is called once per aspect evaluation within the annotate function. It is only injected for the truthfulness aspect (which has a world_knowledge placeholder in its template). The helpfulness templates also have this placeholder for with-answer variants.

Code Reference

Source Location

• Repository: UltraFeedback
• File: src/data_annotation/annotate_preference.py (Lines 85-93 for knowledge construction, Lines 110-111 for injection)

Signature

# World knowledge construction (annotate_preference.py:L85-93)
if subset == "truthful_qa":
    world_knowledge = "\n".join([
        "a subset of correct answers: " + str(example["correct_answers"]),
        "a subset of incorrect_answers: " + str(example["incorrect_answers"])
    ])
elif subset == "false_qa":
    world_knowledge = "The question is based on a false premise."
elif subset == "flan":
    world_knowledge = example["correct_answers"]
else:
    world_knowledge = "No additional world knowledge for reference."

# Injection into template (annotate_preference.py:L110-111)
if aspect == "truthfulness":
    format_input.update({"world_knowledge": world_knowledge})

Import

# No special imports needed - uses Python built-in string operations

I/O Contract

Inputs

Name	Type	Required	Description
subset	str	Yes	Dataset subset name determining knowledge source
example["correct_answers"]	Union[List[str], str]	No	Ground-truth correct answers (TruthfulQA, FLAN)
example["incorrect_answers"]	List[str]	No	Known incorrect answers (TruthfulQA only)

Outputs

Name	Type	Description
world_knowledge	str	Context string to be injected into template's {world_knowledge} placeholder

Usage Examples

TruthfulQA Example

subset = "truthful_qa"
example = {
    "instruction": "What is the capital of Australia?",
    "correct_answers": ["Canberra"],
    "incorrect_answers": ["Sydney", "Melbourne"],
    "completions": [...]
}

# Resulting world_knowledge string:
# "a subset of correct answers: ['Canberra']
#  a subset of incorrect_answers: ['Sydney', 'Melbourne']"

FalseQA Example

subset = "false_qa"
example = {
    "instruction": "Why do fish live in trees?",
    "completions": [...]
}

# Resulting world_knowledge string:
# "The question is based on a false premise."

Generic Example (No World Knowledge)

subset = "sharegpt"
example = {
    "instruction": "Write a poem about autumn.",
    "completions": [...]
}

# Resulting world_knowledge string:
# "No additional world knowledge for reference."

Related Pages

Implements Principle

• Principle:OpenBMB_UltraFeedback_World_Knowledge_Injection