Implementation:SqueezeAILab ETS Extract Answer

Knowledge Sources	ETS
Domains	Evaluation, Text_Processing, Mathematical_Reasoning
Last Updated	2026-02-14 02:00 GMT

Overview

Concrete tool for extracting mathematical answers from model-generated text using cascading pattern matching, provided by math_evaluate.py and answer_extraction.py.

Description

Two primary extraction functions serve different model types:

• extract_shepherd_answer: Uses regex r"The answer is:(.+) ки" for llemma/mistral Shepherd-format models
• extract_answer: General-purpose cascading extractor supporting boxed, natural language, program output, and numeric fallback patterns

Both return "[invalid]" when no answer can be extracted.

Usage

The extraction function is selected in main() based on --model_type. For llemma/mistral models, extract_shepherd_answer is used. The extracted answer is passed to grade_answer() for correctness evaluation.

Code Reference

Source Location

• Repository: ETS
• File: math_evaluate.py (extract_shepherd_answer at L30-39), evaluate/data_processing/answer_extraction.py (extract_answer at L207-243)

Signature

def extract_shepherd_answer(completion):
    """
    Extract answer from Shepherd/llemma format text.
    Looks for pattern: "The answer is:(.+) ки"

    Args:
        completion (str or None): Full model-generated text

    Returns:
        str: Extracted answer string, or "[invalid]" if no match
    """

def extract_answer(pred_str, exhaust=False):
    """
    General-purpose cascading answer extraction.

    Priority order:
    1. "final answer is $...$. I hope" pattern
    2. "boxed{...}" LaTeX notation
    3. "he answer is" natural language
    4. Code output blocks (```output...```)
    5. Last number fallback

    Args:
        pred_str (str): Full model-generated text
        exhaust (bool): If True, return all found answers as list;
                        if False (default), return last match as string

    Returns:
        str or list[str]: Extracted answer(s), or "" if none found
    """

Import

# extract_shepherd_answer is defined in math_evaluate.py
# extract_answer is imported from the evaluation package:
from evaluate.data_processing.answer_extraction import extract_answer

I/O Contract

Inputs

Name	Type	Required	Description
completion / pred_str	str	Yes	Full model-generated trajectory text
exhaust	bool	No	If True, return all matches as list (default: False)

Outputs

Name	Type	Description
answer	str or list[str]	Extracted answer string, "[invalid]" if no match, or list if exhaust=True

Usage Examples

Shepherd Format (llemma/mistral)

text = "Step 1: We solve for x. The answer is: 42 ки"
answer = extract_shepherd_answer(text)
print(answer)  # "42"

# No match
text = "The solution is 42."
answer = extract_shepherd_answer(text)
print(answer)  # "[invalid]"

General Extraction

from evaluate.data_processing.answer_extraction import extract_answer

# Boxed format
text = r"Therefore $\boxed{\frac{1}{2}}$"
answer = extract_answer(text)
print(answer)  # "\\frac{1}{2}"

# Natural language
text = "After calculation, the answer is 7."
answer = extract_answer(text)
print(answer)  # "7"

# Last number fallback
text = "We get 3 + 4 = 7"
answer = extract_answer(text)
print(answer)  # "7"

# Exhaust mode (all matches)
text = r"$\boxed{1}$ or $\boxed{2}$"
answers = extract_answer(text, exhaust=True)
print(answers)  # ["1", "2"]

Related Pages

Implements Principle

• Principle:SqueezeAILab_ETS_Answer_Extraction

Requires Environment

• Environment:SqueezeAILab_ETS_Evaluation_Python_Stack