Implementation:Arize ai Phoenix Legacy Generate

LLM_Evaluation Synthetic_Data

Overview

The Legacy Generate module provides the llm_generate() function, which applies an LLM to each row of a pandas DataFrame via a prompt template to produce synthetic text outputs. This is a companion to llm_classify(): whereas classification constrains outputs to predefined rails, llm_generate() produces free-form text responses.

The function supports both synchronous and asynchronous execution, configurable concurrency, optional system instructions, and a custom output parser that can transform each raw LLM response into structured columns. Typical use cases include generating synthetic irrelevant responses for evaluation benchmarks, producing paraphrased questions, and creating augmented training data.

Internally, the function renders prompts by mapping DataFrame columns to template variables, dispatches them through an executor (async when possible), and collects results into a new DataFrame indexed to match the input.

Code Reference

Attribute	Details
Source File	packages/phoenix-evals/src/phoenix/evals/legacy/generate.py
Repository	Arize-ai/phoenix
Lines	143
Module	phoenix.evals.legacy.generate
Key Symbols	llm_generate()
Dependencies	pandas, phoenix.evals.legacy.executors, phoenix.evals.legacy.models, phoenix.evals.legacy.templates, phoenix.evals.legacy.utils

I/O Contract

llm_generate()

Parameter	Type	Description
dataframe	pd.DataFrame	Input DataFrame where column names correspond to template variable names.
template	Union[PromptTemplate, str]	Prompt template with curly-brace variable placeholders.
model	BaseModel	LLM model instance used to generate responses.
system_instruction	Optional[str]	Optional system message prepended to each prompt.
verbose	bool	If True, prints model invocation details and retry information.
output_parser	Optional[Callable[[str, int], Dict[str, Any]]]	Custom function that transforms each raw response and its row index into a dictionary of output columns. Defaults to {"output": response}.
include_prompt	bool	If True, includes a prompt column with the rendered prompt text.
include_response	bool	If True, includes a response column with the raw LLM output (before parsing).
run_sync	bool	If True, forces synchronous execution.
concurrency	Optional[int]	Number of concurrent async requests. Defaults to the model's default concurrency.
Returns	pd.DataFrame	DataFrame indexed to match the input, with columns from the output parser (default: output), and optionally prompt and response.

Error Handling

When generation fails for a row, the fallback return value is:

Column	Fallback Value
output	"generation-failed"
prompt (if included)	""
response (if included)	""

Usage Examples

from phoenix.evals.legacy.generate import llm_generate
from phoenix.evals.legacy.models import OpenAIModel
from phoenix.evals.legacy.templates import PromptTemplate
import pandas as pd

model = OpenAIModel(model="gpt-4")

# Generate synthetic irrelevant responses
df = pd.DataFrame({
    "input": [
        "What is the capital of France?",
        "How does photosynthesis work?",
    ],
})

template = PromptTemplate(
    template="Generate a plausible but incorrect answer to the "
    "following question:\n{input}"
)

result = llm_generate(
    dataframe=df,
    template=template,
    model=model,
    include_prompt=True,
)
# result has columns: output, prompt

import json

# With a custom output parser
def parse_json_response(response: str, index: int) -> dict:
    parsed = json.loads(response)
    return {
        "question": parsed.get("question", ""),
        "answer": parsed.get("answer", ""),
    }

result = llm_generate(
    dataframe=df,
    template="Generate a JSON with 'question' and 'answer' keys "
    "based on: {input}",
    model=model,
    output_parser=parse_json_response,
)
# result has columns: question, answer

Related Pages

• Arize_ai_Phoenix_Legacy_Classify - Classification counterpart that constrains outputs to predefined rails
• Arize_ai_Phoenix_Legacy_Templates - PromptTemplate and template mapping utilities
• Arize_ai_Phoenix_Legacy_Utils - Progress bar formatting used during generation