Implementation:FMInference FlexLLMGen Prompt Utils

Overview

Concrete tool for constructing few-shot prompts with multiple selection strategies provided by the FlexLLMGen data wrangling application.

Description

prompt_utils.py provides three prompt construction functions: get_manual_prompt() returns handcrafted prompts from constants.PREFIXES keyed by data directory and optional subkey attribute; get_random_prompt() randomly samples k examples from training data and concatenates their text+label; get_validation_prompt() uses SentenceTransformer embeddings and cosine similarity to select the most informative examples from validation errors. Additionally, setup_st_pipeline() initializes the sentence transformer model, and get_hard_samples() retrieves the hardest examples for a given query.

Usage

Call the appropriate prompt function based on the --sample_method CLI argument: "manual", "random", or "validation_clusters".

Code Reference

• Source: flexllmgen/apps/data_wrangle/utils/prompt_utils.py, Lines: 19-56
• Signatures:

def get_manual_prompt(data_dir: str, example: pd.Series) -> str:
    """Get manual prompt for data name.
    Args:
        data_dir: Dataset directory path (key into constants.PREFIXES)
        example: DataFrame row (used for subkey lookup if needed)
    Returns:
        Handcrafted prompt string with labeled examples
    """

def get_random_prompt(train_data: pd.DataFrame, num_examples: int = 10) -> str:
    """Get random examples for prompt from training data.
    Args:
        train_data: Training DataFrame with "text" and "label_str" columns
        num_examples: Number of examples to sample (k)
    Returns:
        Prompt string with randomly sampled examples
    """

def get_validation_prompt(
    validation_path: str, num_examples: int = 10, task: str = "entity_matching"
) -> str:
    """Get prompt from validation errors using embedding similarity.
    Args:
        validation_path: Path to validation results .feather file
        num_examples: Number of examples to select
        task: Task type for filtering
    Returns:
        Prompt string with embedding-selected examples
    """

• Import:

from flexllmgen.apps.data_wrangle.utils.prompt_utils import get_manual_prompt, get_random_prompt, get_validation_prompt

I/O Contract

get_manual_prompt Inputs

Name	Type	Required	Description
data_dir	str	Yes	Dataset directory key
example	pd.Series	Yes	Row for subkey lookup

get_random_prompt Inputs

Name	Type	Required	Description
train_data	pd.DataFrame	Yes	Training data
num_examples	int	No	Number of examples (default 10)

get_validation_prompt Inputs

Name	Type	Required	Description
validation_path	str	Yes	Path to .feather file
num_examples	int	No	Number of examples (default 10)
task	str	No	Task type (default "entity_matching")

Outputs

All functions return str — a few-shot prompt prefix with labeled examples separated by "\n\n".

Usage Examples

from flexllmgen.apps.data_wrangle.utils.prompt_utils import (
    get_manual_prompt, get_random_prompt
)

# Manual prompt (handcrafted)
prompt = get_manual_prompt("fm_data_tasks/data/entity_matching/structured/Amazon-Google", row)

# Random prompt (k=5 random examples)
prompt = get_random_prompt(train_data, num_examples=5)
# Result: "title: X. brand: Y\nYes\n\ntitle: A. brand: B\nNo\n\n..."

# Combine with query
full_prompt = prompt + query_text

Related Pages

• Principle:FMInference_FlexLLMGen_Few_Shot_Prompt_Construction