Implementation:EvolvingLMMs Lab Lmms eval IFEval Instructions Registry

Location: /tmp/kapso_repo_sslb_59s/lmms_eval/tasks/ifeval/instructions_registry.py

Purpose

Registry system for IFEval (Instruction Following Evaluation) that maps instruction types to checker classes and defines conflict relationships between instructions.

Constants

INSTRUCTION_DICT

Dictionary mapping instruction IDs to checker classes from instructions module. Includes:

Keywords: existence, frequency, forbidden_words, letter_frequency

Language: response_language

Length: number_sentences, number_paragraphs, number_words, nth_paragraph_first_word

Content: number_placeholders, postscript

Format: number_bullet_lists, constrained_response, number_highlighted_sections, multiple_sections, json_format, title

Combination: two_responses, repeat_prompt

Start/End: end_checker, quotation

Case: capital_word_frequency, english_capital, english_lowercase

Punctuation: no_comma

INSTRUCTION_CONFLICTS

Dictionary defining which instruction types conflict with each other. Used to prevent invalid instruction combinations in prompts.

Key Functions

conflict_make

def conflict_make(conflicts)

Ensures bidirectional conflict relationships:

If A conflicts with B, adds B conflicts with A
Adds self-conflicts for all instructions
Returns revised conflict dictionary

Implementation Details

Licensed under Apache 2.0 from Google Research
Some instruction types commented out (key_sentences, rephrase_paragraph, rephrase, constrained_start)
Complex conflict rules for format instructions (json_format, constrained_response, two_responses, repeat_prompt)
Language instruction conflicts with many others due to language switching requirements

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment