Implementation:EvolvingLMMs Lab Lmms eval IFEval Instructions Registry

Location: lmms_eval/tasks/ifeval/instructions_registry.py

Principle: Task_Utility_Functions

Purpose

Registry system for IFEval (Instruction Following Evaluation) that maps instruction types to checker classes and defines conflict relationships between instructions.

Constants

Instruction Category Prefixes

• _KEYWORD = "keywords:"
• _LANGUAGE = "language:"
• _LENGTH = "length_constraints:"
• _CONTENT = "detectable_content:"
• _FORMAT = "detectable_format:"
• _MULTITURN = "multi-turn:"
• _COMBINATION = "combination:"
• _STARTEND = "startend:"
• _CHANGE_CASES = "change_case:"
• _PUNCTUATION = "punctuation:"

INSTRUCTION_DICT

Dictionary mapping instruction IDs to checker classes from instructions module. Includes:

Keywords: existence, frequency, forbidden_words, letter_frequency

Language: response_language

Length: number_sentences, number_paragraphs, number_words, nth_paragraph_first_word

Content: number_placeholders, postscript

Format: number_bullet_lists, constrained_response, number_highlighted_sections, multiple_sections, json_format, title

Combination: two_responses, repeat_prompt

Start/End: end_checker, quotation

Case: capital_word_frequency, english_capital, english_lowercase

Punctuation: no_comma

INSTRUCTION_CONFLICTS

Dictionary defining which instruction types conflict with each other. Used to prevent invalid instruction combinations in prompts.

Key Functions

conflict_make

def conflict_make(conflicts)

Ensures bidirectional conflict relationships:

• If A conflicts with B, adds B conflicts with A
• Adds self-conflicts for all instructions
• Returns revised conflict dictionary

Implementation Details

• Licensed under Apache 2.0 from Google Research
• Some instruction types commented out (key_sentences, rephrase_paragraph, rephrase, constrained_start)
• Complex conflict rules for format instructions (json_format, constrained_response, two_responses, repeat_prompt)
• Language instruction conflicts with many others due to language switching requirements