Implementation:EvolvingLMMs Lab Lmms eval Transformation Filters

• • File**: `/tmp/kapso_repo_sslb_59s/lmms_eval/filters/transformation.py`

• • Principle**: Post_Processing_and_Metrics

1. 1. Overview

Transformation Filters provide simple text transformation operations that can be applied to model responses before metric computation. These filters normalize responses to ensure consistent comparison with targets by applying case transformations and dictionary-based mappings.

1. 1. Key Components

1. 1. 1. 1. LowercaseFilter

```python class LowercaseFilter(Filter):

   def __init__(self) -> None:
       pass

   def apply(self, resps, docs):
       def filter_set(inst):
           return [resp.lower() for resp in inst]

       return [filter_set(resp) for resp in resps]

```

• • Purpose**: Convert all responses to lowercase

• • Parameters**:

- `resps`: Nested list of response strings [[resp1, resp2], [resp3], ...] - `docs`: Document list (unused but required by Filter interface)

• • Returns**: Nested list with all responses lowercased

• • Usage Example**:

```python filter = LowercaseFilter() responses = [["Hello", "WORLD"], ["Test"]] filtered = filter.apply(responses, docs)

1. Result: [["hello", "world"], ["test"]]

```

• • Use Cases**:

- Case-insensitive answer matching - Normalizing multiple choice answers (A vs a) - Comparing free-form text responses

1. 1. 1. 2. UppercaseFilter

```python class UppercaseFilter(Filter):

   def __init__(self) -> None:
       pass

   def apply(self, resps, docs):
       def filter_set(inst):
           return [resp.upper() for resp in inst]

       return [filter_set(resp) for resp in resps]

```

• • Purpose**: Convert all responses to uppercase

• • Parameters**:

- `resps`: Nested list of response strings - `docs`: Document list (unused)

• • Returns**: Nested list with all responses uppercased

• • Usage Example**:

```python filter = UppercaseFilter() responses = [["hello", "world"], ["Test"]] filtered = filter.apply(responses, docs)

1. Result: [["HELLO", "WORLD"], ["TEST"]]

```

• • Use Cases**:

- Standardizing to uppercase format - Matching uppercase answer keys - Ensuring consistent letter case for multiple choice

1. 1. 1. 3. MapFilter

```python class MapFilter(Filter):

   def __init__(self, mapping_dict: dict = {}, default_value=None) -> None:
       """
       Initializes the MapFilter with a given mapping dictionary and default value.

       Args:
       - mapping_dict (dict): A dictionary containing the key-value mappings.
                              Default is an empty dictionary.
       - default_value (Any): The value to be returned when a key is not found in the mapping_dict.
                              Default is None.

       Example:
       mapper = MapFilter({'A': 1, 'B': 2}, default_value=0)
       """
       assert isinstance(mapping_dict, dict), "Provided mapping_dict is not a dictionary"
       self.mapping_dict = mapping_dict
       self.default_value = default_value

```

• • Purpose**: Map response strings to other values using a dictionary

• • Initialization Parameters**:

- `mapping_dict` (dict): Key-value mappings for transformation - `default_value` (Any): Value to return when key not in mapping (default: None)

• • Validation**: Asserts mapping_dict is a dictionary

• • apply Method**:

```python def apply(self, resps, docs):

   def filter_set(inst):
       return [self.mapping_dict.get(resp, self.default_value) for resp in inst]

   return [filter_set(resp) for resp in resps]

```

• • Key Features**:

- Uses dict.get() for safe lookup - Returns default_value for missing keys - Preserves structure of nested response lists

• • Usage Examples**:

1. 1. 1. 1. Multiple Choice to Index Mapping

```python mapper = MapFilter({'A': 0, 'B': 1, 'C': 2, 'D': 3}, default_value=-1) responses = [["A", "C"], ["B"], ["E"]] filtered = mapper.apply(responses, docs)

1. Result: [[0, 2], [1], [-1]]

```

1. 1. 1. 1. Answer Normalization

```python mapper = MapFilter({

   'yes': 1,
   'no': 0,
   'YES': 1,
   'NO': 0,
   'y': 1,
   'n': 0,

}, default_value=None) responses = [["yes"], ["NO"], ["maybe"]] filtered = mapper.apply(responses, docs)

1. Result: [[1], [0], [None]]

```

1. 1. 1. 1. Label Standardization

```python mapper = MapFilter({

   'positive': 'pos',
   'negative': 'neg',
   'neutral': 'neu',

}, default_value='unknown') responses = [["positive"], ["negative"], ["other"]] filtered = mapper.apply(responses, docs)

1. Result: [["pos"], ["neg"], ["unknown"]]

```

1. 1. Inheritance Structure

All filters inherit from base `Filter` class: ```python from lmms_eval.api.filter import Filter

class LowercaseFilter(Filter):

   ...

```

• • Required Methods**:

- `apply(self, resps, docs)`: Transform responses

• • Interface Contract**:

- Accept nested list of responses - Accept documents list (may be unused) - Return transformed responses in same structure

1. 1. Common Patterns

1. 1. 1. Chaining Filters

```python

1. Lowercase then map

responses = [["A", "b"], ["C"]]

1. Step 1: Lowercase

lower_filter = LowercaseFilter() lowered = lower_filter.apply(responses, docs)

1. Result: [["a", "b"], ["c"]]

1. Step 2: Map to indices

map_filter = MapFilter({'a': 0, 'b': 1, 'c': 2, 'd': 3}) mapped = map_filter.apply(lowered, docs)

1. Result: [[0, 1], [2]]

```

1. 1. 1. Task Configuration

```yaml

1. In task YAML

filters:

 - name: lowercase
 - name: map
   mapping_dict:
     "yes": 1
     "no": 0
   default_value: -1

```

1. 1. 1. Programmatic Usage

```python from lmms_eval.filters.transformation import LowercaseFilter, MapFilter

1. Define filter pipeline

filters = [

   LowercaseFilter(),
   MapFilter({'a': 0, 'b': 1, 'c': 2, 'd': 3}, default_value=-1)

]

1. Apply filters

filtered_resps = resps for filter_obj in filters:

   filtered_resps = filter_obj.apply(filtered_resps, docs)

```

1. 1. Response Structure

1. 1. 1. Input Structure

```python resps = [

   ["response1", "response2"],  # Multiple responses for doc 1
   ["response3"],               # Single response for doc 2
   ["response4", "response5", "response6"],  # Multiple for doc 3

] ```

1. 1. 1. Output Structure

Same nested list structure, with transformations applied: ```python filtered_resps = [

   [transformed1, transformed2],
   [transformed3],
   [transformed4, transformed5, transformed6],

] ```

1. 1. Design Decisions

1. **No-Op Initialization**: Lowercase and Uppercase filters have empty `__init__` but defined for consistency

2. **Nested List Processing**: Inner `filter_set` function processes each document's response set

3. **Unused docs Parameter**: Kept for interface consistency even though transformation filters don't need documents

4. **MapFilter Default**: None as default allows detecting unmapped values

5. **Assertion in MapFilter**: Validates mapping_dict type at initialization for early error detection

6. **Immutable Operations**: Filters create new lists rather than modifying in-place

1. 1. Integration with Framework

1. 1. 1. Filter Registry

Filters are registered and loaded by name: ```python from lmms_eval.filters.transformation import LowercaseFilter, MapFilter

FILTER_REGISTRY = {

   "lowercase": LowercaseFilter,
   "uppercase": UppercaseFilter,
   "map": MapFilter,

} ```

1. 1. 1. Task Application

Tasks specify filters in configuration: ```python class MyTask(Task):

   def __init__(self):
       self._config = TaskConfig(
           filters=[
               {"name": "lowercase"},
               {"name": "map", "mapping_dict": {"a": 0, "b": 1}}
           ]
       )

```

1. 1. 1. Evaluation Pipeline

```python

1. 1. Model generates responses

raw_responses = model.generate(requests)

1. 2. Apply filters

filtered_responses = raw_responses for filter_config in task.config.filters:

   filter_obj = create_filter(filter_config)
   filtered_responses = filter_obj.apply(filtered_responses, docs)

1. 3. Compute metrics with filtered responses

metrics = compute_metrics(filtered_responses, targets) ```

1. 1. Use Cases by Task Type

1. 1. 1. Multiple Choice Tasks

```python

1. Normalize letter answers

LowercaseFilter()

1. Convert to indices for computation

MapFilter({'a': 0, 'b': 1, 'c': 2, 'd': 3}) ```

1. 1. 1. Yes/No Classification

```python

1. Standardize format

LowercaseFilter()

1. Map to binary

MapFilter({'yes': 1, 'no': 0}, default_value=-1) ```

1. 1. 1. Text Normalization

```python

1. Simple case normalization

LowercaseFilter()

1. or

UppercaseFilter() ```

1. 1. 1. Label Mapping

```python

1. Map verbose labels to codes

MapFilter({

   'very positive': 2,
   'positive': 1,
   'neutral': 0,
   'negative': -1,
   'very negative': -2,

}, default_value=0) ```

1. 1. Error Handling

1. 1. 1. MapFilter Validation

```python

1. Invalid mapping_dict type

mapper = MapFilter("not a dict") # Raises AssertionError ```

1. 1. 1. Missing Keys

```python mapper = MapFilter({'a': 1, 'b': 2}, default_value=-1) responses = "a", "c" filtered = mapper.apply(responses, docs)

1. Result: 1, -1 # 'c' not in mapping, uses default

```

1. 1. 1. Empty Responses

```python filter = LowercaseFilter() responses = [[]] # Empty response set filtered = filter.apply(responses, docs)

1. Result: [[]] # Handles empty lists gracefully

```

1. 1. Related Components

- Post_Processing_and_Metrics: Principle these filters support - `lmms_eval.api.filter`: Base Filter class - `lmms_eval.filters.extraction`: More complex extraction filters - Task_Directory_Structure: Tasks configure these filters

1. 1. Best Practices

1. Apply case normalization before mapping 2. Use appropriate default values for MapFilter 3. Document filter pipeline in task configs 4. Test filter combinations with edge cases 5. Consider order of filter application 6. Use descriptive mapping dictionary keys 7. Handle missing keys gracefully with defaults 8. Keep transformations simple and composable

1. 1. Testing Examples

```python

1. Test LowercaseFilter

def test_lowercase_filter():

   filter = LowercaseFilter()
   resps = [["HELLO", "World"], ["TEST"]]
   result = filter.apply(resps, [])
   assert result == [["hello", "world"], ["test"]]

1. Test UppercaseFilter

def test_uppercase_filter():

   filter = UppercaseFilter()
   resps = [["hello", "World"], ["test"]]
   result = filter.apply(resps, [])
   assert result == [["HELLO", "WORLD"], ["TEST"]]

1. Test MapFilter

def test_map_filter():

   mapper = MapFilter({'a': 0, 'b': 1}, default_value=-1)
   resps = [["a", "b"], ["c"]]
   result = mapper.apply(resps, [])
   assert result == [[0, 1], [-1]]

1. Test MapFilter with None default

def test_map_filter_none_default():

   mapper = MapFilter({'yes': 1, 'no': 0})
   resps = "yes", "maybe"
   result = mapper.apply(resps, [])
   assert result == 1, None

```