Implementation:FMInference FlexLLMGen OptTokenizer

Knowledge Sources	FlexLLMGen HELM Benchmark
Domains	Benchmark_Integration, NLP
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for wrapping HuggingFace tokenizers into HELM-compatible tokenizer services provided by the FlexLLMGen HELM integration.

Description

OptTokenizer wraps AutoTokenizer and provides tokenize(TokenizationRequest) -> TokenizationRequestResult and decode(DecodeRequest) -> DecodeRequestResult methods. It handles encoding with optional truncation, tokenization without encoding, and decoding with configurable tokenization cleanup. Special handling for Galactica models sets pad/eos tokens from config.

Usage

Instantiate with a HuggingFace model name before running HELM scenarios. Pass as tokenizer_service to HELM's AdapterFactory and metric.evaluate().

Code Reference

• Source: flexllmgen/apps/helm_run.py, Lines: 34-88
• Signature:

class OptTokenizer:
    def __init__(self, name):
        """
        Args:
            name: HuggingFace model name (e.g., "facebook/opt-30b")
        """
        self.tokenizer = AutoTokenizer.from_pretrained(name, padding_side="left")
        self.tokenizer.add_bos_token = False

    def tokenize(self, request: TokenizationRequest) -> TokenizationRequestResult:
        ...

    def decode(self, request: DecodeRequest) -> DecodeRequestResult:
        ...

• Import:

from flexllmgen.apps.helm_run import OptTokenizer

I/O Contract

Inputs

Parameter	Type	Required	Description
name	str	Yes	HuggingFace model name

For tokenize():

Parameter	Type	Required	Description
request	TokenizationRequest	Yes	HELM tokenization request with text, encode flag, truncation, max_length

For decode():

Parameter	Type	Required	Description
request	DecodeRequest	Yes	HELM decode request with token ids

Outputs

• OptTokenizer instance
• tokenize() returns TokenizationRequestResult
• decode() returns DecodeRequestResult

Usage Examples

from flexllmgen.apps.helm_run import OptTokenizer
from helm.common.tokenization_request import TokenizationRequest

tokenizer = OptTokenizer("facebook/opt-30b")

# Tokenize text for HELM
request = TokenizationRequest(text="Hello world", encode=True)
result = tokenizer.tokenize(request)
# result.tokens contains TokenizationToken values

# Decode tokens back to text
from helm.common.tokenization_request import DecodeRequest
decode_req = DecodeRequest(tokens=[2, 31414, 232])
decode_result = tokenizer.decode(decode_req)
# decode_result.text contains decoded string

Related Pages

• Principle:FMInference_FlexLLMGen_HELM_Tokenizer_Adaptation