Implementation:Deepseek ai Janus LlamaTokenizerFast Decode

Knowledge Sources	Janus HuggingFace Tokenizers
Domains	NLP, Tokenization
Last Updated	2026-02-10 09:30 GMT

Overview

HuggingFace LlamaTokenizerFast.decode method used to convert generated token IDs back to text in the Janus pipeline.

Description

The decode method converts a list of token IDs back into a human-readable string. In Janus, the tokenizer is accessed via vl_chat_processor.tokenizer and uses the skip_special_tokens=True flag to strip special tokens from the output.

External Reference

• HuggingFace Tokenizer Documentation

Usage

Call after language_model.generate() to convert the output token IDs into a text answer string.

Code Reference

Source Location

• Repository: External — HuggingFace Transformers
• File (tokenizer attribute): janus/models/processing_vlm.py:L100

Signature

LlamaTokenizerFast.decode(
    token_ids: List[int],
    skip_special_tokens: bool = False,
) -> str

Import

# Accessed via processor attribute:
# tokenizer = vl_chat_processor.tokenizer
# answer = tokenizer.decode(output_ids, skip_special_tokens=True)

I/O Contract

Inputs

Name	Type	Required	Description
token_ids	List[int]	Yes	Generated token IDs from language_model.generate()
skip_special_tokens	bool	No	Strip special tokens from output (default False, typically set True)

Outputs

Name	Type	Description
text	str	Decoded text answer

Usage Examples

Basic Decoding

# After generation
outputs = vl_gpt.language_model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True,
)

# Decode the generated token IDs to text
answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(answer)

Related Pages

Implements Principle

• Principle:Deepseek_ai_Janus_Response_Decoding