Principle:Deepseek ai Janus Response Decoding
| Knowledge Sources | |
|---|---|
| Domains | NLP, Tokenization |
| Last Updated | 2026-02-10 09:30 GMT |
Overview
A procedure for converting generated token IDs back into human-readable text by reversing the tokenization process.
Description
Response decoding is the final step in the text generation pipeline. After the language model produces a sequence of token IDs, these must be mapped back to text strings. This involves looking up each token ID in the vocabulary, concatenating the resulting subword pieces, and optionally stripping special tokens (like BOS, EOS, padding) from the output.
Usage
Use this principle as the final step in the multimodal understanding pipeline, after autoregressive text generation has produced output token IDs.
Theoretical Basis
Token decoding reverses the tokenization:
- Token ID → subword: Each integer ID maps to a vocabulary entry (subword, word, or special token)
- Subword merging: Adjacent subwords are concatenated following the tokenizer's merge rules
- Special token removal: When skip_special_tokens=True, BOS/EOS/PAD tokens are excluded from the output
The Janus tokenizer (LlamaTokenizerFast) uses a SentencePiece-based vocabulary with special DeepSeek tokens.