Implementation:Microsoft Semantic kernel EnglishRoberta Encoder
| Knowledge Sources | |
|---|---|
| Domains | Tokenization, NLP |
| Last Updated | 2026-02-11 00:00 GMT |
Overview
Concrete BPE (Byte Pair Encoding) encoder vocabulary JSON file for the English RoBERTa tokenizer model, used in Semantic Kernel Concepts samples for text tokenization.
Description
This file is a large JSON dictionary (approximately 50,259 lines) that maps token strings to their integer indices for the English RoBERTa tokenizer. It is a standard BPE encoder vocabulary file used in GPT-2/RoBERTa-style tokenization. Each entry maps a text token (such as a character, subword, or word fragment) to a unique integer ID starting from 0.
The vocabulary begins with single ASCII characters ("!" mapped to 0, "\\"" to 1, "#" to 2, etc.) and progresses through subword tokens, common words, and multi-character sequences. The file contains 50,257 vocabulary entries in total, which is the standard GPT-2 vocabulary size.
Note: This is a very large data file (50,259 lines). It is a static resource and does not contain executable code.
Usage
This file is loaded by the Concepts sample project when demonstrating custom tokenization with the English RoBERTa model. It is typically used alongside companion files (vocab.bpe for merge rules and dict.txt for additional vocabulary mapping) to implement a complete BPE tokenizer. Developers use it to understand how Semantic Kernel handles token counting and text segmentation for language model inputs.
Code Reference
Source Location
- Repository: Microsoft_Semantic_kernel
- File: dotnet/samples/Concepts/Resources/EnglishRoberta/encoder.json
- Lines: 1-50259
Signature
{
"!": 0,
"\"": 1,
"#": 2,
"$": 3,
"%": 4,
"&": 5,
"'": 6,
"(": 7,
")": 8,
"*": 9,
"+": 10,
",": 11,
"-": 12,
".": 13,
"/": 14,
"0": 15,
"1": 16,
"2": 17,
"3": 18,
"4": 19
}
Import
// Loaded as an embedded resource or from disk in the Concepts sample:
using var encoderStream = File.OpenRead(
Path.Combine("Resources", "EnglishRoberta", "encoder.json"));
// Used to construct a RoBERTa tokenizer:
var encoder = JsonSerializer.Deserialize<Dictionary<string, int>>(encoderStream);
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| (file content) | JSON object | yes | The entire JSON file is consumed as a dictionary mapping string tokens to integer IDs. |
Outputs
| Name | Type | Description |
|---|---|---|
| Token-to-ID mapping | Dictionary<string, int> | A lookup table with 50,257 entries mapping BPE token strings to their corresponding integer indices used by the RoBERTa model. |
Usage Examples
Loading the Encoder for Tokenization
// Load the encoder vocabulary
var encoderJson = await File.ReadAllTextAsync(
Path.Combine("Resources", "EnglishRoberta", "encoder.json"));
var encoder = JsonSerializer.Deserialize<Dictionary<string, int>>(encoderJson);
// Look up a token
int tokenId = encoder["hello"]; // Returns the integer ID for "hello"
// The encoder contains 50,257 entries (standard GPT-2 vocabulary size)
Console.WriteLine($"Vocabulary size: {encoder.Count}");