Implementation:Ggml org Llama cpp Jinja Lexer
| Knowledge Sources | |
|---|---|
| Domains | Template_Engine, Parsing |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Implements the Jinja template lexer that converts source text into a token stream.
Description
This file is the first stage of the Jinja template processing pipeline within llama.cpp. The lexer::tokenize() method processes source text character by character, normalizing line endings and handling Jinja delimiters ({{, }}, {%, %}, {#, #}). It tracks state to distinguish between text content and Jinja expressions/statements. Within Jinja blocks, it recognizes numeric literals, string literals (with escape sequences), identifiers, operators (comparison, arithmetic, unary), and punctuation. It preserves original character positions for error reporting and handles whitespace stripping markers (-) on block delimiters.
Usage
Used as the first stage when processing Jinja chat templates. The lexer converts template source text into tokens that are then consumed by the parser to build an AST.
Code Reference
Source Location
- Repository: Ggml_org_Llama_cpp
- File: common/jinja/lexer.cpp
- Lines: 1-341
Signature
namespace jinja {
// String trimming helpers
static void string_lstrip(std::string & s, const char * chars);
static void string_rstrip(std::string & s, const char * chars);
// Main tokenization entry point
lexer_result lexer::tokenize(const std::string & source);
// Internal: consume characters while predicate holds (handles escape chars)
// auto consume_while = [&](const pred & predicate) -> std::string;
} // namespace jinja
Import
#include "lexer.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| source | const std::string & | Yes | Jinja template source text to tokenize |
Outputs
| Name | Type | Description |
|---|---|---|
| lexer_result | struct | Contains the vector of tokens and the normalized source string |
| tokens | vector<token> | Sequence of tokens with type, value, and source position for each |
Usage Examples
#include "lexer.h"
// Tokenize a Jinja template
std::string source = "Hello {{ name }}! {% if greeting %}Welcome{% endif %}";
auto result = jinja::lexer::tokenize(source);
// result.tokens contains:
// - TEXT token: "Hello "
// - EXPRESSION_START: "{{"
// - IDENTIFIER: "name"
// - EXPRESSION_END: "}}"
// - TEXT token: "! "
// - STATEMENT_START: "{%"
// - IDENTIFIER: "if"
// - IDENTIFIER: "greeting"
// - STATEMENT_END: "%}"
// - TEXT token: "Welcome"
// - STATEMENT_START: "{%"
// - IDENTIFIER: "endif"
// - STATEMENT_END: "%}"