Implementation:Ggml org Llama cpp Jinja Parser
| Knowledge Sources | |
|---|---|
| Domains | Template_Engine, Parsing |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Implements the Jinja template parser that consumes a token stream and produces an AST (abstract syntax tree).
Description
This file implements the second stage of the Jinja template processing pipeline within llama.cpp. The parser class implements a recursive descent parser consuming tokens from the lexer. It handles Jinja statements (if/elif/else, for, set, macro, filter, call, block), expressions (binary operations, unary operations, member access, function calls, slicing, ternary, comprehensions), and literals (strings, numbers, arrays, tuples, objects). It uses peek() and expect() for token lookahead and consumption, and preserves source positions in all AST nodes via mk_stmt() for error reporting.
Usage
Used as the second stage when processing Jinja chat templates. The parser consumes the token stream from the lexer and produces a structured AST that the runtime can execute.
Code Reference
Source Location
- Repository: Ggml_org_Llama_cpp
- File: common/jinja/parser.cpp
- Lines: 1-591
Signature
namespace jinja {
class parser {
const std::vector<token> & tokens;
size_t current = 0;
std::string source;
public:
parser(const std::vector<token> & t, const std::string & src);
program parse();
template<typename T, typename... Args>
std::unique_ptr<T> mk_stmt(size_t start_pos, Args&&... args);
private:
const token & peek(size_t offset = 0) const;
token expect(token::type type, const std::string & error);
void expect_identifier(const std::string & name);
bool is(token::type type) const;
bool is_identifier(const std::string & name) const;
};
} // namespace jinja
Import
#include "parser.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| tokens | const vector<token> & | Yes | Token stream produced by the lexer |
| source | const std::string & | Yes | Original source text for error reporting with position information |
Outputs
| Name | Type | Description |
|---|---|---|
| program | struct | AST root node containing a vector of statement nodes representing the parsed template |
Usage Examples
#include "lexer.h"
#include "parser.h"
// Tokenize and parse a Jinja template
std::string source = "{% for msg in messages %}{{ msg.content }}{% endfor %}";
auto lex_result = jinja::lexer::tokenize(source);
jinja::parser p(lex_result.tokens, lex_result.source);
auto ast = p.parse();
// ast now contains:
// program {
// for_statement {
// variable: "msg"
// iterable: identifier("messages")
// body: [
// expression_statement {
// member_expression { object: "msg", property: "content" }
// }
// ]
// }
// }