Implementation:Ollama Ollama Llama Grammar
| Knowledge Sources | |
|---|---|
| Domains | Grammar, Constrained Generation |
| Last Updated | 2025-02-15 00:00 GMT |
Overview
Implements the grammar-constrained decoding engine, which parses GBNF grammar rules and enforces them during token generation to ensure output conforms to a specified grammar.
Description
Includes UTF-8 decoding helpers, GBNF grammar parsing (converting text grammar rules into internal llama_grammar_element structures), grammar state management using stacks of rule element pointers, and token candidate filtering. The llama_grammar_apply_impl function filters token candidates by checking which tokens can advance the grammar state, setting rejected tokens' logits to negative infinity. llama_grammar_accept_impl advances the grammar state when a token is accepted. Uses lazy grammar evaluation with trigger tokens/patterns to avoid unnecessary constraint checking. A repetition threshold (MAX_REPETITION_THRESHOLD = 2000) limits grammar complexity.
Usage
Use this for structured output generation where output must conform to a formal grammar (JSON, code, etc.). This is central to Ollama's function calling and structured output features.
Code Reference
Source Location
- Repository: Ollama
- File: llama/llama.cpp/src/llama-grammar.cpp
- Lines: 1-1471
Signature
static std::pair<uint32_t, const char *> decode_utf8(const char * src);
static std::pair<std::vector<uint32_t>, llama_partial_utf8> decode_utf8(
const std::string & src, llama_partial_utf8 partial_start);
struct llama_grammar * llama_grammar_init_impl(
const struct llama_vocab * vocab,
const struct ollama_vocab * ollama_vocab,
const char * grammar_str,
const char * grammar_root,
bool lazy,
const char ** trigger_patterns,
size_t num_trigger_patterns,
const llama_token * trigger_tokens,
size_t num_trigger_tokens);
void llama_grammar_apply_impl(
const struct llama_grammar & grammar,
llama_token_data_array * cur_p);
void llama_grammar_accept_impl(
struct llama_grammar & grammar,
llama_token token);
Import
#include "llama-grammar.h"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| grammar_str | const char * | Yes | GBNF grammar string defining the constraints |
| grammar_root | const char * | Yes | Name of the root rule in the grammar |
| cur_p | llama_token_data_array * | Yes | Token candidates to filter (for apply) |
| token | llama_token | Yes | Accepted token to advance grammar state |
Outputs
| Name | Type | Description |
|---|---|---|
| grammar | llama_grammar * | Initialized grammar state |
| cur_p | llama_token_data_array * | Filtered token candidates (invalid tokens set to -inf logits) |
Usage Examples
#include "llama-grammar.h"
// Initialize grammar for JSON output
const char * json_grammar = R"(
root ::= "{" ws members ws "}"
members ::= pair ("," ws pair)*
pair ::= string ":" ws value
value ::= string | number | "true" | "false" | "null"
)";
auto * grammar = llama_grammar_init_impl(vocab, nullptr,
json_grammar, "root", false, nullptr, 0, nullptr, 0);
// Apply grammar constraints to token candidates
llama_grammar_apply_impl(*grammar, &candidates);
// Accept a token
llama_grammar_accept_impl(*grammar, sampled_token);
// Clean up
llama_grammar_free_impl(grammar);