Principle:Ollama Ollama Structured Output
| Knowledge Sources | |
|---|---|
| Domains | Structured Output, Grammar |
| Last Updated | 2025-02-15 00:00 GMT |
Overview
Structured Output constrains LLM generation to produce valid output conforming to a specified schema, typically JSON, by converting the schema into a formal grammar and using grammar-guided decoding to mask invalid tokens at each generation step.
Core Concepts
JSON Schema to Grammar Conversion
The structured output system converts a user-provided JSON Schema into a BNF (Backus-Naur Form) grammar that describes all valid JSON documents matching the schema. This conversion handles JSON Schema features including object properties, required fields, array types, enum values, nested schemas, and type constraints. The resulting grammar is a set of production rules that the decoder can enforce character-by-character.
Grammar-Guided Decoding
During inference, the grammar sampler maintains a parse state that tracks the current position within the grammar's production rules. At each token generation step, the sampler evaluates which tokens would produce valid continuations according to the grammar state. Tokens that would produce invalid output have their logits set to negative infinity, effectively preventing the model from selecting them. After a token is selected, the grammar state advances to reflect the new partial output.
Token-Level vs. Character-Level Matching
A key challenge is that grammar rules operate at the character level while LLM generation operates at the token level. A single token may encode multiple characters, and the grammar must validate the entire character sequence that a token would produce. The grammar sampler decodes each candidate token to its character representation and checks whether those characters form a valid continuation of the current parse state.
Performance Optimization
Evaluating grammar validity for every token in the vocabulary at every step is computationally expensive. The implementation uses an optimization where it first checks if the highest-probability token (the model's preferred next token) is grammar-valid. If so, that token is accepted immediately without evaluating the full vocabulary. Only when the top token is rejected does the system fall back to evaluating and masking the entire vocabulary.
Implementation Notes
The grammar infrastructure leverages llama.cpp's grammar implementation, accessed through the llama package. The GrammarSampler in sample/samplers.go wraps this with Go-level integration. JSON Schema to BNF conversion occurs when the user provides a format parameter in the API request. The grammar is initialized with the model's complete vocabulary and token-to-string mapping so it can perform token-level validation.