Principle:Mlc ai Web llm Schema Definition

Overview

Schema Definition is the technique of specifying output structure constraints that guide language model generation to produce schema-conforming text. In @mlc-ai/web-llm, schemas are expressed as JSON Schema strings, EBNF grammars, or structural tag definitions. The schema is compiled into a grammar that constrains token generation at each autoregressive decoding step, guaranteeing the output conforms to the specification. This eliminates unreliable regex-based post-processing and enables direct JSON.parse() on model output.

Description

Schema definition allows developers to declare the expected shape of LLM output before inference begins. The declaration is provided through the response_format field of a ChatCompletionRequest. Three constraint mechanisms are supported:

• JSON Schema (type: "json_object" with schema field) -- A JSON Schema string describing the required properties, types, and constraints of the output object. The schema is compiled into a context-free grammar via GrammarCompiler.compileJSONSchema().
• EBNF Grammar (type: "grammar" with grammar field) -- A raw EBNF grammar string specifying the formal production rules the output must follow. Compiled via GrammarCompiler.compileGrammar().
• Structural Tags (type: "structural_tag" with structural_tag field) -- A tag-delimited constraint format where specific trigger strings (e.g. <tool_call>) activate grammar-constrained regions while allowing free-form text outside the triggered spans. Compiled via GrammarCompiler.compileStructuralTag().

A fourth mode, type: "json_object" without a schema, forces the output to be valid JSON without constraining its internal structure.

When no response_format is set (or type: "text"), the model generates unconstrained free-form text.

Usage

Use schema definition when:

• You need guaranteed structured output from an LLM -- JSON objects matching a precise schema, data extraction into typed records, form filling, or any case where free-form text is insufficient.
• You want to call JSON.parse() on the model output without try/catch because validity is guaranteed at the token level.
• You are implementing function calling or tool use and need the model to produce valid JSON argument objects.
• You need tag-delimited constrained regions in otherwise free-form text (e.g. MCP-style tool call blocks within a natural language response).

Do not use schema definition when:

• You only need free-form natural language output.
• The desired output structure cannot be expressed as a JSON Schema, EBNF grammar, or structural tag definition.

Theoretical Basis

Grammar-constrained decoding works by applying a token-level bitmask to logits before sampling. At each autoregressive step:

1. The GrammarMatcher computes which tokens are valid continuations given the current parse state of the grammar.
2. Invalid tokens receive -infinity logit values (masked out).
3. The model samples only from valid tokens.
4. The grammar matcher updates its internal state with the accepted token.

This guarantees syntactic correctness without degrading semantic quality within the valid token space. The model's probability distribution is preserved over all tokens that are grammatically valid at each position.

Supported Constraint Types

Type	Field	Compiler Method	Use Case
json_object	schema (optional)	compileJSONSchema() or compileBuiltinJSONGrammar()	Structured JSON output
grammar	grammar (required)	compileGrammar()	Custom formal grammars
structural_tag	structural_tag (required)	compileStructuralTag()	Tag-delimited constrained regions
text	none	N/A	Free-form text (no constraints)

Key Properties

• Syntactic guarantee: The output is guaranteed to parse correctly against the specified schema.
• Semantic preservation: Within the set of valid tokens, the model's original probability distribution is unchanged.
• Caching: When the same schema is reused across requests, the compiled grammar and matcher are cached and reset rather than recompiled, reducing overhead.
• Async initialization: Grammar compilation runs concurrently with prompt prefilling to hide latency.

Usage Examples

Defining a JSON Schema Constraint

import * as webllm from "@mlc-ai/web-llm";

// Define a JSON Schema as a string
const schema = JSON.stringify({
  type: "object",
  properties: {
    name: { type: "string" },
    age: { type: "integer" },
    is_student: { type: "boolean" },
  },
  required: ["name", "age", "is_student"],
});

const request: webllm.ChatCompletionRequest = {
  stream: false,
  messages: [
    {
      role: "user",
      content: "Generate a JSON object for a person named Alice who is 30 and not a student.",
    },
  ],
  max_tokens: 128,
  response_format: {
    type: "json_object",
    schema: schema,
  } as webllm.ResponseFormat,
};

Defining an EBNF Grammar Constraint

const ebnfGrammar = String.raw`
root ::= basic_array | basic_object
basic_any ::= basic_number | basic_string | basic_boolean | basic_null | basic_array | basic_object
basic_integer ::= ("0" | "-"? [1-9] [0-9]*) ".0"?
basic_number ::= ("0" | "-"? [1-9] [0-9]*) ("." [0-9]+)? ([eE] [+-]? [0-9]+)?
basic_string ::= (([\"] basic_string_1 [\"]))
basic_string_1 ::= "" | [^"\\\x00-\x1F] basic_string_1 | "\\" escape basic_string_1
escape ::= ["\\/bfnrt] | "u" [A-Fa-f0-9] [A-Fa-f0-9] [A-Fa-f0-9] [A-Fa-f0-9]
basic_boolean ::= "true" | "false"
basic_null ::= "null"
basic_array ::= "[" ("" | ws basic_any (ws "," ws basic_any)*) ws "]"
basic_object ::= "{" ("" | ws basic_string ws ":" ws basic_any ( ws "," ws basic_string ws ":" ws basic_any)*) ws "}"
ws ::= [ \n\t]*
`;

const request: webllm.ChatCompletionRequest = {
  stream: false,
  messages: [{ role: "user", content: "Introduce yourself in JSON" }],
  max_tokens: 128,
  response_format: {
    type: "grammar",
    grammar: ebnfGrammar,
  } as webllm.ResponseFormat,
};

Defining a Structural Tag Constraint

const tools = [
  {
    name: "get_weather",
    schema: {
      type: "object",
      properties: {
        location: { type: "string" },
        unit: { type: "string", enum: ["celsius", "fahrenheit"] },
      },
      required: ["location"],
    },
  },
];

const responseFormat: webllm.ResponseFormat = {
  type: "structural_tag",
  structural_tag: {
    type: "structural_tag",
    format: {
      type: "triggered_tags",
      triggers: ["<tool_call>"],
      tags: tools.map((tool) => ({
        begin: `<tool_call>\n{"name": "${tool.name}", "arguments": `,
        content: { type: "json_schema", json_schema: tool.schema },
        end: "}\n</tool_call>",
      })),
      at_least_one: true,
      stop_after_first: false,
    },
  },
};

Related Pages

• Implementation: Response Format -- Implementation:Mlc_ai_Web_llm_Response_Format
• Principle: Grammar-Constrained Decoding -- The decoding algorithm that enforces schema constraints at token level
• Principle: Structured Output Parsing -- Pattern for parsing the guaranteed-valid output