Implementation:Mlc ai Web llm Response Format

Overview

ResponseFormat is a TypeScript interface provided by @mlc-ai/web-llm that defines output format constraints for chat completion requests. It is assigned to the response_format field of ChatCompletionRequest and supports four modes: "text" (no constraints), "json_object" (valid JSON with optional JSON Schema), "grammar" (EBNF grammar), and "structural_tag" (tag-delimited constrained regions). When set to a constrained mode, the engine compiles the specification into a GrammarMatcher via the @mlc-ai/web-xgrammar library at inference time.

Description

The ResponseFormat interface is the primary user-facing API for structured output in web-llm. It is defined in src/openai_api_protocols/chat_completion.ts (lines 1194-1223) and re-exported from the package's public API.

The interface has four fields:

type (optional) -- One of "text", "json_object", "grammar", or "structural_tag". Determines which constraint mechanism is used.
schema (optional) -- A JSON Schema string. Only valid when type is "json_object". Specifies the exact structure the JSON output must conform to, including property names, types, and required fields.
grammar (optional) -- An EBNF grammar string. Required when type is "grammar", and must not be specified otherwise. Defines formal production rules for the output format.
structural_tag (optional) -- A StructuralTagLike object or string. Required when type is "structural_tag". Defines trigger-based constrained regions within otherwise free-form text.

Validation Rules

The request validation logic (in the same file) enforces these constraints:

If schema is provided, type must be "json_object".
If grammar is provided, type must be "grammar". If type is "grammar", grammar must be provided.
If structural_tag is provided, type must be "structural_tag". If type is "structural_tag", structural_tag must be provided.

Code Reference

Interface Definition

Source: src/openai_api_protocols/chat_completion.ts, lines 1194-1223

export interface ResponseFormat {
  /**
   * Must be one of `text`, `json_object`, `grammar`, or `structural_tag`.
   */
  type?: "text" | "json_object" | "grammar" | "structural_tag";
  /**
   * A schema string in the format of the schema of a JSON file.
   * `type` needs to be `json_object`.
   */
  schema?: string;
  /**
   * An EBNF-formatted string. Needs to be specified when, and only specified when,
   * `type` is `grammar`. The grammar will be normalized (simplified) by default.
   * EBNF grammar: see https://www.w3.org/TR/xml/#sec-notation.
   */
  grammar?: string;
  /**
   * A structural tag definition. Needs to be specified when, and only when,
   * `type` is `structural_tag`.
   */
  structural_tag?: StructuralTagLike | string;
}

Where ResponseFormat is Used in the Request

Source: src/openai_api_protocols/chat_completion.ts, line 251

export interface ChatCompletionRequestBase {
  // ... other fields ...
  response_format?: ResponseFormat;
  // ... other fields ...
}

Import

import { ResponseFormat } from "@mlc-ai/web-llm";
// Or construct inline within the request object

I/O Contract

Direction	Type	Description
Input	User-defined schema, grammar, or tag definition	A JSON Schema string, EBNF grammar string, or `StructuralTagLike` object
Output	`ResponseFormat` object	Assigned to `request.response_format` on a `ChatCompletionRequest`

Type-to-Field Mapping

`type` Value	Required Field	Compiled By
`"text"`	none	No grammar compilation
`"json_object"`	`schema` (optional)	`GrammarCompiler.compileJSONSchema()` if schema provided; `compileBuiltinJSONGrammar()` otherwise
`"grammar"`	`grammar` (required)	`GrammarCompiler.compileGrammar()`
`"structural_tag"`	`structural_tag` (required)	`GrammarCompiler.compileStructuralTag()`

Usage Examples

Example 1: JSON Schema Mode

import * as webllm from "@mlc-ai/web-llm";

const engine = await webllm.CreateMLCEngine("Phi-3.5-mini-instruct-q4f16_1-MLC");

// Define schema as a JSON Schema string
const personSchema = JSON.stringify({
  type: "object",
  properties: {
    name: { type: "string" },
    house: {
      type: "string",
      enum: ["Gryffindor", "Hufflepuff", "Ravenclaw", "Slytherin"],
    },
    blood_status: {
      type: "string",
      enum: ["Pure-blood", "Half-blood", "Muggle-born"],
    },
    wand: {
      type: "object",
      properties: {
        wood: { type: "string" },
        core: { type: "string" },
        length: { type: "number" },
      },
      required: ["wood", "core", "length"],
    },
    alive: { type: "boolean" },
  },
  required: ["name", "house", "blood_status", "wand", "alive"],
});

const request: webllm.ChatCompletionRequest = {
  stream: false,
  messages: [
    {
      role: "user",
      content:
        "Hermione Granger is a character in Harry Potter. " +
        "Fill in the following information about her in JSON format.",
    },
  ],
  max_tokens: 256,
  response_format: {
    type: "json_object",
    schema: personSchema,
  } as webllm.ResponseFormat,
};

const reply = await engine.chat.completions.create(request);
console.log(reply.choices[0].message.content);
// Output is guaranteed to be valid JSON matching personSchema

Example 2: EBNF Grammar Mode

import * as webllm from "@mlc-ai/web-llm";

const engine = await webllm.CreateMLCEngine("Llama-3.2-3B-Instruct-q4f16_1-MLC");

const jsonGrammarStr = String.raw`
root ::= basic_array | basic_object
basic_any ::= basic_number | basic_string | basic_boolean | basic_null | basic_array | basic_object
basic_integer ::= ("0" | "-"? [1-9] [0-9]*) ".0"?
basic_number ::= ("0" | "-"? [1-9] [0-9]*) ("." [0-9]+)? ([eE] [+-]? [0-9]+)?
basic_string ::= (([\"] basic_string_1 [\"]))
basic_string_1 ::= "" | [^"\\\x00-\x1F] basic_string_1 | "\\" escape basic_string_1
escape ::= ["\\/bfnrt] | "u" [A-Fa-f0-9] [A-Fa-f0-9] [A-Fa-f0-9] [A-Fa-f0-9]
basic_boolean ::= "true" | "false"
basic_null ::= "null"
basic_array ::= "[" ("" | ws basic_any (ws "," ws basic_any)*) ws "]"
basic_object ::= "{" ("" | ws basic_string ws ":" ws basic_any ( ws "," ws basic_string ws ":" ws basic_any)*) ws "}"
ws ::= [ \n\t]*
`;

const request: webllm.ChatCompletionRequest = {
  stream: false,
  messages: [{ role: "user", content: "Introduce yourself in JSON" }],
  max_tokens: 128,
  response_format: {
    type: "grammar",
    grammar: jsonGrammarStr,
  } as webllm.ResponseFormat,
};

const reply = await engine.chatCompletion(request);
console.log(reply.choices[0].message.content);
// Output is guaranteed to be valid JSON (arrays or objects)

Example 3: Structural Tag Mode

import * as webllm from "@mlc-ai/web-llm";

const engine = await webllm.CreateMLCEngine("Llama-3.2-1B-Instruct-q4f16_1-MLC");

const tools = [
  {
    name: "get_weather",
    schema: {
      type: "object",
      properties: {
        location: { type: "string", description: "City name" },
        unit: { type: "string", enum: ["celsius", "fahrenheit"] },
      },
      required: ["location"],
    },
  },
  {
    name: "get_time",
    schema: {
      type: "object",
      properties: {
        timezone: { type: "string", description: "IANA timezone name" },
      },
      required: [],
    },
  },
];

const responseFormat: webllm.ResponseFormat = {
  type: "structural_tag",
  structural_tag: {
    type: "structural_tag",
    format: {
      type: "triggered_tags",
      triggers: ["<tool_call>"],
      tags: tools.map((tool) => ({
        begin: `<tool_call>\n{"name": "${tool.name}", "arguments": `,
        content: { type: "json_schema", json_schema: tool.schema },
        end: "}\n</tool_call>",
      })),
      at_least_one: true,
      stop_after_first: false,
    },
  },
};

const reply = await engine.chat.completions.create({
  stream: false,
  messages: [
    { role: "system", content: "You are a tool-calling assistant." },
    { role: "user", content: "What is the weather in Paris and the time in UTC?" },
  ],
  max_tokens: 1024,
  response_format: responseFormat,
});
console.log(reply.choices[0].message.content);
// Output contains <tool_call> blocks with grammar-constrained JSON arguments

Example 4: Plain JSON Mode (No Schema)

import * as webllm from "@mlc-ai/web-llm";

const engine = await webllm.CreateMLCEngine("Llama-3.2-3B-Instruct-q4f16_1-MLC");

const request: webllm.ChatCompletionRequest = {
  stream: false,
  messages: [
    { role: "user", content: "Write a short JSON file introducing yourself." },
  ],
  n: 2,
  max_tokens: 128,
  response_format: { type: "json_object" } as webllm.ResponseFormat,
};

const reply = await engine.chatCompletion(request);
console.log(reply.choices[0].message.content);
// Output is valid JSON, but not constrained to any particular schema

Related Pages

Principle: Schema Definition -- Principle:Mlc_ai_Web_llm_Schema_Definition
Implementation: Grammar Matcher Decoding -- The decoding-loop integration that applies grammar constraints derived from ResponseFormat
Implementation: JSON Parse Output -- Pattern for parsing the guaranteed-valid output produced by constrained inference

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment