Principle:Deepset ai Haystack LLM Chat Generation
Overview
LLM chat generation extends text generation to multi-turn conversations with role-based messaging, supporting system instructions, user queries, and assistant responses. It enables interactive, context-aware dialogue with large language models and supports tool/function calling for agentic workflows.
Domains
- NLP
- Generation
Theory
Chat completion models process sequences of role-tagged messages (system, user, assistant) and generate contextually appropriate responses. This paradigm extends simple prompt-in/text-out generation to a more structured conversational interface.
Role-Based Message Processing
Unlike flat text generation, chat models operate on ordered sequences of messages, each tagged with a role that provides semantic context:
- System: Establishes the model's behavior, persona, constraints, and output format. Typically appears once at the beginning of the conversation. The system message persists across the entire conversation and influences all subsequent responses.
- User: Represents human input -- questions, instructions, or information provided by the end user.
- Assistant: Contains prior model responses, enabling multi-turn context. Can also be used for few-shot examples where the model is shown desired response patterns.
- Tool/Function: Contains results from tool or function calls, enabling the model to incorporate external data or computation results into its responses.
Multi-Turn Conversation
Chat generation maintains conversational context by accepting the full message history as input. The model conditions its response on the entire sequence of prior messages, enabling:
- Context continuity: The model can reference and build upon previous exchanges.
- Clarification and follow-up: Users can ask follow-up questions without repeating context.
- Progressive refinement: Iterative improvement of outputs through back-and-forth interaction.
Tool and Function Calling
Modern chat models support tool calling (also known as function calling), which enables agentic workflows:
- The model is provided with definitions of available tools (name, description, parameter schema).
- When the model determines a tool should be used, it generates a structured tool call instead of (or alongside) text output, specifying the tool name and arguments.
- The application executes the tool and returns the result as a tool-role message.
- The model incorporates the tool result to produce its final response.
This mechanism allows chat models to interact with external systems (databases, APIs, calculators, search engines) in a structured, reliable manner.
Structured Output
Chat generation can enforce structured output formats through response format specifications:
- JSON mode: Ensures the model's response is valid JSON.
- JSON schema: Enforces a specific schema structure on the output, using either a JSON schema definition or a Pydantic model.
Structured outputs are particularly valuable for applications that need to parse model responses programmatically.
Streaming in Chat Context
Streaming in chat generation works similarly to text generation -- tokens are delivered incrementally via callbacks. In the chat context, streaming also supports incremental delivery of tool call arguments, enabling real-time processing of function calls as they are generated.
Relationship to Text Generation
Chat generation and text generation share the same underlying autoregressive language model architecture and decoding strategies (temperature, top_p, etc.). The key difference is the input/output interface:
- Text generation: String prompt in, string response out.
- Chat generation: List of role-tagged messages in, role-tagged message(s) out.
Chat generation is the more general paradigm and is the standard interface for modern LLM APIs.