Workflow:Groq Groq python Streaming Chat Completion
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Inference, Streaming |
| Last Updated | 2026-02-15 16:00 GMT |
Overview
End-to-end process for streaming chat completions token-by-token from Groq-hosted LLMs using Server-Sent Events.
Description
This workflow covers the procedure for receiving incremental token-by-token responses from Groq's inference API using SSE (Server-Sent Events) streaming. Instead of waiting for the full response, the client receives partial message deltas as they are generated, enabling real-time display of output. The workflow supports both synchronous iteration (for/in loop) and asynchronous iteration (async for/in loop), making it suitable for interactive applications, CLI tools, and web backends that need to push tokens to users as they arrive.
Usage
Execute this workflow when you need real-time, incremental delivery of generated text rather than waiting for the complete response. This is appropriate for interactive chat interfaces, long-form generation where perceived latency matters, or any application that benefits from displaying partial results as they stream in.
Execution Steps
Step 1: Client Initialization
Instantiate the Groq client (synchronous or asynchronous) with authentication credentials. Configuration is identical to non-streaming usage: API key, timeouts, retries, and optional HTTP client customization all apply.
Key considerations:
- Use Groq() for synchronous streaming or AsyncGroq() for asynchronous streaming
- All client configuration options (timeouts, retries, base URL) apply equally to streaming requests
- For high-concurrency async scenarios, consider using aiohttp backend via DefaultAioHttpClient
Step 2: Message Construction
Build the messages array defining the conversation context. This step is identical to non-streaming chat completion: system messages set behavior, user messages contain the prompt, and assistant messages provide history.
Key considerations:
- Message format is identical to non-streaming requests
- Streaming does not affect the message schema or content types supported
Step 3: Streaming Request Execution
Call the chat completions create endpoint with stream=True. Instead of returning a single ChatCompletion object, the client returns a Stream (or AsyncStream) iterator that yields ChatCompletionChunk objects as they arrive over the SSE connection.
Key considerations:
- The stream=True parameter is the only difference from non-streaming requests
- All other parameters (model, temperature, max_tokens, etc.) work identically
- The returned Stream object is iterable; use a for loop (sync) or async for loop (async)
Step 4: Chunk Processing
Iterate over the stream to receive ChatCompletionChunk objects. Each chunk contains a choices array where choices[0].delta holds the incremental content. The delta object has a content field with the new token(s). Concatenate these deltas to build the full response. The final chunk includes the finish_reason and, for Groq, usage statistics in the x_groq.usage field.
Key considerations:
- Each chunk's delta.content may be None (especially the first and last chunks)
- The finish_reason on the final chunk indicates why generation stopped
- Usage statistics (token counts) are only available on the final chunk via x_groq.usage
- The stream must be fully consumed or explicitly closed to release the HTTP connection