Principle:Togethercomputer Together python Chat Completion Request

Attribute	Value
Principle Name	Chat_Completion_Request
Overview	Mechanism for sending conversation messages to a language model and receiving generated completions.
Domain	NLP, API_Client, Inference
Repository	togethercomputer/together-python
Last Updated	2026-02-15 16:00 GMT

Description

Chat completion request is the core inference operation that sends a structured conversation to a hosted large language model (LLM) and receives generated text responses. It is the primary mechanism for interacting with Together AI's chat-capable models.

The request accepts a list of conversation messages and a model identifier as required inputs, along with an extensive set of optional parameters that control the generation process. These parameters fall into several categories:

Stopping Criteria

max_tokens -- Maximum number of tokens to generate before stopping.
stop -- A list of strings that, when encountered in the generated text, cause generation to halt.

Sampling Parameters

temperature -- Controls randomness. Higher values (e.g., 1.0) produce more diverse output; lower values (e.g., 0.1) produce more deterministic output.
top_p -- Nucleus sampling. Only considers tokens whose cumulative probability exceeds this threshold.
top_k -- Limits token selection to the top K most likely candidates.
min_p -- Minimum probability threshold a token must reach to be considered during sampling.
repetition_penalty -- Penalizes repeated sequences to encourage diversity.
presence_penalty -- Penalizes tokens based on whether they have appeared in the text so far.
frequency_penalty -- Penalizes tokens based on how frequently they have appeared.
logit_bias -- Directly modifies the logits of specific tokens.
seed -- Enables reproducible generation when set to a fixed value.

Output Modes

stream -- When enabled, returns an iterator of incremental chunks via Server-Sent Events (SSE) instead of waiting for the complete response.
n -- Number of independent completions to generate.
logprobs -- Returns log probabilities for the top-k tokens at each position.
echo -- Echoes the input prompt in the output, useful with logprobs.

Structured Output

response_format -- Constrains the model output to a specific format: JSON object, JSON schema, or regex pattern.
tools -- Defines available functions the model can call.
tool_choice -- Controls whether and which function the model should call.

Safety

safety_model -- Applies a moderation model to filter generated tokens.

Usage

Use chat completion requests whenever you need to generate text responses from a language model given a conversation context.

When to use:

Single-turn question answering
Multi-turn conversational AI
Function/tool calling workflows
Structured data extraction with JSON mode
Streaming real-time responses to users
Generating multiple candidate responses for selection

When not to use:

For embedding generation -- use the embeddings API instead
For image generation -- use the images API instead
For non-chat text completion -- use the completions API instead
For batch processing of many requests -- consider the batches API

Theoretical Basis

The request configures autoregressive text generation with several sampling strategies:

Temperature sampling scales the logit distribution before applying softmax, controlling the entropy of the token probability distribution.
Nucleus sampling (top_p) dynamically adjusts the vocabulary size by only considering tokens in the smallest set whose cumulative probability exceeds the threshold.
Top-k filtering truncates the distribution to the K most probable tokens.
Min-p filtering removes tokens below a minimum probability threshold, providing adaptive vocabulary pruning.
Penalty mechanisms (repetition, presence, frequency) modify logits post-hoc to discourage repetitive generation.
Logit bias provides direct manipulation of individual token probabilities.

Streaming uses the Server-Sent Events (SSE) protocol, where each generated token (or small group of tokens) is sent as an individual event, enabling progressive rendering in user interfaces.

Knowledge Sources

Source	Type	URI
Together AI Chat Completions API	Doc	Together AI Chat Overview
Together AI Inference Parameters	Doc	Together AI Chat Completions Reference
Together AI JSON Mode	Doc	Together AI JSON Mode
Together AI Function Calling	Doc	Together AI Function Calling

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment