Implementation:Vllm project Vllm Logprobs

Knowledge Sources	vllm
Domains	Logprobs, Sampling, OpenAI_API
Last Updated	2026-02-08 00:00 GMT

Overview

Defines data structures for storing token log probabilities and ranks, with a GC-optimized flat storage implementation for high-throughput inference.

Description

This module provides two primary data structures: Logprob, a dataclass storing per-token log probability, vocabulary rank, and decoded text; and FlatLogprobs, a memory-efficient container that flattens nested dictionaries of logprobs into primitive-type lists. FlatLogprobs implements the MutableSequence interface to maintain backward compatibility with list-based code while reducing garbage collection overhead from O(positions * top_k) objects down to a constant number of list objects. Helper functions create_prompt_logprobs, create_sample_logprobs, and append_logprobs_for_next_position manage logprob container lifecycle.

Usage

Use this module when handling logprobs in vLLM's sampling pipeline and OpenAI-compatible API responses. The FlatLogprobs container is preferred for production deployments where GC pressure from large numbers of logprob objects affects latency.

Code Reference

Source Location

• Repository: vllm
• File: vllm/logprobs.py
• Lines: 1-206

Signature

@dataclass
class Logprob:
    logprob: float
    rank: int | None = None
    decoded_token: str | None = None

LogprobsOnePosition = dict[int, Logprob]

@dataclass
class FlatLogprobs(MutableSequence[LogprobsOnePosition | None]):
    start_indices: list[int]
    end_indices: list[int]
    token_ids: list[int]
    logprobs: list[float]
    ranks: list[int | None]
    decoded_tokens: list[str | None]

    def append(self, logprobs_one_position: LogprobsOnePosition | None) -> None: ...
    def append_fast(self, token_ids, logprobs, ranks, decoded_tokens) -> None: ...
    def __getitem__(self, index: int | slice): ...
    def __len__(self) -> int: ...

PromptLogprobs = FlatLogprobs | list[LogprobsOnePosition | None]
SampleLogprobs = FlatLogprobs | list[LogprobsOnePosition]

def create_prompt_logprobs(flat_logprobs: bool) -> PromptLogprobs: ...
def create_sample_logprobs(flat_logprobs: bool) -> SampleLogprobs: ...
def append_logprobs_for_next_position(
    request_logprobs, token_ids, logprobs, decoded_tokens, rank, num_logprobs
) -> None: ...

Import

from vllm.logprobs import (
    Logprob, FlatLogprobs, LogprobsOnePosition,
    PromptLogprobs, SampleLogprobs,
    create_prompt_logprobs, create_sample_logprobs,
    append_logprobs_for_next_position,
)

I/O Contract

Inputs

Name	Type	Required	Description
logprob	float	Yes	Log probability value for a token
rank	int or None	No	Vocabulary rank of the token (1-indexed)
decoded_token	str or None	No	Decoded string representation of the token
flat_logprobs	bool	Yes	Whether to use FlatLogprobs (True) or list-based storage (False)
token_ids	list[int]	Yes (append)	Token IDs for one position's logprobs
logprobs	list[float]	Yes (append)	Corresponding log probability values
decoded_tokens	Iterable[str or None]	Yes (append)	Decoded token strings
rank	int	Yes (append)	Rank of the sampled token
num_logprobs	int	Yes (append)	Number of top logprobs requested (-1 for all)

Outputs

Name	Type	Description
PromptLogprobs	FlatLogprobs or list	Container storing prompt logprobs per position
SampleLogprobs	FlatLogprobs or list	Container storing sample/decode logprobs per position
LogprobsOnePosition	dict[int, Logprob]	Dictionary mapping token_id to Logprob for one position

Usage Examples

from vllm.logprobs import (
    create_prompt_logprobs,
    create_sample_logprobs,
    append_logprobs_for_next_position,
    Logprob,
)

# Create a flat logprobs container for prompt tokens
prompt_logprobs = create_prompt_logprobs(flat_logprobs=True)

# Create a flat logprobs container for decode tokens
sample_logprobs = create_sample_logprobs(flat_logprobs=True)

# Append logprobs for one decode step
append_logprobs_for_next_position(
    request_logprobs=sample_logprobs,
    token_ids=[1234, 5678, 9012],   # sampled + top-k token IDs
    logprobs=[-0.5, -1.2, -2.3],    # corresponding log probs
    decoded_tokens=["hello", "hi", "hey"],
    rank=1,          # rank of the sampled token
    num_logprobs=3,  # number of top logprobs requested
)

# Access logprobs for a specific position (returns dict[int, Logprob])
position_logprobs = sample_logprobs[0]

Related Pages

• Environment:Vllm_project_Vllm_Python