Implementation:Romsto Speculative Decoding Autoregressive Generate

Knowledge Sources	Speculative Decoding
Domains	NLP, Inference
Last Updated	2026-02-14 04:30 GMT

Overview

Concrete tool for standard sequential token-by-token text generation, serving as the baseline for comparing speculative decoding throughput.

Description

The autoregressive_generate function implements standard autoregressive text generation for decoder-only models. It generates tokens one at a time in a loop: compute logits from the model, apply the sampling strategy, select a token, and append it to the sequence. The function supports optional KV-cache for faster sequential decoding and configurable sampling strategies via the LogitsProcessor interface.

This function is used in the CLI comparison tool as the baseline against which speculative decoding and NASD throughput are measured.

Usage

Import this function when you need standard autoregressive generation as a baseline or when speculative decoding is not applicable. It is called by the InferenceCLI when the target generation toggle is enabled.

Code Reference

Source Location

• Repository: Speculative-Decoding
• File: sampling/base_decoding.py
• Lines: L9-65

Signature

@torch.no_grad()
def autoregressive_generate(
    inputs: List[int],
    model: Module,
    max_gen_len: int = 40,
    logits_processor: LogitsProcessor = GreedyProcessor(),
    eos_tokens_id: int | List[int] = 1,
    pad_token_id: int = 0,
    use_cache: bool = False,
    debug: bool = False,
) -> List[int]:
    """
    Generate text sequence autoregressively.

    Args:
        inputs (List[int]): input sequence of batch size 1.
        model (Module): model to use for inference.
        max_gen_len (int): maximum length of generated sequence.
        logits_processor (LogitsProcessor): sampling strategy.
        eos_tokens_id (int or List[int]): end token ID(s).
        pad_token_id (int): pad token ID.
        use_cache (bool): whether to use KV-cache.
        debug (bool): debug mode.

    Returns:
        List[int]: generated token sequence.
    """

Import

from sampling import autoregressive_generate

I/O Contract

Inputs

Name	Type	Required	Description
inputs	List[int]	Yes	Tokenized input prompt (batch size 1)
model	torch.nn.Module	Yes	Decoder-only language model
max_gen_len	int	No	Maximum new tokens to generate (default: 40)
logits_processor	LogitsProcessor	No	Sampling strategy (default: GreedyProcessor())
eos_tokens_id	int or List[int]	No	End-of-sequence token ID(s) (default: 1)
pad_token_id	int	No	Padding token ID (default: 0)
use_cache	bool	No	Enable KV-cache (default: False)
debug	bool	No	Enable debug output (default: False)

Outputs

Name	Type	Description
generated_ids	List[int]	Generated token IDs (excludes the prompt, includes up to and including the EOS token if hit)

Usage Examples

Basic Autoregressive Generation

from transformers import AutoModelForCausalLM, AutoTokenizer
from sampling import autoregressive_generate
from utils.logits_processor import GreedyProcessor

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct", device_map="cuda"
)
model.eval()
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Prepare input
prompt = "What is machine learning?"
chat = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(chat, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt").input_ids[0].tolist()

# Generate
output_ids = autoregressive_generate(
    inputs,
    model,
    max_gen_len=100,
    logits_processor=GreedyProcessor(),
    eos_tokens_id=[tokenizer.eos_token_id],
    use_cache=True,
)

# Decode
print(tokenizer.decode(output_ids, skip_special_tokens=True))

Related Pages

Implements Principle

• Principle:Romsto_Speculative_Decoding_Autoregressive_Generation

Requires Environment

• Environment:Romsto_Speculative_Decoding_CUDA_PyTorch