Implementation:Romsto Speculative Decoding InferenceCLI

Knowledge Sources	Speculative Decoding
Domains	Software_Engineering, CLI_Design, Benchmarking
Last Updated	2026-02-14 04:30 GMT

Overview

Concrete tool for interactively comparing speculative decoding, NASD, and autoregressive generation with configurable parameters and throughput measurement.

Description

The InferenceCLI class provides a complete REPL application that loads target and drafter models, initializes n-gram storage, and enters an interactive loop where users can submit prompts for generation and use slash commands to configure parameters.

On initialization, it loads:

Target model (default: meta-llama/Llama-3.2-3B-Instruct with int8 quantization)
Drafter model (default: meta-llama/Llama-3.2-1B-Instruct with int8 quantization)
Tokenizer (from target model)
NGramStorage(n=3) seeded from target's vocab size

The _infer method runs all enabled generation methods sequentially with a fixed seed (42), measures throughput for each, and prints comparison percentages.

The _perform_command method handles 15+ slash commands for runtime configuration.

Usage

Run this application via python infer.py --device cuda to start the interactive CLI. Enter text prompts to generate and compare outputs. Use slash commands to adjust parameters between generations.

Code Reference

Source Location

Repository: Speculative-Decoding
File: infer.py
Lines: L18-398

Signature

class InferenceCLI:

    def __init__(self, device: str = "cuda"):
        """
        Initialize CLI with model loading and REPL startup.

        Args:
            device (str): Compute device ("cuda" or "cpu").

        Default state:
            gamma=4, gen_len=35, debug=False, spec=True,
            ngram_gen=True, target_gen=True, dr=False,
            cache=False, chat=True, processor=GreedyProcessor()
        """

    def _load_models(self):
        """Load target, drafter, tokenizer, and NGramStorage."""

    def _perform_command(self, command: str):
        """Process slash commands for runtime configuration."""

    def _infer(self, prefix: str):
        """Run all enabled generation methods and compare throughput."""

    def _run(self):
        """Main REPL loop."""

    def _set_seed(self, seed: int):
        """Fix all random seeds for reproducibility."""

Import / Launch

# Launch from command line:
# python infer.py --device cuda

# Or import the class:
from infer import InferenceCLI

I/O Contract

Inputs

Name	Type	Required	Description
device	str	No	Compute device, passed via --device CLI arg (default: "cuda")

Slash Commands

Command	Description	Default
/speculative	Toggle speculative decoding	True
/ngram	Toggle NASD generation	True
/target	Toggle target autoregressive generation	True
/drafter	Toggle drafter autoregressive generation	False
/cache	Toggle KV-cache	False
/debug	Toggle debug visualization	False
/chat	Toggle chat template	True
/gamma <int>	Set draft count per round	4
/length <int>	Set max generation length	35
/processor <name> [args]	Set sampling strategy	greedy
/top_k_filler <int>	Set n-gram update enrichment k	3
/set_ngramstorage <basic/onelevel> <n>	Change n-gram storage type and order	basic 3
/reset_in_between	Toggle n-gram reset between generations	True
/clear	Clear terminal screen	N/A
/quit	Exit the application	N/A

Outputs

Name	Type	Description
Console output	str	Generated text, acceptance rates, and throughput (tokens/s) for each enabled method, plus throughput comparison percentages

Usage Examples

Basic CLI Session

# Launch the CLI
python infer.py --device cuda

# Example interaction:
# > What is machine learning?
# ========== Speculative ==========
# Out: Machine learning is a subset of...
# Acceptance rate: 0.750
# Throughput: 45.2 tokens/s
# ========== Speculative ==========
# ========== Ngram Assisted ==========
# Out: Machine learning is a subset of...
# Acceptance rate: 0.400
# Throughput: 38.1 tokens/s
# ========== Ngram Assisted ==========
# Throughput increase: 118.6%
# =========== Target AR ===========
# Out: Machine learning is a subset of...
# Throughput: 25.3 tokens/s
# =========== Target AR ===========
# Throughput increase: 178.7%

Configuring Parameters

# Set nucleus sampling
# > /processor nucleus 0.7 0.9

# Increase draft count
# > /gamma 6

# Use single-level n-gram storage with 4-grams
# > /set_ngramstorage onelevel 4

# Disable speculative decoding, keep only NASD and baseline
# > /speculative
# > What is the meaning of life?

Related Pages

Implements Principle

Principle:Romsto_Speculative_Decoding_Interactive_CLI

Requires Environment

Environment:Romsto_Speculative_Decoding_CUDA_PyTorch

Uses Heuristic

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment