Implementation:Haotian liu LLaVA Predictor Class

Knowledge Sources	Haotian_liu_LLaVA Cog Documentation
Domains	Inference, Deployment, Vision_Language
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for deploying LLaVA-v1.5-13b as a cloud-hosted streaming inference endpoint on the Replicate platform via the Cog framework.

Description

The Predictor class implements Cog's BasePredictor interface to wrap LLaVA-v1.5-13b for hosted API inference. During setup(), it downloads model weights (LLaVA-v1.5-13b and CLIP-ViT-L-14-336) from Replicate's weight mirror using pget for fast parallel downloads, then loads the model via load_pretrained_model. The predict() method accepts an image, prompt, and generation parameters, preprocesses the image with the CLIP processor, constructs a conversation using the llava_v1 template, tokenizes with image token insertion, and runs threaded generation with TextIteratorStreamer to yield tokens as a ConcatenateIterator for real-time streaming responses.

Usage

Use this class when deploying LLaVA on the Replicate platform. It provides a standard Cog prediction interface with streaming output support. Not intended for local inference; use run_llava.py or the CLI interface for local usage instead.

Code Reference

Source Location

Repository: Haotian_liu_LLaVA
File: predict.py
Lines: 78-155

Signature

class Predictor(BasePredictor):
    def setup(self) -> None:
        """Load the model into memory to make running multiple predictions efficient."""
        ...

    def predict(
        self,
        image: Path = Input(description="Input image"),
        prompt: str = Input(description="Prompt to use for text generation"),
        top_p: float = Input(description="Top-p sampling", ge=0.0, le=1.0, default=1.0),
        temperature: float = Input(description="Sampling temperature", default=0.2, ge=0.0),
        max_tokens: int = Input(description="Maximum tokens to generate", default=1024, ge=0),
    ) -> ConcatenateIterator[str]:
        """Run a single prediction on the model."""
        ...

Import

# Used by the Cog framework; not typically imported directly
from predict import Predictor

I/O Contract

Inputs

Name	Type	Required	Description
image	Path	Yes	Path to input image file
prompt	str	Yes	Text prompt for the model
top_p	float	No	Top-p nucleus sampling threshold (default: 1.0)
temperature	float	No	Sampling temperature (default: 0.2, 0 = deterministic)
max_tokens	int	No	Maximum number of tokens to generate (default: 1024)

Outputs

Name	Type	Description
predict() yields	ConcatenateIterator[str]	Streaming text tokens concatenated into a full response

Usage Examples

Cog Deployment

# Build and run with Cog
cog build
cog predict -i image=@input.jpg -i prompt="Describe this image in detail."

Replicate API Call

import replicate

output = replicate.run(
    "yorickvp/llava-v1.5-13b:latest",
    input={
        "image": open("input.jpg", "rb"),
        "prompt": "What is shown in this image?",
        "temperature": 0.2,
        "top_p": 1.0,
        "max_tokens": 1024,
    }
)
# output is a streaming iterator of text chunks
for chunk in output:
    print(chunk, end="")

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment