Implementation:AUTOMATIC1111 Stable diffusion webui CLIP Interrogator

Knowledge Sources	AUTOMATIC1111_Stable_diffusion_webui
Domains	Image_Interrogation, CLIP, BLIP
Last Updated	2025-05-15 00:00 GMT

Overview

Implements image-to-text interrogation using BLIP for caption generation and CLIP for matching images against category dictionaries of artists, styles, and mediums.

Description

The CLIP Interrogator module provides the InterrogateModels class that combines two models to reverse-engineer text descriptions from images. First, a BLIP model generates a natural language caption for the image. Then, a CLIP model encodes the image and compares its features against text embeddings from category dictionaries (artists, flavors, mediums, movements) loaded from text files. The top-matching terms from each category are appended to the BLIP caption. Categories are loaded from .txt files in a content directory, with support for .topN suffixes to control how many matches per category are returned. The module handles model lifecycle management including lazy loading, half-precision support, device placement (with a separate device_interrogate), and memory cleanup. A fake fairscale module is injected to avoid import errors. Category files are automatically downloaded from the clip-interrogator repository if not present.

Usage

Use this module to generate text descriptions of images for prompt engineering, image cataloging, or understanding what Stable Diffusion would interpret from a given image.

Code Reference

Source Location

• Repository: AUTOMATIC1111_Stable_diffusion_webui
• File: modules/interrogate.py
• Lines: 1-222

Signature

def category_types() -> list[str]
def download_default_clip_interrogate_categories(content_dir: str) -> None

class InterrogateModels:
    def __init__(self, content_dir: str) -> None
    def categories(self) -> list[Category]
    def load_blip_model(self) -> models.blip.BlipDecoder
    def load_clip_model(self) -> tuple[clip.model, transforms.Compose]
    def load(self) -> None
    def send_clip_to_ram(self) -> None
    def send_blip_to_ram(self) -> None
    def unload(self) -> None
    def rank(self, image_features, text_array, top_count=1) -> list[tuple[str, float]]
    def generate_caption(self, pil_image) -> str
    def interrogate(self, pil_image) -> str

Import

from modules.interrogate import InterrogateModels

I/O Contract

Inputs

Name	Type	Required	Description
content_dir	str	Yes	Path to the directory containing category text files
pil_image	PIL.Image	Yes	The image to interrogate
image_features	torch.Tensor	Yes	CLIP-encoded image features (used by rank method)
text_array	list[str]	Yes	List of text candidates to rank against image features
top_count	int	No	Number of top matches to return per category (default 1)

Outputs

Name	Type	Description
result	str	A comma-separated string starting with the BLIP caption followed by top-ranked CLIP category matches
ranked_matches	list[tuple[str, float]]	List of (text, score) pairs ranked by similarity to the image

Usage Examples

from modules.interrogate import InterrogateModels
from PIL import Image

interrogator = InterrogateModels(content_dir="/path/to/interrogate")

image = Image.open("artwork.png")
description = interrogator.interrogate(image)
print(description)
# e.g., "a painting of a woman in a red dress, by claude monet, impressionism, oil on canvas"

Related Pages

• Principle:AUTOMATIC1111_Stable_diffusion_webui_Image_Interrogation