Implementation:AUTOMATIC1111 Stable diffusion webui CLIP Interrogator
| Knowledge Sources | |
|---|---|
| Domains | Image_Interrogation, CLIP, BLIP |
| Last Updated | 2025-05-15 00:00 GMT |
Overview
Implements image-to-text interrogation using BLIP for caption generation and CLIP for matching images against category dictionaries of artists, styles, and mediums.
Description
The CLIP Interrogator module provides the InterrogateModels class that combines two models to reverse-engineer text descriptions from images. First, a BLIP model generates a natural language caption for the image. Then, a CLIP model encodes the image and compares its features against text embeddings from category dictionaries (artists, flavors, mediums, movements) loaded from text files. The top-matching terms from each category are appended to the BLIP caption. Categories are loaded from .txt files in a content directory, with support for .topN suffixes to control how many matches per category are returned. The module handles model lifecycle management including lazy loading, half-precision support, device placement (with a separate device_interrogate), and memory cleanup. A fake fairscale module is injected to avoid import errors. Category files are automatically downloaded from the clip-interrogator repository if not present.
Usage
Use this module to generate text descriptions of images for prompt engineering, image cataloging, or understanding what Stable Diffusion would interpret from a given image.
Code Reference
Source Location
- Repository: AUTOMATIC1111_Stable_diffusion_webui
- File: modules/interrogate.py
- Lines: 1-222
Signature
def category_types() -> list[str]
def download_default_clip_interrogate_categories(content_dir: str) -> None
class InterrogateModels:
def __init__(self, content_dir: str) -> None
def categories(self) -> list[Category]
def load_blip_model(self) -> models.blip.BlipDecoder
def load_clip_model(self) -> tuple[clip.model, transforms.Compose]
def load(self) -> None
def send_clip_to_ram(self) -> None
def send_blip_to_ram(self) -> None
def unload(self) -> None
def rank(self, image_features, text_array, top_count=1) -> list[tuple[str, float]]
def generate_caption(self, pil_image) -> str
def interrogate(self, pil_image) -> str
Import
from modules.interrogate import InterrogateModels
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| content_dir | str | Yes | Path to the directory containing category text files |
| pil_image | PIL.Image | Yes | The image to interrogate |
| image_features | torch.Tensor | Yes | CLIP-encoded image features (used by rank method) |
| text_array | list[str] | Yes | List of text candidates to rank against image features |
| top_count | int | No | Number of top matches to return per category (default 1) |
Outputs
| Name | Type | Description |
|---|---|---|
| result | str | A comma-separated string starting with the BLIP caption followed by top-ranked CLIP category matches |
| ranked_matches | list[tuple[str, float]] | List of (text, score) pairs ranked by similarity to the image |
Usage Examples
from modules.interrogate import InterrogateModels
from PIL import Image
interrogator = InterrogateModels(content_dir="/path/to/interrogate")
image = Image.open("artwork.png")
description = interrogator.interrogate(image)
print(description)
# e.g., "a painting of a woman in a red dress, by claude monet, impressionism, oil on canvas"