Workflow:Openai CLIP Prompt engineered classification
| Knowledge Sources | |
|---|---|
| Domains | Computer_Vision, Zero_Shot_Learning, Prompt_Engineering |
| Last Updated | 2026-02-13 22:00 GMT |
Overview
End-to-end process for improving zero-shot image classification accuracy by ensembling predictions from multiple prompt templates per class label.
Description
This workflow extends basic zero-shot classification by using multiple text prompt templates for each class (e.g., "a photo of a {class}", "a sculpture of a {class}", "a drawing of a {class}"). For each class, the text embeddings from all templates are averaged into a single representative vector, creating a more robust classifier. This prompt ensembling technique, described in the CLIP paper, improves top-1 accuracy by several percentage points compared to using a single template, as it accounts for the polysemy and visual diversity of object categories.
Goal: Produce an ensembled zero-shot classifier from multiple prompt templates and evaluate it on a large-scale image classification benchmark.
Scope: From a class label list and a set of prompt templates to dataset-level accuracy metrics.
Strategy: Encodes each class name through multiple prompt templates, averages the resulting text embeddings per class to form classifier weight vectors, then evaluates against an image dataset by computing cosine similarity between image features and the ensembled text features.
Usage
Execute this workflow when you need the highest possible zero-shot classification accuracy on a benchmark dataset, or when you want to study the effect of prompt engineering on CLIP classification performance. This is particularly relevant for ImageNet evaluation and for understanding how text prompt diversity affects vision-language model performance. Requires a dataset with ground truth labels for accuracy measurement.
Execution Steps
Step 1: Environment setup
Install the CLIP package and its dependencies. Additionally, install any dataset-specific packages needed (e.g., ImageNetV2 for benchmark evaluation). Verify GPU availability as both text encoding of many prompts and image feature extraction benefit from acceleration.
Key considerations:
- Large-scale evaluation (e.g., ImageNet with 1000 classes and 80 templates) requires encoding 80,000 text prompts
- GPU is strongly recommended for both text and image encoding at scale
- The official ILSVRC2012 dataset requires separate access; ImageNetV2 is a publicly available alternative
Step 2: Model loading
Load a pretrained CLIP model. The choice of model variant directly affects classification accuracy; larger models (ViT-L/14) perform substantially better but require more memory and compute.
Key considerations:
- Model selection significantly impacts final accuracy
- ViT-B/32 is the fastest; ViT-L/14@336px is the most accurate
- Both the vision and text encoders are used in this workflow
Step 3: Class label and template preparation
Define the list of class names for the target dataset and a set of prompt templates. Templates are natural language phrases containing a placeholder that gets filled with each class name. The quality and diversity of templates affects classification accuracy.
Key considerations:
- Use dataset-specific class names (some may need manual adjustment, e.g., "kite" to "kite (bird of prey)" to resolve ambiguity)
- Templates should cover diverse visual contexts: photos, drawings, renderings, different scales, different perspectives
- The CLIP paper uses 80 templates for ImageNet, though a subset of 7 optimally selected templates can match or exceed full ensemble performance
- Template selection can be optimized via forward selection on a validation set
Step 4: Zero-shot classifier weight construction
For each class, encode all template-filled prompts through the text encoder, normalize the resulting embeddings, average them into a single per-class vector, and re-normalize. Stack all per-class vectors to form the classifier weight matrix.
What happens:
- For each of N classes: generate K text prompts by filling each template with the class name
- Tokenize and encode all K prompts through the text encoder
- L2-normalize each text embedding
- Average the K normalized embeddings to produce one representative vector per class
- Re-normalize the averaged vector to unit length
- Stack all N vectors into a weight matrix of shape [embedding_dim, N]
Step 5: Dataset loading and image feature extraction
Load the evaluation dataset with the CLIP preprocessing transform. Extract image features for all images in the dataset using batched inference through the frozen vision encoder.
Key considerations:
- Use data loaders with appropriate batch size and number of workers
- All image features must be L2-normalized before comparison with classifier weights
- Feature extraction is the most time-consuming step for large datasets
Step 6: Accuracy evaluation
Compute classification logits by taking the dot product of normalized image features with the ensembled classifier weight matrix, scaled by 100. Measure top-1 and top-5 accuracy across the full evaluation dataset.
What happens:
- Compute logits = 100 * (image_features @ classifier_weights) for each batch
- Determine top-k predictions from the logits
- Compare predictions against ground truth labels
- Accumulate correct predictions across all batches
- Report top-1 and top-5 accuracy percentages