Principle:Cleanlab Cleanlab Segmentation Issue Visualization
| Knowledge Sources | |
|---|---|
| Domains | Machine Learning, Data Quality, Semantic Segmentation, Visualization |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
Segmentation issue visualization is the practice of rendering pixel-level label errors in semantic segmentation datasets through color-coded overlays and statistical summaries, enabling human inspection of annotation quality and systematic confusion patterns.
Description
Semantic segmentation assigns a class label to every pixel in an image. When these pixel-level annotations contain errors, they can be difficult to identify programmatically with full confidence, because segmentation boundaries are inherently ambiguous. Visualization bridges the gap between automated error detection and human judgment by presenting detected issues in a format that enables rapid visual assessment.
Effective segmentation issue visualization involves several key components:
- Error overlay rendering: Pixels flagged as potential label issues are highlighted (typically in a contrasting color such as red) on the original image or label map. This immediately draws attention to problematic regions without requiring the viewer to compare raw arrays.
- Multi-panel comparison: Showing the given labels, model predictions, and detected errors side-by-side allows the viewer to understand why a pixel was flagged. If the given label shows one class but the model confidently predicts another, the disagreement provides evidence for a mislabel.
- Issue prioritization: Images are ranked by the severity of their issues (e.g., total number of flagged pixels) so that the most problematic images appear first. This respects the reviewer's time by surfacing the worst cases early.
- Label swap frequency analysis: Beyond individual image inspection, aggregating which class pairs are most commonly confused reveals systematic annotation patterns. For example, if "road" pixels are frequently predicted as "sidewalk," this may indicate an ambiguous annotation guideline rather than random errors.
- Class-specific filtering: Allowing users to isolate issues involving a particular class enables focused review when certain classes are known to be problematic or safety-critical.
Usage
Use segmentation issue visualization when:
- You have run a segmentation label issue detector and obtained a pixel-level boolean issue mask.
- You need to visually validate whether flagged pixels are genuine errors or false positives.
- You want to understand systematic patterns in annotation errors across your dataset.
- You need to communicate annotation quality findings to a labeling team.
- You want to focus review efforts on a specific class that is particularly important for your application.
Theoretical Basis
1. Error Overlay with Binary Colormap:
The issue mask is an (N, H, W) boolean array. For visualization, a binary colormap maps False to transparent and True to a highlight color (e.g., red). This is rendered as an overlay on the image or label map using:
overlay[i][j][k] = error_cmap(issues[i][j][k])
Optional class exclusion masks out pixels of specified classes before display:
display_mask = issues[i] AND (NOT (labels[i] in exclude_classes))
2. Multi-Panel Layout:
Up to three panels are shown per image:
- Panel 1 (Given Labels): Each pixel is colored according to its annotated class using a distinct colormap. This shows what the annotator recorded.
- Panel 2 (Predicted Labels): Each pixel is colored according to argmax(pred_probs), showing the model's best guess. Disagreements between Panel 1 and Panel 2 suggest potential errors.
- Panel 3 (Detected Errors): The binary error overlay highlights which specific pixels are flagged.
3. Issue Prioritization by Severity:
Images are sorted by descending total issue count:
severity[i] = sum(issues[i]) display_order = argsort(-severity)[:top]
This ensures the most problematic images are reviewed first.
4. Label Swap Frequency Table:
For each flagged pixel, the given label and predicted label form a (given, predicted) pair. Aggregating these pairs across all flagged pixels in the dataset produces a confusion-style frequency table:
count[given_class][predicted_class] += 1 for each flagged pixel
Sorting by frequency reveals the most common systematic annotation errors. This is analogous to a confusion matrix but restricted to pixels the model identifies as mislabeled.
5. HSV-Based Distinct Colormap Generation:
To ensure K classes are visually distinguishable, colors are distributed across the HSV color wheel. The approach distributes linear samples across the hue dimension, then applies brightness modulation to create additional visual separation:
hues = linspace(0, 1, num_colors) colors = hsv(redistribute(hues))
Lower-indexed colors are darkened, and higher-indexed colors are lightened, maximizing perceptual distinctness even for large numbers of classes.
6. Class-Specific Filtering:
Isolating issues for a target class k combines two conditions:
class_issues[i] = (issues[i] AND labels[i] == k) OR (issues[i] AND argmax(pred_probs[i]) == k)
This captures both directions of confusion: pixels wrongly labeled as class k, and pixels that should be class k but are labeled as something else.