Principle:Cleanlab Cleanlab Label Quality Scoring
| Knowledge Sources | |
|---|---|
| Domains | Machine_Learning, Data_Quality |
| Last Updated | 2026-02-09 19:00 GMT |
Overview
Method for assigning a numeric quality score to each example's label, quantifying the likelihood that the given label is correct.
Description
Label quality scoring assigns each example a score between 0 and 1, where lower scores indicate labels that are more likely to be incorrect. This provides a continuous, fine-grained measure of label quality rather than a binary label-issue / not-issue decision. Three scoring methods are available, each capturing different aspects of model uncertainty:
- self_confidence -- the model's predicted probability for the given label class, representing the model's direct confidence that the given label is correct.
- normalized_margin -- the difference between the probability of the given label and the probability of the most likely alternative class, capturing how "close" the decision is between the given label and the next best option.
- confidence_weighted_entropy -- the entropy of the predicted distribution weighted by the inverse of the self-confidence, penalizing examples where the model is both uncertain and unconfident about the given label.
These scores enable ranking all examples by label quality, which is essential for prioritized human review workflows and for integration with filtering methods that need continuous scores to select the most problematic examples.
Usage
Use when you need to rank all examples by their label quality to prioritize review of potentially mislabeled data. Unlike binary filtering, quality scores allow you to set your own threshold or review the top-N worst examples. Scores are also used internally by filtering and ordering functions.
Theoretical Basis
Let k = labels[i] be the given label for example i and P = pred_probs[i] be the model's predicted probability distribution over all classes.
Self-confidence:
self_confidence[i] = P[k]
This is simply the predicted probability of the given label. A low value means the model does not think the given label is correct.
Normalized margin:
normalized_margin[i] = P[k] - max_{j != k}(P[j])
This measures the gap between the model's confidence in the given label and its confidence in the most likely alternative. Values near -1 indicate the model strongly prefers a different class; values near +1 indicate strong agreement with the given label. The scores are then normalized to the [0, 1] range.
Confidence-weighted entropy:
entropy[i] = -sum_{c}( P[c] * log(P[c]) )
confidence_weighted_entropy[i] = entropy[i] / P[k]
This combines the overall uncertainty of the prediction with the model's confidence in the given label. High entropy and low self-confidence together produce very low quality scores. The raw values are then normalized to [0, 1] and inverted so that lower scores indicate worse label quality.
All three methods produce scores in [0, 1] with the property that lower scores indicate labels more likely to be incorrect.