Jump to content

Connect Leeroopedia MCP: Equip your AI agents to search best practices, build plans, verify code, diagnose failures, and look up hyperparameter defaults.

Implementation:Speechbrain Speechbrain EER And MinDCF

From Leeroopedia


Property Value
Implementation Name EER And MinDCF
Type API Doc
Repository speechbrain/speechbrain
Source File speechbrain/utils/metric_stats.py:L809-865 (EER), L868-937 (minDCF)
Import from speechbrain.utils.metric_stats import EER, minDCF
Related Principle Principle:Speechbrain_Speechbrain_Verification_Metrics

API Signatures

EER

def EER(positive_scores, negative_scores):
    """Computes the EER (and its threshold).

    Arguments
    ---------
    positive_scores : torch.tensor
        The scores from entries of the same class.
    negative_scores : torch.tensor
        The scores from entries of different classes.

    Returns
    -------
    EER : float
        The EER score (between 0.0 and 1.0).
    threshold : float
        The corresponding threshold for the EER score.
    """

minDCF

def minDCF(
    positive_scores,
    negative_scores,
    c_miss=1.0,
    c_fa=1.0,
    p_target=0.01,
):
    """Computes the minDCF metric for speaker verification evaluation.

    Arguments
    ---------
    positive_scores : torch.tensor
        The scores from entries of the same class.
    negative_scores : torch.tensor
        The scores from entries of different classes.
    c_miss : float
        Cost assigned to a missing error (default 1.0).
    c_fa : float
        Cost assigned to a false alarm (default 1.0).
    p_target : float
        Prior probability of having a target (default 0.01).

    Returns
    -------
    minDCF : float
        The minDCF score.
    threshold : float
        The corresponding threshold for the minDCF score.
    """

Description

These two functions compute the standard speaker verification evaluation metrics. EER finds the operating point where the false acceptance rate equals the false rejection rate. minDCF computes the minimum detection cost function, a cost-weighted combination of miss and false alarm probabilities optimized over all possible thresholds.

Parameters

EER Parameters

Parameter Type Default Description
positive_scores torch.Tensor required Scores from same-speaker (target) trials. Higher scores indicate greater similarity.
negative_scores torch.Tensor required Scores from different-speaker (non-target) trials.

minDCF Parameters

Parameter Type Default Description
positive_scores torch.Tensor required Scores from same-speaker (target) trials.
negative_scores torch.Tensor required Scores from different-speaker (non-target) trials.
c_miss float 1.0 Cost of a missed detection (failing to identify a target speaker).
c_fa float 1.0 Cost of a false alarm (incorrectly accepting a non-target speaker).
p_target float 0.01 Prior probability of the target speaker appearing in a trial.

Inputs

  • positive_scores: A 1-D torch.Tensor of cosine similarity (or other metric) scores for trials where the enrollment and test utterances belong to the same speaker.
  • negative_scores: A 1-D torch.Tensor of scores for trials where the utterances belong to different speakers.

Outputs

EER Returns

Output Type Description
eer float The Equal Error Rate, between 0.0 and 1.0. Multiply by 100 for percentage.
threshold float The score threshold at which FAR equals FRR.

minDCF Returns

Output Type Description
min_dcf float The minimum detection cost function value.
threshold float The score threshold that achieves the minimum DCF.

Implementation Details

EER Algorithm

def EER(positive_scores, negative_scores):
    # 1. Compute candidate thresholds from all scores
    thresholds, _ = torch.sort(torch.cat([positive_scores, negative_scores]))
    thresholds = torch.unique(thresholds)

    # 2. Add intermediate thresholds for finer resolution
    intermediate_thresholds = (thresholds[0:-1] + thresholds[1:]) / 2
    thresholds, _ = torch.sort(torch.cat([thresholds, intermediate_thresholds]))

    # 3. Search for threshold where |FAR - FRR| is minimized
    min_index = 0
    final_FRR = 0
    final_FAR = 0
    for i, cur_thresh in enumerate(thresholds):
        FRR = (positive_scores <= cur_thresh).sum().float() / positive_scores.shape[0]
        FAR = (negative_scores > cur_thresh).sum().float() / negative_scores.shape[0]
        if (FAR - FRR).abs().item() < abs(final_FAR - final_FRR) or i == 0:
            min_index = i
            final_FRR = FRR.item()
            final_FAR = FAR.item()

    # 4. Return average of FAR and FRR at the best threshold
    EER = (final_FAR + final_FRR) / 2
    return float(EER), float(thresholds[min_index])

Steps:

  1. All positive and negative scores are sorted and deduplicated to form candidate thresholds.
  2. Intermediate thresholds (midpoints) are inserted between consecutive thresholds for finer granularity.
  3. For each threshold, FAR (fraction of negative scores above threshold) and FRR (fraction of positive scores at or below threshold) are computed.
  4. The threshold minimizing |FAR - FRR| is selected.
  5. EER is reported as (FAR + FRR) / 2 since exact equality may not occur.

minDCF Algorithm

def minDCF(positive_scores, negative_scores, c_miss=1.0, c_fa=1.0, p_target=0.01):
    # 1. Same threshold computation as EER
    thresholds, _ = torch.sort(torch.cat([positive_scores, negative_scores]))
    thresholds = torch.unique(thresholds)
    intermediate_thresholds = (thresholds[0:-1] + thresholds[1:]) / 2
    thresholds, _ = torch.sort(torch.cat([thresholds, intermediate_thresholds]))

    # 2. Vectorized computation of p_miss and p_fa for all thresholds
    positive_scores = torch.cat(len(thresholds) * [positive_scores.unsqueeze(0)])
    pos_scores_threshold = positive_scores.transpose(0, 1) <= thresholds
    p_miss = pos_scores_threshold.sum(0).float() / positive_scores.shape[1]

    negative_scores = torch.cat(len(thresholds) * [negative_scores.unsqueeze(0)])
    neg_scores_threshold = negative_scores.transpose(0, 1) > thresholds
    p_fa = neg_scores_threshold.sum(0).float() / negative_scores.shape[1]

    # 3. Compute DCF for all thresholds and find minimum
    c_det = c_miss * p_miss * p_target + c_fa * p_fa * (1 - p_target)
    c_min, min_index = torch.min(c_det, dim=0)

    return float(c_min), float(thresholds[min_index])

Key difference from EER: The minDCF computation is vectorized -- it broadcasts the score tensors against all thresholds simultaneously, computing p_miss and p_fa in a single operation. This is more memory-intensive but significantly faster for large score sets.

Usage Examples

Basic EER and minDCF Computation

import torch
from speechbrain.utils.metric_stats import EER, minDCF

# Example scores from a verification experiment
positive_scores = torch.tensor([0.85, 0.92, 0.78, 0.88, 0.95, 0.82])
negative_scores = torch.tensor([0.31, 0.45, 0.22, 0.38, 0.15, 0.42])

# Compute EER
eer, eer_threshold = EER(positive_scores, negative_scores)
print(f"EER: {eer * 100:.2f}%")          # e.g., "EER: 0.00%"
print(f"EER threshold: {eer_threshold:.4f}")

# Compute minDCF with default parameters (NIST SRE 2010)
min_dcf, dcf_threshold = minDCF(positive_scores, negative_scores)
print(f"minDCF: {min_dcf:.4f}")
print(f"minDCF threshold: {dcf_threshold:.4f}")

# Compute minDCF with custom cost parameters
min_dcf_custom, _ = minDCF(
    positive_scores, negative_scores,
    c_miss=10.0, c_fa=1.0, p_target=0.001
)
print(f"minDCF (high miss cost): {min_dcf_custom:.4f}")

Integration with Verification Pipeline

import torch
from speechbrain.utils.metric_stats import EER, minDCF

# After running get_verification_scores()
positive_scores, negative_scores = get_verification_scores(veri_test)

# Convert to tensors
pos_tensor = torch.tensor(positive_scores)
neg_tensor = torch.tensor(negative_scores)

# Evaluate
eer, th = EER(pos_tensor, neg_tensor)
min_dcf, th = minDCF(pos_tensor, neg_tensor)

print(f"EER: {eer * 100:.2f}%")
print(f"minDCF: {min_dcf * 100:.4f}")

Using the doctest Examples

>>> import torch
>>> from speechbrain.utils.metric_stats import EER, minDCF
>>> positive_scores = torch.tensor([0.6, 0.7, 0.8, 0.5])
>>> negative_scores = torch.tensor([0.4, 0.3, 0.2, 0.1])
>>> val_eer, threshold = EER(positive_scores, negative_scores)
>>> val_eer
0.0
>>> val_minDCF, threshold = minDCF(positive_scores, negative_scores)
>>> val_minDCF
0.0

Implementation Notes

  • Threshold resolution: Both functions add intermediate thresholds between consecutive score values, effectively doubling the search resolution. This provides more accurate estimates when score distributions are sparse.
  • Memory considerations for minDCF: The vectorized approach in minDCF creates tensors of shape (num_thresholds, num_scores). For very large score sets (millions of trials), this may require significant memory. The EER function uses an iterative approach that is more memory-efficient.
  • Score scale: Both functions are agnostic to score scale -- they work with cosine similarities (range [-1, 1]), PLDA log-likelihood ratios, or any other scoring metric.
  • Return types: Both functions return Python floats (not tensors) for easy printing and logging.

See Also

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment
Retrieved from "https://leeroopedia.com/index.php?title=Implementation:Speechbrain_Speechbrain_EER_And_MinDCF&oldid=18061"