Implementation:Explodinggradients Ragas Loss Classes

Loss Classes

Loss Classes implement the Optimization Loss Functions principle in the Ragas evaluation toolkit. They provide concrete fitness/loss functions for measuring alignment between metric predictions and human annotations during prompt optimization.

Source Location

• File: src/ragas/losses.py
• Loss (abstract base): Lines 8-27
• MSELoss: Lines 30-44
• BinaryMetricLoss: Lines 47-113

Import

from ragas.losses import BinaryMetricLoss, MSELoss

To import the abstract base class:

from ragas.losses import Loss

Class Hierarchy

Loss (ABC)
  ├── MSELoss
  └── BinaryMetricLoss

Loss (Abstract Base Class)

Definition (Lines 8-27)

class Loss(ABC):
    """Abstract base class for all loss functions."""

    @abstractmethod
    def __call__(self, predicted: List, actual: List) -> float:
        raise NotImplementedError

All loss functions implement the __call__ protocol so they can be invoked as loss_fn(predicted, actual). The class also provides a Pydantic core schema handler for integration with Pydantic-based validation.

MSELoss

Definition (Lines 30-44)

class MSELoss(Loss):
    """Mean Squared Error loss function."""

    reduction: Literal["mean", "sum"] = "mean"

    def __call__(self, predicted: List[float], actual: List[float]) -> float

Constructor

loss = MSELoss(reduction="mean")  # or reduction="sum"

Parameter	Type	Default	Description
reduction	Literal["mean", "sum"]	"mean"	How to reduce the per-sample squared errors.

Behavior

• mean -- Returns the average of squared errors: sum((p - a) ** 2 for p, a in zip(predicted, actual)) / len(errors)
• sum -- Returns the sum of squared errors without averaging.
• Raises ValueError for any other reduction value.

Usage Example

from ragas.losses import MSELoss

loss_fn = MSELoss(reduction="mean")
score = loss_fn(predicted=[0.8, 0.6, 0.9], actual=[1.0, 0.5, 0.7])
# score = ((0.8-1.0)**2 + (0.6-0.5)**2 + (0.9-0.7)**2) / 3
# score = (0.04 + 0.01 + 0.04) / 3 = 0.03

BinaryMetricLoss

Definition (Lines 47-113)

class BinaryMetricLoss(Loss):
    """Computes the loss for binary metrics. Supports accuracy and F1-score."""

    metric: Literal["accuracy", "f1_score"] = "accuracy"

    def __call__(self, predicted: List[int], actual: List[int]) -> float

Constructor

loss = BinaryMetricLoss(metric="accuracy")  # or metric="f1_score"

Parameter	Type	Default	Description
metric	Literal["accuracy", "f1_score"]	"accuracy"	Which classification metric to compute.

Validation

Raises ValueError if the predicted and actual lists have different lengths.

Accuracy Mode (Lines 81-91)

def _accuracy(self, predicted: list[int], actual: List[int]) -> float:
    correct = sum(p == a for p, a in zip(predicted, actual))
    return correct / len(actual)

Returns the proportion of predictions that exactly match the ground truth.

F1-Score Mode (Lines 93-113)

def _f1_score(self, predicted: List[int], actual: List[int]) -> float:
    tp = sum(p == 1 and a == 1 for p, a in zip(predicted, actual))
    fp = sum(p == 1 and a == 0 for p, a in zip(predicted, actual))
    fn = sum(p == 0 and a == 1 for p, a in zip(predicted, actual))

    precision = tp / (tp + fp) if tp + fp > 0 else 0
    recall = tp / (tp + fn) if tp + fn > 0 else 0
    f1 = (2 * precision * recall) / (precision + recall) if precision + recall > 0 else 0
    return f1

Computes the F1-score as the harmonic mean of precision and recall. Returns 0 when both precision and recall are 0.

Usage Example

from ragas.losses import BinaryMetricLoss

# Accuracy mode
loss_fn = BinaryMetricLoss(metric="accuracy")
score = loss_fn(predicted=[1, 0, 1, 1], actual=[1, 1, 1, 0])
# 2 correct out of 4 => score = 0.5

# F1-score mode
loss_fn = BinaryMetricLoss(metric="f1_score")
score = loss_fn(predicted=[1, 0, 1, 1], actual=[1, 1, 1, 0])
# tp=2, fp=1, fn=1, precision=2/3, recall=2/3, f1=2/3

Integration with Optimizers

Both loss classes are passed to optimizer optimize() methods:

from ragas.optimizers import GeneticOptimizer
from ragas.losses import BinaryMetricLoss

optimizer = GeneticOptimizer(metric=my_metric, llm=my_llm)
best_prompts = optimizer.optimize(
    dataset=annotations,
    loss=BinaryMetricLoss(metric="accuracy"),
    config={"population_size": 3},
)

The GeneticOptimizer.evaluate_fitness() method calls the loss as loss_fn(y_true, y_pred) to score each candidate. The DSPyOptimizer wraps the loss into a DSPy-compatible metric function.

Implements

• Principle: Optimization Loss Functions

See Also

• Principle:Explodinggradients_Ragas_Optimization_Loss_Functions
• GeneticOptimizer Class -- Uses loss for fitness evaluation.
• DSPyOptimizer Class -- Wraps loss as DSPy metric.
• Metric Get Correlation -- Complementary measure using statistical correlation.