Implementation:Vibrantlabsai Ragas StringMetricsV2

Knowledge Sources	Vibrantlabsai_Ragas
Domains	Evaluation, Metrics
Last Updated	2026-02-12 00:00 GMT

Overview

Provides a collection of string-based evaluation metrics (ExactMatch, StringPresence, and NonLLMStringSimilarity) that operate without requiring an LLM, using direct string comparison and character-level distance measures.

Description

This module defines three lightweight, non-LLM string comparison metrics in the V2 collections architecture:

ExactMatch checks whether the reference and response strings are exactly identical, returning 1.0 for a match and 0.0 otherwise. This is the simplest possible evaluation metric.

StringPresence checks whether the reference string is contained anywhere within the response string. This is useful for verifying that a specific keyword, entity, or phrase appears in a generated response.

NonLLMStringSimilarity computes a normalized similarity score between two strings using one of four distance measures from the rapidfuzz library: Levenshtein, Hamming, Jaro, or Jaro-Winkler. The distance is computed at the character level, then converted to a similarity score in the range 0.0 to 1.0 by subtracting the normalized distance from 1.

A helper enum DistanceMeasure is provided to select which distance algorithm to use with NonLLMStringSimilarity.

All three metrics inherit from BaseMetric and implement the async ascore method. They do not require any LLM or embedding dependencies, making them fast and deterministic.

Usage

Use ExactMatch when you need a strict binary check that two strings are identical. Use StringPresence when you want to verify that a particular substring (such as an expected entity or keyword) is present in the output. Use NonLLMStringSimilarity when you need a graded similarity score that accounts for typographical differences, partial matches, or word reordering at the character level.

These metrics are part of the V2 collections system, which provides automatic validation and a consistent async API via the BaseMetric base class. Unlike V1 string metrics, the V2 versions use class-based design with pure async execution.

Code Reference

Source Location

• Repository: Vibrantlabsai_Ragas
• File: src/ragas/metrics/collections/_string.py

Signature

class DistanceMeasure(Enum):
    LEVENSHTEIN = "levenshtein"
    HAMMING = "hamming"
    JARO = "jaro"
    JARO_WINKLER = "jaro_winkler"

class ExactMatch(BaseMetric):
    def __init__(self, name: str = "exact_match", **base_kwargs): ...
    async def ascore(self, reference: str, response: str) -> MetricResult: ...

class StringPresence(BaseMetric):
    def __init__(self, name: str = "string_present", **base_kwargs): ...
    async def ascore(self, reference: str, response: str) -> MetricResult: ...

class NonLLMStringSimilarity(BaseMetric):
    def __init__(
        self,
        name: str = "non_llm_string_similarity",
        distance_measure: DistanceMeasure = DistanceMeasure.LEVENSHTEIN,
        **base_kwargs,
    ): ...
    async def ascore(self, reference: str, response: str) -> MetricResult: ...

Import

from ragas.metrics.collections import ExactMatch, StringPresence, NonLLMStringSimilarity, DistanceMeasure

I/O Contract

ExactMatch Inputs

Name	Type	Required	Description
reference	str	Yes	The reference/ground truth text
response	str	Yes	The response text to evaluate

StringPresence Inputs

Name	Type	Required	Description
reference	str	Yes	The reference string to search for
response	str	Yes	The response text to search in

NonLLMStringSimilarity Inputs

Name	Type	Required	Description
reference	str	Yes	The reference/ground truth text
response	str	Yes	The response text to evaluate

NonLLMStringSimilarity Constructor Parameters

Name	Type	Required	Description
name	str	No	Metric name (default: "non_llm_string_similarity")
distance_measure	DistanceMeasure	No	Distance algorithm to use (default: LEVENSHTEIN). Options: LEVENSHTEIN, HAMMING, JARO, JARO_WINKLER

Outputs

Name	Type	Description
score	MetricResult (float value)	Score between 0.0 and 1.0 (1.0 indicates exact match or full similarity)

Usage Examples

ExactMatch Basic Usage

from ragas.metrics.collections import ExactMatch

metric = ExactMatch()

result = await metric.ascore(
    reference="Hello World",
    response="Hello World"
)
print(f"Score: {result.value}")  # 1.0

result = await metric.ascore(
    reference="Hello World",
    response="hello world"
)
print(f"Score: {result.value}")  # 0.0

StringPresence Basic Usage

from ragas.metrics.collections import StringPresence

metric = StringPresence()

result = await metric.ascore(
    reference="Paris",
    response="The capital of France is Paris."
)
print(f"Score: {result.value}")  # 1.0

result = await metric.ascore(
    reference="London",
    response="The capital of France is Paris."
)
print(f"Score: {result.value}")  # 0.0

NonLLMStringSimilarity Basic Usage

from ragas.metrics.collections import NonLLMStringSimilarity, DistanceMeasure

# Default Levenshtein distance
metric = NonLLMStringSimilarity()
result = await metric.ascore(
    reference="The capital of France is Paris.",
    response="Paris is the capital of France."
)
print(f"Score: {result.value}")

# Jaro-Winkler distance
metric_jw = NonLLMStringSimilarity(distance_measure=DistanceMeasure.JARO_WINKLER)
result = await metric_jw.ascore(
    reference="Hello World",
    response="Hello Wrold"
)
print(f"Score: {result.value}")

Batch Scoring

from ragas.metrics.collections import ExactMatch

metric = ExactMatch()

results = await metric.abatch_score([
    {"reference": "Text 1", "response": "Text 1"},
    {"reference": "Text 2", "response": "Different"},
])
for r in results:
    print(f"Score: {r.value}")

Related Pages

• Environment:Vibrantlabsai_Ragas_Python_3_9_Core_Environment