Environment:Vibrantlabsai Ragas Optional NLP Metrics Environment

Knowledge Sources	Ragas Traditional Metrics
Domains	NLP, LLM_Evaluation, Metrics
Last Updated	2026-02-12 10:00 GMT

Overview

Optional dependency environment for Ragas NLP-based metrics including BLEU, ROUGE, chrF, string similarity, NLI faithfulness, data comparison, and correlation-based scoring.

Description

Several Ragas metrics rely on optional Python packages that are not included in the core installation. These metrics use traditional NLP scoring (BLEU, ROUGE, chrF), string distance algorithms (rapidfuzz), data frame comparison (datacompy/pandas), statistical correlation (scipy), and transformer-based NLI models (transformers). Each optional dependency is lazily imported and raises a clear `ImportError` with installation instructions when missing.

Usage

Use this environment when you need to run any of the following metrics:

• BleuScore: Requires `sacrebleu`
• RougeScore: Requires `rouge_score`
• ChrfScore: Requires `sacrebleu`
• NonLLMStringSimilarity: Requires `rapidfuzz`
• DataCompyScore: Requires `pandas` and `datacompy`
• Faithfulness (NLI mode): Requires `transformers`
• NumericMetric / RankingMetric (correlation): Requires `scipy`
• HuggingFace Tokenizer: Requires `transformers`
• HuggingFace Embeddings (local): Requires `sentence-transformers`
• HuggingFace Embeddings (API): Requires `huggingface-hub`

System Requirements

Category	Requirement	Notes
OS	Linux, macOS, Windows	Cross-platform support
Python	>= 3.9	Same as core Ragas requirement
Disk	~1-5GB	Transformer models can be large if using local NLI or embeddings

Dependencies

Python Packages

NLP Scoring Metrics:

• `sacrebleu` — Required for BleuScore and ChrfScore
• `rouge_score` — Required for RougeScore
• `rapidfuzz` — Required for NonLLMStringSimilarity (string distance metrics)

Data Comparison:

• `pandas` — Required for DataCompyScore and DataFrame export
• `datacompy` — Required for DataCompyScore CSV comparison

Statistical:

• `scipy` — Required for NumericMetric and RankingMetric correlation calculations

Transformer Models:

• `transformers` — Required for Faithfulness NLI classifier and HuggingFace tokenizer
• `sentence-transformers` — Required for local HuggingFace embedding models
• `huggingface-hub` — Required for HuggingFace API-based embeddings

Credentials

• `HF_TOKEN` or `HUGGINGFACE_TOKEN`: Optional. Required only for gated HuggingFace models.

Quick Install

# Install all optional NLP dependencies at once
pip install ragas[all]

# Or install individually as needed
pip install sacrebleu rouge_score rapidfuzz pandas datacompy scipy
pip install transformers sentence-transformers huggingface-hub

Code Evidence

BLEU score lazy import from `src/ragas/metrics/_bleu_score.py:22-26`:

try:
    from sacrebleu import corpus_bleu
except ImportError:
    raise ImportError(
        "sacrebleu is required for bleu score. Please install it using `pip install sacrebleu`"
    )

ROUGE score lazy import from `src/ragas/metrics/_rouge_score.py:21-26`:

try:
    from rouge_score import rouge_scorer
except ImportError as e:
    raise ImportError(
        f"{e.name} is required for rouge score. Please install it using `pip install {e.name}"
    )

String similarity lazy import from `src/ragas/metrics/_string.py:70-75`:

try:
    from rapidfuzz import distance
except ImportError:
    raise ImportError(
        "rapidfuzz is required for string distance. Please install it using `pip install rapidfuzz`"
    )

Faithfulness NLI model import from `src/ragas/metrics/_faithfulness.py:224-229`:

try:
    from transformers import AutoModelForSequenceClassification
except ImportError:
    raise ImportError(
        "Huggingface transformers must be installed to use this feature, try `pip install transformers`"
    )

Scipy for correlation from `src/ragas/metrics/numeric.py:82`:

except ImportError:
    raise ImportError(
        "scipy is required for correlation calculation. "
        "Please install it using `pip install scipy`"
    )

HuggingFace local embeddings from `src/ragas/embeddings/huggingface_provider.py:63-69`:

try:
    from sentence_transformers import SentenceTransformer
except ImportError:
    raise ImportError(
        "Local HuggingFace models require sentence-transformers. "
        "Install with: pip install sentence-transformers"
    )

Common Errors

Error Message	Cause	Solution
`ImportError: sacrebleu is required for bleu score`	sacrebleu not installed	`pip install sacrebleu`
`ImportError: rapidfuzz is required for string distance`	rapidfuzz not installed	`pip install rapidfuzz`
`ImportError: Huggingface transformers must be installed`	transformers not installed, needed for NLI faithfulness	`pip install transformers`
`ImportError: scipy is required for correlation calculation`	scipy not installed, needed for NumericMetric	`pip install scipy`
`ImportError: Local HuggingFace models require sentence-transformers`	sentence-transformers not installed	`pip install sentence-transformers`
`ImportError: datacompy is required`	datacompy/pandas not installed	`pip install pandas datacompy`

Compatibility Notes

• Lazy Imports: All optional dependencies use lazy importing via `try/except ImportError`. The core Ragas package works without any of these installed.
• Device Parameter: HuggingFace embeddings and NLI models accept an optional `device` parameter (e.g., `"cpu"`, `"cuda"`). Default is `None` (auto-detect).
• Collections vs Legacy: Both the legacy metrics (`src/ragas/metrics/_*.py`) and the modern collections (`src/ragas/metrics/collections/`) have the same optional dependency requirements with identical lazy import patterns.