Implementation:Protectai Llm guard Output Toxicity

Knowledge Sources	Protectai_Llm_guard
Domains	Toxicity_Detection, NLP
Last Updated	2026-02-14 12:00 GMT

Overview

Toxicity is an output scanner that detects toxic language in LLM responses by delegating to the input-side InputToxicity scanner.

Description

The Toxicity output scanner is a thin wrapper around the corresponding input scanner InputToxicity. It classifies LLM outputs to determine whether they contain toxic, harmful, or abusive language. The scanner uses a text classification model to score the toxicity of the output. The threshold parameter sets the minimum confidence score required for the text to be classified as toxic. The match_type parameter controls whether the entire output is evaluated as a whole (FULL) or split into individual sentences for per-sentence evaluation (SENTENCE). The sentence-level mode is useful for catching localized toxic content within otherwise acceptable outputs.

Usage

Use this scanner to ensure LLM outputs are free from toxic, abusive, or harmful language. This is essential for any user-facing application, particularly chatbots, content generation tools, educational platforms, and customer service systems. The scanner helps maintain safe and respectful interactions.

Code Reference

Source Location

• Repository: Protectai_Llm_guard
• File: llm_guard/output_scanners/toxicity.py
• Lines: 1-42

Signature

class Toxicity(Scanner):
    def __init__(
        self,
        *,
        model: Model | None = None,
        threshold: float = 0.7,
        match_type: MatchType | str = MatchType.FULL,
        use_onnx: bool = False,
    ) -> None: ...

    def scan(self, prompt: str, output: str) -> tuple[str, bool, float]: ...

Import

from llm_guard.output_scanners import Toxicity

I/O Contract

Inputs

Name	Type	Required	Description
prompt	str	Yes	The input prompt
output	str	Yes	The LLM output to scan for toxic language

Constructor Parameters

Name	Type	Required	Default	Description
model	None	No	None	Custom toxicity classification model
threshold	float	No	0.7	Minimum confidence score to flag as toxic
match_type	str	No	MatchType.FULL	Matching strategy: FULL (entire text) or SENTENCE (per-sentence)
use_onnx	bool	No	False	Whether to use ONNX runtime for inference

Outputs

Name	Type	Description
sanitized_output	str	The output (potentially modified)
is_valid	bool	Whether the output passed the scan (True if no toxicity detected)
risk_score	float	Risk score (-1.0 to 1.0)

Usage Examples

Basic Usage

from llm_guard.output_scanners import Toxicity

scanner = Toxicity(threshold=0.7)

prompt = "Tell me a story"
output = "Once upon a time, there was a kind princess who helped everyone in the kingdom."

sanitized_output, is_valid, risk_score = scanner.scan(prompt, output)

if is_valid:
    print("Output is free of toxic content")
else:
    print(f"Toxic content detected (risk: {risk_score})")

Sentence-Level Detection

from llm_guard.output_scanners import Toxicity

scanner = Toxicity(threshold=0.7, match_type="sentence")

prompt = "Describe the characters"
output = "The hero was brave and kind. The villain was a horrible disgusting creature that deserved to suffer."

sanitized_output, is_valid, risk_score = scanner.scan(prompt, output)
# Sentence-level scanning may flag the second sentence
print(f"Valid: {is_valid}, Risk: {risk_score}")

Related Pages

• Principle:Protectai_Llm_guard_Toxicity_Detection