Overview
Toxicity is an output scanner that detects toxic language in LLM responses by delegating to the input-side InputToxicity scanner.
Description
The Toxicity output scanner is a thin wrapper around the corresponding input scanner InputToxicity. It classifies LLM outputs to determine whether they contain toxic, harmful, or abusive language. The scanner uses a text classification model to score the toxicity of the output. The threshold parameter sets the minimum confidence score required for the text to be classified as toxic. The match_type parameter controls whether the entire output is evaluated as a whole (FULL) or split into individual sentences for per-sentence evaluation (SENTENCE). The sentence-level mode is useful for catching localized toxic content within otherwise acceptable outputs.
Usage
Use this scanner to ensure LLM outputs are free from toxic, abusive, or harmful language. This is essential for any user-facing application, particularly chatbots, content generation tools, educational platforms, and customer service systems. The scanner helps maintain safe and respectful interactions.
Code Reference
Source Location
Signature
class Toxicity(Scanner):
def __init__(
self,
*,
model: Model | None = None,
threshold: float = 0.7,
match_type: MatchType | str = MatchType.FULL,
use_onnx: bool = False,
) -> None: ...
def scan(self, prompt: str, output: str) -> tuple[str, bool, float]: ...
Import
from llm_guard.output_scanners import Toxicity
I/O Contract
Inputs
| Name |
Type |
Required |
Description
|
| prompt |
str |
Yes |
The input prompt
|
| output |
str |
Yes |
The LLM output to scan for toxic language
|
Constructor Parameters
| Name |
Type |
Required |
Default |
Description
|
| model |
None |
No |
None |
Custom toxicity classification model
|
| threshold |
float |
No |
0.7 |
Minimum confidence score to flag as toxic
|
| match_type |
str |
No |
MatchType.FULL |
Matching strategy: FULL (entire text) or SENTENCE (per-sentence)
|
| use_onnx |
bool |
No |
False |
Whether to use ONNX runtime for inference
|
Outputs
| Name |
Type |
Description
|
| sanitized_output |
str |
The output (potentially modified)
|
| is_valid |
bool |
Whether the output passed the scan (True if no toxicity detected)
|
| risk_score |
float |
Risk score (-1.0 to 1.0)
|
Usage Examples
Basic Usage
from llm_guard.output_scanners import Toxicity
scanner = Toxicity(threshold=0.7)
prompt = "Tell me a story"
output = "Once upon a time, there was a kind princess who helped everyone in the kingdom."
sanitized_output, is_valid, risk_score = scanner.scan(prompt, output)
if is_valid:
print("Output is free of toxic content")
else:
print(f"Toxic content detected (risk: {risk_score})")
Sentence-Level Detection
from llm_guard.output_scanners import Toxicity
scanner = Toxicity(threshold=0.7, match_type="sentence")
prompt = "Describe the characters"
output = "The hero was brave and kind. The villain was a horrible disgusting creature that deserved to suffer."
sanitized_output, is_valid, risk_score = scanner.scan(prompt, output)
# Sentence-level scanning may flag the second sentence
print(f"Valid: {is_valid}, Risk: {risk_score}")
Related Pages
