Principle:Protectai Llm guard Language Detection
| Knowledge Sources | |
|---|---|
| Domains | Language_Detection, NLP |
| Last Updated | 2026-02-14 12:00 GMT |
Overview
Identifying the natural language of text to enforce language policies.
Description
Language Detection is an NLP principle that determines which natural language a piece of text is written in and validates it against a set of permitted languages. Enforcing language policies is essential in multilingual deployments where a language model may be configured to serve only specific languages, or where regulatory requirements mandate that interactions occur in designated languages.
The principle uses multi-class language classification powered by the XLM-RoBERTa architecture, a cross-lingual transformer model pre-trained on text from 100+ languages. This model achieves high accuracy in language identification because its pre-training on massive multilingual corpora gives it deep familiarity with the lexical, morphological, and syntactic patterns unique to each language.
Detection is validated against a user-defined whitelist of ISO 639-1 language codes (e.g., "en" for English, "fr" for French, "de" for German). The system supports both sentence-level detection (identifying language switches within a single text) and full-text detection (classifying the dominant language of the entire input). Sentence-level detection is particularly valuable for identifying code-switching or mixed-language inputs that may violate language policies.
Usage
Use this principle when you need to enforce that interactions with a language model occur in approved languages. Common applications include customer service systems that should only process queries in supported languages, compliance systems where legal requirements dictate communication language, content moderation for region-specific platforms, and preventing language-based prompt injection attempts where attackers switch to a language the model is less robust in. Sentence-level detection is recommended when mixed-language inputs are a concern.
Theoretical Basis
The language identification algorithm works as follows:
Text Segmentation (optional):
- If sentence-level detection is enabled, split the text into individual sentences
- Otherwise, treat the full text as a single unit
Language Classification:
- Tokenize the text using the XLM-RoBERTa tokenizer (SentencePiece-based)
- Pass tokens through the cross-lingual transformer encoder
- Apply a classification head with softmax over supported language classes
- Each segment receives a probability distribution over all detectable languages
Validation:
- Determine the predicted language as argmax of the probability distribution
- Map the predicted language to its ISO 639-1 code
- Check whether the predicted language code appears in the user-defined whitelist
Decision:
- If the detected language is NOT in the whitelist, flag the text
- For sentence-level analysis, flag if ANY sentence is in a non-whitelisted language
- Return the detected language code and confidence score alongside the decision