Environment:Lakeraai Pint benchmark Python 310 With Transformers

Knowledge Sources	PINT Benchmark HuggingFace Transformers SetFit Documentation
Domains	NLP, Model_Evaluation, Infrastructure
Last Updated	2026-02-14 15:00 GMT

Overview

Python 3.10+ environment with PyTorch, HuggingFace Transformers, and SetFit for evaluating prompt injection detection models via the PINT Benchmark.

Description

This environment provides the runtime context required to load and run HuggingFace-hosted prompt injection detection models. It includes the full PyTorch stack for GPU-accelerated inference (with CPU fallback), the HuggingFace Transformers library for model loading and pipeline construction, and the SetFit library for few-shot classification models. The environment automatically detects CUDA availability and configures the inference device accordingly.

Usage

Use this environment when evaluating HuggingFace-hosted models with the PINT Benchmark. It is the mandatory prerequisite for the HuggingFaceModelEvaluation class, which wraps model loading, tokenization, chunked inference, and result aggregation. If you are only benchmarking API-based systems (e.g., Lakera Guard, AWS Bedrock), this environment is not required.

System Requirements

Category	Requirement	Notes
OS	Linux, macOS, or Windows (WSL2)	Standard Python-supported platforms
Hardware	CPU (minimum) or NVIDIA GPU	GPU with CUDA support recommended for faster inference
Disk	5GB+ free space	For model weight downloads and caching
Python	3.10+	Specified in pyproject.toml: `python = "^3.10"`

Dependencies

System Packages

Python 3.10 or higher
poetry (package manager, used for project setup)

Python Packages

torch (PyTorch — GPU inference and tensor operations)
transformers (HuggingFace — AutoModelForSequenceClassification, AutoTokenizer, pipeline)
setfit (SetFit — SetFitModel.from_pretrained for few-shot models)

Credentials

No API credentials are required for this environment. HuggingFace model downloads are public by default. If accessing gated models, set:

HF_TOKEN: HuggingFace API token (Read access) for downloading gated models like meta-llama/Prompt-Guard-86M

Quick Install

# Install core HuggingFace model evaluation dependencies
pip install torch transformers setfit

Code Evidence

GPU/CPU device detection from benchmark/utils/evaluate_hugging_face_model.py:52:

self.classifier = pipeline(
    "text-classification",
    model=self.model,
    tokenizer=self.tokenizer,
    max_length=self.max_length,
    truncation=True,
    device=torch.device("cuda" if torch.cuda.is_available() else "cpu"),
)

SetFit model loading from benchmark/utils/evaluate_hugging_face_model.py:60-61:

if self.is_setfit:
    return SetFitModel.from_pretrained(self.model_name)

Standard model loading from benchmark/utils/evaluate_hugging_face_model.py:62-63:

else:
    return AutoModelForSequenceClassification.from_pretrained(self.model_name)

Import declarations from benchmark/utils/evaluate_hugging_face_model.py:1-6:

import math
from typing import Any, Optional

import torch
from setfit import SetFitModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

Common Errors

Error Message	Cause	Solution
`ModuleNotFoundError: No module named 'torch'`	PyTorch not installed	`pip install torch`
`ModuleNotFoundError: No module named 'transformers'`	Transformers not installed	`pip install transformers`
`ModuleNotFoundError: No module named 'setfit'`	SetFit not installed (only needed for SetFit models)	`pip install setfit`
`OSError: ... is not a local folder and is not a valid model identifier`	Model name typo or gated model without auth	Verify model name; set `HF_TOKEN` env var for gated models
`RuntimeError: CUDA out of memory`	GPU VRAM insufficient for model	Use CPU fallback or a GPU with more VRAM

Compatibility Notes

CPU-only systems: Fully supported. The code auto-detects CUDA availability and falls back to CPU via torch.device("cuda" if torch.cuda.is_available() else "cpu").
SetFit models: Require the separate setfit package and a different tokenizer model (e.g., sentence-transformers/paraphrase-MiniLM-L3-v2). Set is_setfit=True and provide tokenizer_model.
Jupyter Notebook: This environment is designed to run inside the pint-benchmark.ipynb notebook. The HuggingFaceModelEvaluation class is pre-imported.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment