Principle:Protectai Modelscan Unsafe Operator Detection
| Knowledge Sources | |
|---|---|
| Domains | ML_Security, Static_Analysis |
| Last Updated | 2026-02-14 12:00 GMT |
Overview
A static analysis technique that identifies references to dangerous Python operators, functions, and modules embedded within serialized model files without executing them.
Description
Unsafe Operator Detection is the core detection mechanism in model security scanning. It works by extracting all referenced global names (module + operator pairs) from serialized data and comparing them against a curated list of known-dangerous operations. The detection is format-specific:
- Pickle-based formats (pickle, PyTorch, NumPy): The pickle bytecode is disassembled using pickletools.genops() to extract GLOBAL, INST, and STACK_GLOBAL opcodes that reference Python modules and functions.
- TensorFlow SavedModel: Protocol buffer structures are parsed to extract graph operation names (e.g., ReadFile, WriteFile).
- Keras models (H5 and .keras): Model configuration JSON is parsed to detect Lambda layers containing arbitrary Python code.
Each detected unsafe reference produces an Issue with structured details: the source module, the specific operator, the file where it was found, and a severity classification. This structured output enables automated triage and reporting.
Usage
Apply this principle when:
- Understanding how modelscan detects malicious code in model files
- Customizing the unsafe globals list for organization-specific threat models
- Implementing a new scanner that needs to report detected operators
- Building security tooling that processes modelscan findings
Theoretical Basis
The detection follows a pattern matching against a threat dictionary approach:
Pickle Bytecode Analysis
# Pseudo-code for pickle unsafe operator detection
for opcode, arg, pos in pickletools.genops(pickle_bytes):
if opcode.name == "GLOBAL":
# arg format: "module\noperator"
module, operator = arg.split("\n")
elif opcode.name == "STACK_GLOBAL":
# module and operator are on the stack
module, operator = stack[-2], stack[-1]
# Look up in unsafe globals dictionary
for severity, modules in unsafe_globals.items():
if module in modules:
ops = modules[module]
if ops == "*" or operator in ops:
report_issue(module, operator, severity)
TensorFlow Operation Detection
# Pseudo-code for TF operation detection
saved_model = parse_protobuf(pb_file)
for graph in saved_model.meta_graphs:
for node in graph.graph_def.node:
if node.op in unsafe_tf_operators:
severity = unsafe_tf_operators[node.op]
report_issue("tensorflow", node.op, severity)
Keras Lambda Detection
# Pseudo-code for Keras Lambda detection
config = json.loads(model_config_string)
for layer in walk_layers(config):
if layer["class_name"] in unsafe_keras_operators:
severity = unsafe_keras_operators[layer["class_name"]]
report_issue("keras", layer["class_name"], severity)
The output of detection is an OperatorIssueDetails record containing:
- module: The Python module containing the unsafe operation
- operator: The specific function or class name
- severity: Classification from the unsafe globals dictionary
- source: The file path where the operator was found
- scanner: Which scanner detected it (for provenance)