Heuristic:PeterL1n BackgroundMattingV2 ONNX Patch Method Compatibility
| Knowledge Sources | |
|---|---|
| Domains | Computer_Vision, Deployment |
| Last Updated | 2026-02-09 02:00 GMT |
Overview
Use `roi_align` for cropping and `scatter_element` for replacing patches when exporting to ONNX; use `unfold`/`scatter_nd` for PyTorch and TorchScript.
Description
The refinement network's novel patch-based architecture crops and replaces image patches during inference. Different backends support different tensor operations, so the codebase provides multiple interchangeable methods for these operations. Choosing the wrong method for a given export target will cause export failures or runtime errors.
Usage
Use this heuristic when exporting models to ONNX or TorchScript, or when deploying to non-PyTorch inference backends. The default configuration is optimized for PyTorch; ONNX export requires explicit method overrides.
The Insight (Rule of Thumb)
- Action: Set `patch_crop_method` and `patch_replace_method` based on the target inference backend.
- Values:
- PyTorch / TorchScript: `unfold` (crop) + `scatter_nd` (replace) — best performance.
- ONNX: `roi_align` (crop) + `scatter_element` (replace) — best compatibility.
- Fallback: `gather` (crop) — pixel-by-pixel, slowest but most portable.
- Additional tip: If `sampling` mode is not supported by the inference backend, try `thresholding` mode instead.
- Trade-off: `unfold`/`scatter_nd` are fastest in PyTorch but may not export correctly to ONNX. `roi_align`/`scatter_element` are compatible with ONNX but may be slower.
Reasoning
The `unfold` operation is a native PyTorch tensor view that avoids memory copies, making it the fastest option. However, ONNX does not reliably support the `unfold` op across all backends. The `roi_align` operation from torchvision is well-supported in ONNX (opset 11+) and produces identical results. Similarly, `scatter_nd` uses advanced indexing that maps well to PyTorch but may fail in ONNX; `scatter_element` uses `Tensor.scatter_` which has broader ONNX support. The code comments explicitly document these compatibility trade-offs.
Code evidence from `model/refiner.py:205-211`:
if self.patch_crop_method == 'unfold':
# Use unfold. Best performance for PyTorch and TorchScript.
return x.permute(0, 2, 3, 1) \
.unfold(1, size + 2 * padding, size) \
.unfold(2, size + 2 * padding, size)[idx[0], idx[1], idx[2]]
elif self.patch_crop_method == 'roi_align':
# Use roi_align. Best compatibility for ONNX.
Code evidence from `model/refiner.py:244-253`:
if self.patch_replace_method == 'scatter_nd':
# Use scatter_nd. Best performance for PyTorch and TorchScript.
...
else:
# Use scatter_element. Best compatibility for ONNX.
Export defaults from `export_onnx.py:64-65`:
parser.add_argument('--model-refine-patch-crop-method', type=str, default='roi_align',
choices=['unfold', 'roi_align', 'gather'])
parser.add_argument('--model-refine-patch-replace-method', type=str, default='scatter_element',
choices=['scatter_nd', 'scatter_element'])