Principle:Tencent Ncnn Python Bindings Build
| Knowledge Sources | |
|---|---|
| Domains | Build Systems, Language Interoperability |
| Last Updated | 2026-02-09 19:00 GMT |
Overview
A build system pattern for packaging C++ libraries as installable Python packages by orchestrating CMake for native compilation, pybind11 for binding generation, and setuptools for Python packaging and distribution.
Description
Python bindings build is a build system architecture that bridges the gap between high-performance C++ libraries and the Python ecosystem. The goal is to produce a Python package (installable via pip) that exposes C++ functionality through a natural Python API, while handling the substantial complexity of cross-platform native compilation.
The system coordinates three distinct build technologies. CMake serves as the native build system, responsible for compiling the C++ library itself, managing platform-specific compiler flags, locating system dependencies (such as Vulkan SDK or OpenMP), and producing shared library artifacts. CMake handles the complexity of supporting multiple operating systems, architectures, and compilers through its generator-based abstraction.
pybind11 is a header-only C++ library that generates Python extension modules. It provides macros and template machinery to expose C++ classes, functions, and enumerations to Python with automatic type conversion. pybind11 handles the translation between Python objects and C++ types, reference counting, exception propagation, and the Python buffer protocol for zero-copy array interoperability with NumPy.
setuptools (or its modern equivalents) manages the Python packaging layer. It defines package metadata, dependencies, version information, and the build process invocation. A custom build extension class overrides the standard setuptools build step to invoke CMake with appropriate arguments, ensuring that the native compilation happens as part of the standard pip install workflow.
Usage
This principle applies when exposing C++ libraries to Python users:
- Machine learning frameworks: Wrapping inference engines for use in Python scripts and notebooks.
- Scientific computing: Providing Python interfaces to high-performance numerical libraries.
- Wheel distribution: Building platform-specific binary wheels for PyPI distribution.
- Development workflows: Allowing
pip install -e .for editable development installs.
Theoretical Basis
The build orchestration flow in pseudo-code:
// setup.py entry point
class CMakeBuild(setuptools.build_ext):
function build_extension(ext):
// Determine output directory for the .so/.pyd file
ext_dir = get_extension_output_dir(ext)
// Assemble CMake arguments
cmake_args = [
"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=" + ext_dir,
"-DPYTHON_EXECUTABLE=" + sys.executable,
"-DCMAKE_BUILD_TYPE=Release",
"-DNCNN_PYTHON=ON",
]
// Platform-specific adjustments
if platform == "windows":
cmake_args += ["-A", architecture] // x64, ARM64
if platform == "macos":
cmake_args += ["-DCMAKE_OSX_ARCHITECTURES=arm64;x86_64"]
// Configure step
run("cmake", source_dir, *cmake_args, cwd=build_dir)
// Build step
run("cmake", "--build", build_dir, "--config", "Release", "-j", num_cpus)
setup(
name = "ncnn",
version = detect_version(),
ext_modules = [CMakeExtension("ncnn")],
cmdclass = {"build_ext": CMakeBuild},
install_requires = ["numpy", "pybind11"],
)
The pybind11 module definition on the C++ side:
// C++ binding definition
PYBIND11_MODULE(ncnn, m) {
// Expose the Net class
py::class_<Net>(m, "Net")
.def(py::init<>())
.def("load_param", &Net::load_param)
.def("load_model", &Net::load_model)
.def("create_extractor", &Net::create_extractor);
// Expose the Mat class with buffer protocol for NumPy interop
py::class_<Mat>(m, "Mat", py::buffer_protocol())
.def_buffer([](Mat &m) -> py::buffer_info { ... })
.def_property_readonly("shape", ...);
}