Workflow:Microsoft Onnxruntime Python Inference Pipeline
| Knowledge Sources | |
|---|---|
| Domains | ML_Inference, Model_Deployment |
| Last Updated | 2026-02-10 04:30 GMT |
Overview
End-to-end process for loading a pre-trained ONNX model and running inference predictions using the ONNX Runtime Python API.
Description
This workflow covers the standard procedure for performing inference with ONNX Runtime in Python. It begins with loading an ONNX model file into an InferenceSession, inspecting the model's expected input and output metadata, preparing input data as NumPy arrays conforming to the model's schema, executing inference, and extracting prediction results. The workflow supports configuring execution providers (CPU, CUDA, TensorRT, etc.) and session options for performance tuning such as graph optimization level, thread pool sizing, and profiling.
Usage
Execute this workflow when you have a pre-trained model in ONNX format and need to run inference predictions in a Python application. This applies whether the model was originally trained in PyTorch, TensorFlow, scikit-learn, or any other framework that exports to ONNX. The workflow is suitable for both single predictions and batch inference scenarios.
Execution Steps
Step 1: Configure Session Options
Set up session-level configuration to control runtime behavior. This includes setting the graph optimization level (disable, basic, extended, or full), configuring intra-op and inter-op thread pool sizes for CPU parallelism, and optionally enabling profiling to capture per-operator timing data.
Key considerations:
- Graph optimization levels affect startup time vs inference speed tradeoff
- Thread pool sizing should match available CPU cores
- Profiling adds overhead and should only be enabled during benchmarking
Step 2: Create Inference Session
Initialize an InferenceSession by loading the ONNX model file from disk or from a byte buffer. Specify the desired execution providers in priority order (e.g., CUDA first, CPU as fallback). The session will attempt to place operators on the highest-priority available provider and fall back to lower-priority ones for unsupported operations.
Key considerations:
- Provider order matters: list preferred accelerators first
- Session creation involves model loading, graph optimization, and memory planning
- The session object is thread-safe for concurrent inference calls
Step 3: Inspect Model Metadata
Query the session for input and output metadata to understand the model's interface. Each input and output has a name, element type, and shape. Use this information to prepare correctly shaped and typed input data.
What happens:
- Retrieve input names, types, and shapes via session input metadata
- Retrieve output names, types, and shapes via session output metadata
- Identify dynamic dimensions (marked as None or symbolic names)
Step 4: Prepare Input Data
Construct input tensors as NumPy arrays matching the model's expected input schema. Each input must have the correct element type (float32, int64, etc.) and shape. Package inputs into a dictionary mapping input names to their corresponding NumPy arrays.
Key considerations:
- Data types must exactly match the model schema
- Dynamic dimensions accept any valid size at runtime
- Batch dimension is typically the first axis
Step 5: Execute Inference
Call the session's run method with the prepared input feed dictionary. Optionally specify which output names to retrieve (or None to get all outputs). The runtime executes the optimized computation graph across the configured execution providers.
Key considerations:
- Specify output names to avoid computing unnecessary outputs
- RunOptions can control logging verbosity and timeout
- The run call is synchronous and returns results directly
Step 6: Process Results
Extract prediction results from the returned output list. Each output is a NumPy array corresponding to the requested output names. Post-process results as needed for the application (e.g., argmax for classification, denormalization for regression).
Key considerations:
- Output order matches the requested output names order
- Results are NumPy arrays ready for further processing
- For classification, apply argmax or softmax as appropriate