Principle:Pytorch Serve Non PyTorch Model Serving

Field	Value
source	Pytorch_Serve
domains	ML_Ops, Inference
last_updated	2026-02-13 18:52 GMT

Overview

Non-PyTorch Model Serving is the principle of deploying machine learning models built with non-PyTorch frameworks (such as XGBoost, scikit-learn, or LightGBM) through TorchServe's handler abstraction, leveraging its model management, scaling, and API infrastructure without requiring PyTorch model serialization.

Description

This principle addresses what it means to serve non-PyTorch models within the TorchServe ecosystem. TorchServe provides a mature serving infrastructure -- including REST/gRPC endpoints, model versioning, batching, and worker scaling -- that is valuable beyond PyTorch models alone. By implementing a custom handler, any model that can be loaded in Python can be served through TorchServe.

The key aspects of non-PyTorch model serving include:

• Custom handler abstraction -- The BaseHandler interface defines initialize(), preprocess(), inference(), and postprocess() methods. A custom handler overrides these methods to load and execute any ML framework's model.
• Model artifact packaging -- Non-PyTorch models are serialized using their native formats (e.g., pickle for scikit-learn, JSON/binary for XGBoost) and packaged into a .mar (Model Archive) file alongside the handler code.
• Framework-agnostic inference -- The handler loads the model using the appropriate library (e.g., xgboost.Booster, joblib.load) and invokes its prediction API directly, bypassing PyTorch's tensor operations entirely.
• Unified serving API -- Clients interact with the same REST/gRPC interface regardless of the underlying model framework.

import xgboost as xgb
import numpy as np
import os
from ts.torch_handler.base_handler import BaseHandler

class XGBoostIrisHandler(BaseHandler):
    def initialize(self, context):
        properties = context.system_properties
        model_dir = properties.get("model_dir")
        self.model = xgb.Booster()
        self.model.load_model(os.path.join(model_dir, "iris_model.json"))

    def preprocess(self, data):
        inputs = []
        for row in data:
            values = row.get("body")
            inputs.append(values)
        return xgb.DMatrix(np.array(inputs))

    def inference(self, data):
        predictions = self.model.predict(data)
        return predictions.tolist()

    def postprocess(self, inference_output):
        return [{"predictions": inference_output}]

Usage

Apply this principle when:

• The organization has standardized on TorchServe as its model serving platform but needs to deploy models from other ML frameworks.
• XGBoost, scikit-learn, LightGBM, or other non-PyTorch models must be served with production-grade infrastructure (health checks, metrics, logging, batching).
• A unified API surface is desired across all deployed models regardless of their training framework.
• Migration from ad-hoc serving solutions (Flask, FastAPI wrappers) to a managed model server is underway.
• The model does not benefit from GPU acceleration and runs efficiently on CPU, making PyTorch conversion unnecessary.

Theoretical Basis

Non-PyTorch model serving leverages the handler pattern, an architectural design where a standardized interface decouples the serving infrastructure from model-specific logic.

The TorchServe handler lifecycle follows a four-stage pipeline:

1. Initialize -- Load the model artifact from disk into memory. For non-PyTorch models, this uses the native framework's deserialization (e.g., xgb.Booster.load_model() for XGBoost, joblib.load() for scikit-learn). This stage runs once when the worker process starts.
2. Preprocess -- Transform raw HTTP request data into the format expected by the model. This may involve JSON parsing, feature extraction, type conversion, and construction of framework-specific data structures (e.g., xgb.DMatrix).
3. Inference -- Execute the model's prediction method. For tree-based models like XGBoost, this traverses the ensemble of decision trees and aggregates their outputs. The computational characteristics differ fundamentally from neural network inference -- tree traversal is branching and memory-bound rather than compute-bound.
4. Postprocess -- Transform model outputs into the HTTP response format. This includes converting numpy arrays to JSON-serializable types and applying any output transformations (e.g., argmax for classification, label mapping).

For XGBoost specifically, inference uses gradient boosted decision trees:

• An ensemble of T trees produces predictions: y_hat = sum(f_t(x)) for t = 1..T.
• Each tree f_t partitions the feature space via learned split conditions.
• For classification, the raw scores are passed through a softmax function to produce class probabilities.

The Model Archive (.mar) format bundles all artifacts into a deployable unit:

• The serialized model file (e.g., iris_model.json).
• The custom handler Python file.
• A manifest specifying the handler entry point and model metadata.
• Any additional dependency files or configuration.

Related Pages

• Implementation:Pytorch_Serve_XGBoost_Iris_Handler