Implementation:Pytorch Serve TsUtils

Overview

Pytorch_Serve_TsUtils provides the core TorchServe lifecycle utility functions used by test harnesses and automation scripts. It wraps the torchserve CLI to start, stop, register, infer, and manage models and workflows in a cross-platform manner.

Source

Property	Value
File	ts_scripts/tsutils.py
Lines	192
Language	Python
Key Class	LogPipeTillTheEnd
Key Functions	start_torchserve, stop_torchserve, register_model, run_inference, unregister_model, generate_grpc_client_stubs, register_workflow, unregister_workflow, workflow_prediction

Key Class

LogPipeTillTheEnd (lines 31–40)

class LogPipeTillTheEnd(threading.Thread):
    ...

A background thread that continuously reads from a subprocess pipe and writes output to the log. This ensures that TorchServe's stdout and stderr are captured in real time without blocking the main thread.

Key Functions

start_torchserve(ncs, model_store, ...) (lines 43–96)

def start_torchserve(ncs, model_store, models=None, config_file=None, ...):
    ...

Starts a TorchServe instance by constructing and executing the appropriate CLI command. Parameters include:

Parameter	Description
ncs	No config snapshots flag
model_store	Path to the model store directory
models	Optional initial models to load at startup
config_file	Optional path to a TorchServe config properties file

The function builds a cross-platform command (handling differences between Linux/macOS and Windows), spawns the server as a subprocess, and attaches LogPipeTillTheEnd threads to capture output. It waits for the server to become ready before returning.

stop_torchserve() (lines 99–112)

def stop_torchserve():
    ...

Gracefully stops a running TorchServe instance using the torchserve --stop command. Handles platform-specific process termination and waits for the server to fully shut down.

register_model(model_name, protocol, host, port) (lines 116–131)

def register_model(model_name, protocol="http", host="localhost", port="8081"):
    ...

Registers a model with the TorchServe Management API. Constructs and sends the appropriate REST request to the management endpoint, supporting both HTTP and HTTPS protocols.

run_inference(model_name, file_name, ...) (lines 134–141)

def run_inference(model_name, file_name, ...):
    ...

Sends an inference request to the TorchServe Inference API for a registered model. Submits the input file as the request payload and returns the prediction response.

unregister_model() (lines 144–148)

def unregister_model():
    ...

Removes a previously registered model from TorchServe via the Management API.

generate_grpc_client_stubs() (lines 151–161)

def generate_grpc_client_stubs():
    ...

Generates Python gRPC client stubs from the TorchServe protobuf definitions. This enables tests and scripts to communicate with TorchServe over gRPC in addition to REST.

register_workflow() (lines 164–170)

def register_workflow():
    ...

Registers a workflow (a DAG of models) with TorchServe's Workflow Management API.

unregister_workflow() (lines 173–177)

def unregister_workflow():
    ...

Removes a registered workflow from TorchServe.

workflow_prediction() (lines 180–192)

def workflow_prediction():
    ...

Sends a prediction request through a registered workflow and returns the result.

Cross-Platform Support

Throughout the module, commands are constructed conditionally based on the operating system. On Windows, the script uses torchserve.exe and adjusts path separators accordingly, while on Linux and macOS it uses the standard shell commands. This ensures that test automation works uniformly across all supported platforms.

Relationship to Principles

This implementation directly supports the Pytorch_Serve_Server_Lifecycle principle. The functions in tsutils.py encapsulate the full server lifecycle -- starting, stopping, model registration, inference, and cleanup -- providing a programmatic interface to what would otherwise be manual CLI operations.

Metadata

Pytorch_Serve Pytorch_Serve_Server_Lifecycle 2026-02-13 18:52 GMT