Workflow:Tensorflow Serving Kubernetes Deployment
| Knowledge Sources | |
|---|---|
| Domains | ML_Ops, Model_Serving, Infrastructure |
| Last Updated | 2026-02-13 17:00 GMT |
Overview
End-to-end process for deploying TensorFlow Serving as a scalable, load-balanced service on Kubernetes, from building a model-embedded Docker image to creating Deployments and Services for production inference.
Description
This workflow covers deploying TensorFlow Serving in a Kubernetes cluster for production-grade model serving. It starts with packaging a trained SavedModel into a custom Docker image, pushing it to a container registry, and then creating Kubernetes Deployment and Service resources to run replicated model servers behind a load balancer. The deployment exposes gRPC and/or REST endpoints for client inference and supports horizontal scaling.
Usage
Execute this workflow when you need to serve TensorFlow models at scale in a production environment with high availability, automatic scaling, and load balancing. This is the standard deployment pattern for organizations running Kubernetes-based infrastructure.
Execution Steps
Step 1: Prepare the SavedModel
Export or download the trained TensorFlow model in SavedModel format. Verify the model directory structure contains saved_model.pb and a variables/ directory. Place the model in a versioned subdirectory following TensorFlow Serving conventions.
Key considerations:
- The model must be in SavedModel format (not checkpoint or frozen graph)
- Version the model directory with a numeric subdirectory name (e.g., /model/123/)
- Verify the model loads correctly in a local Docker test before building the deployment image
- Ensure the model has proper SignatureDefs for the intended inference API
Step 2: Build a Custom Docker Image
Create a Docker image with the model baked in by running a base TensorFlow Serving container, copying the model into its models directory, and committing the container as a new image. This produces a self-contained image that requires no external model storage at runtime.
Key considerations:
- Start from the official tensorflow/serving base image (CPU or GPU variant)
- Copy the model to /models/{model_name} inside the container
- Set the MODEL_NAME environment variable via docker commit --change
- Tag the image for your target container registry
Step 3: Push to Container Registry
Tag the custom Docker image for the target container registry (e.g., Google Container Registry, Amazon ECR, Docker Hub) and push it. This makes the image available for Kubernetes to pull when creating pods.
Key considerations:
- Configure authentication for the target registry (e.g., gcloud auth configure-docker)
- Tag with the registry prefix (e.g., gcr.io/project/image_name)
- Consider using image tags or digests for version tracking
- Verify the push completes successfully before creating Kubernetes resources
Step 4: Create Kubernetes Cluster
Provision a Kubernetes cluster with sufficient resources to run the model serving workload. Configure kubectl credentials to interact with the cluster. Ensure the cluster has access to pull images from the container registry.
Key considerations:
- Size nodes based on model memory requirements and expected concurrency
- For GPU models, provision nodes with GPU accelerators and install device plugins
- Configure image pull secrets if using a private container registry
- Consider node pools with different machine types for cost optimization
Step 5: Deploy with Kubernetes Resources
Create a Kubernetes Deployment to manage replicated model server pods and a Service with a LoadBalancer to expose the inference endpoints externally. The Deployment specifies the container image, port mappings, resource limits, and replica count.
Key considerations:
- The Deployment manages pod replicas and handles rolling updates
- Expose port 8500 for gRPC and/or port 8501 for REST via the Service
- Set resource requests and limits based on model memory and CPU requirements
- Use readiness probes to ensure traffic is only sent to fully-initialized pods
Step 6: Query the Deployed Service
Connect clients to the external IP address assigned by the LoadBalancer Service. Send inference requests via gRPC or REST to validate the deployment. Monitor pod status, service endpoints, and inference latency.
Key considerations:
- Wait for the LoadBalancer to be provisioned and assigned an external IP
- The external IP is listed in kubectl describe service under LoadBalancer Ingress
- Test with representative requests to verify end-to-end inference correctness
- Use kubectl get pods and kubectl logs to troubleshoot any issues