Principle:Ray project Ray Request Routing And Handling
| Knowledge Sources | |
|---|---|
| Domains | Model_Serving, Load_Balancing |
| Last Updated | 2026-02-13 17:00 GMT |
Overview
A mechanism for routing client requests to available deployment replicas using random selection and returning results via distributed futures.
Description
Request Routing and Handling is the runtime mechanism that dispatches requests from a DeploymentHandle to one of the available replica actors. The router maintains a ReplicaSet of available replicas (updated via long-polling from the controller) and assigns each request to a randomly selected replica. The replica executes the handler method via reflection and returns the result through Ray's object store.
Usage
Use the deployment handle's .remote() method to send requests after deployment. Optionally use .method() to invoke a specific method on the deployment class.
Theoretical Basis
Request routing implements random load balancing across replicas. Each request is independently assigned to a random replica, providing:
- Simple implementation
- Good load distribution for uniform workloads
- No coordination overhead