Workflow:Apache Dolphinscheduler RPC Service Communication
| Knowledge Sources | |
|---|---|
| Domains | Distributed_Systems, RPC_Framework, Networking |
| Last Updated | 2026-02-10 10:00 GMT |
Overview
End-to-end process for establishing and using inter-service RPC communication between DolphinScheduler components using the custom Netty-based framework with JDK dynamic proxies.
Description
This workflow describes how DolphinScheduler's distributed components (Master, Worker, Alert, API) communicate through a custom RPC framework built on Netty. The framework uses JDK dynamic proxies to provide a transparent service invocation model where callers interact with Java interfaces while the framework handles serialization, network transport, and response handling. The process covers server-side method registration, client proxy creation, synchronous/asynchronous invocation, and retry strategies.
Usage
Execute this workflow when you need to understand or extend the inter-service communication layer, add new RPC service methods, or troubleshoot communication issues between DolphinScheduler's distributed components (e.g., Master dispatching tasks to Workers, API server triggering workflows on Master).
Execution Steps
Step 1: Define Service Contract Interface
Create a Java interface defining the RPC service contract. Each method represents a remote operation that can be invoked across network boundaries. The interface is annotated with @RpcService and each method with @RpcMethod to enable automatic discovery and registration.
Key considerations:
- Interface should be in the extract-common or extract-{component} module for shared visibility
- Method parameters and return types must be serializable
- Request/response objects follow the transportor pattern (e.g., WorkflowManualTriggerRequest/Response)
- The interface acts as the single source of truth for the contract between services
Step 2: Implement Server Side Handler
Implement the service interface on the server side (e.g., in the Master or Worker module). The implementation contains the actual business logic. The framework's Spring-based method invoker discovery automatically detects implementations annotated with @RpcService and registers their methods with the NettyRemotingServer.
Key considerations:
- Server implementation is discovered via Spring component scanning
- JdkDynamicServerHandler routes incoming requests to the correct method invoker
- Each registered method is identified by its interface name and method signature
- Server starts NettyRemotingServer which binds to a configurable port
Step 3: Create Client Proxy
On the client side, use the Clients utility class to create a dynamic proxy for the service interface. The proxy transparently converts method calls into network requests. The fluent API pattern is: Clients.withService(IService.class).withHost(serverAddress).
Key considerations:
- Clients.withService() returns a builder that creates JDK dynamic proxies
- The proxy handles serialization of method arguments into Transporter messages
- A singleton NettyRemotingClient instance is shared across all proxies
- Host addresses are resolved from the cluster registry (ZooKeeper/etcd)
Step 4: Execute Synchronous RPC Call
Invoke the remote method through the proxy interface. The framework serializes the request, sends it over a Netty channel, and blocks waiting for the response. The Transporter protocol wraps the payload with headers for routing, and the encoder/decoder handle wire format conversion.
Key considerations:
- Synchronous calls use a CountDownLatch-based future mechanism
- The Transporter message format includes: header (method identifier, request ID) + body (serialized payload)
- Channel connections are cached and reused via getOrCreateChannel
- Heartbeat messages maintain channel liveness
Step 5: Handle Retry and Error Recovery
When an RPC call fails (network timeout, server unavailable), the retry strategy determines whether and how to retry. The framework supports configurable retry policies including retry count, backoff intervals, and circuit-breaking. Failed calls produce metrics for monitoring.
Key considerations:
- Retry strategies are configurable per-call or per-service
- Channel failures trigger automatic reconnection on next call
- Metrics are collected for call duration, success/failure rates
- Async invocation is also supported for fire-and-forget patterns