Environment:Apache Dolphinscheduler Netty Runtime

Knowledge Sources	Apache DolphinScheduler Netty Project
Domains	Infrastructure, Distributed_Systems
Last Updated	2026-02-10 10:00 GMT

Overview

Netty 4.1.53.Final network framework for RPC communication between Master, Worker, and API Server components with Epoll/NIO auto-detection.

Description

DolphinScheduler uses Netty 4.1.53.Final as the transport layer for its custom RPC framework. The Netty server handles Master-Worker communication including task dispatch, lifecycle events, and failover coordination. The framework automatically detects Linux Epoll availability for optimized I/O and falls back to Java NIO on other platforms. Thread pools are dynamically sized based on available CPU cores.

Usage

Use this environment for Master Server and Worker Server components that participate in RPC communication. The Netty runtime is automatically configured when these services start. Network tuning (buffer sizes, backlog, keep-alive) affects RPC performance and reliability.

System Requirements

Category	Requirement	Notes
OS	Linux (recommended for Epoll)	Falls back to NIO on macOS/Windows
CPU	2+ cores	Worker threads = CPUs * 2
Network	Low-latency inter-node connectivity	TCP_NODELAY enabled by default
Kernel	Linux 2.6+ for Epoll support	Epoll.isAvailable() auto-detects

Dependencies

Java Dependencies (managed by Maven BOM)

`netty-all` = 4.1.53.Final
`netty-transport-native-epoll` (Linux Epoll transport)
`protostuff-core` = 1.7.2 (RPC serialization)
`protostuff-runtime` = 1.7.2

Credentials

No credentials required. Netty RPC communication is currently unencrypted within the cluster network.

Quick Install

# Netty is bundled as a Maven dependency - no separate installation needed.
# Verify Epoll availability at runtime:
java -cp "lib/*" -Dio.netty.transport.noNative=false \
  io.netty.channel.epoll.Epoll

Code Evidence

Epoll vs NIO auto-detection from `NettyRemotingServer.java:80-86`:

if (Epoll.isAvailable()) {
    this.bossGroup = new EpollEventLoopGroup(1, bossThreadFactory);
    this.workGroup = new EpollEventLoopGroup(serverConfig.getWorkerThread(), workerThreadFactory);
} else {
    this.bossGroup = new NioEventLoopGroup(1, bossThreadFactory);
    this.workGroup = new NioEventLoopGroup(serverConfig.getWorkerThread(), workerThreadFactory);
}

Server socket configuration from `NettyServerConfig.java:38-75`:

private int soBacklog = 1024;
private boolean tcpNoDelay = true;
private boolean soKeepalive = true;
private int sendBufferSize = 65535;
private int receiveBufferSize = 65535;
private int workerThread = Runtime.getRuntime().availableProcessors() * 2;
private long connectionIdleTime = Duration.ofSeconds(60).toMillis();

Server bootstrap channel options from `NettyRemotingServer.java:91-99`:

ServerBootstrap serverBootstrap = new ServerBootstrap()
    .group(this.bossGroup, this.workGroup)
    .channel(NettyUtils.getServerSocketChannelClass())
    .option(ChannelOption.SO_REUSEADDR, true)
    .option(ChannelOption.SO_BACKLOG, serverConfig.getSoBacklog())
    .childOption(ChannelOption.SO_KEEPALIVE, serverConfig.isSoKeepalive())
    .childOption(ChannelOption.TCP_NODELAY, serverConfig.isTcpNoDelay())
    .childOption(ChannelOption.SO_SNDBUF, serverConfig.getSendBufferSize())
    .childOption(ChannelOption.SO_RCVBUF, serverConfig.getReceiveBufferSize());

Common Errors

Error Message	Cause	Solution
`bind fail` at port X	Port already in use	Change `listenPort` or kill the conflicting process
`Too many open files`	File descriptor limit exceeded	Increase `ulimit -n` (recommended: 65535+)
`Connection refused`	Remote Netty server not running	Verify Master/Worker service is started
`Channel inactive`	Connection idle timeout (60s)	Check network stability; increase `connectionIdleTime` if needed

Compatibility Notes

Linux: Uses Epoll for optimal performance (O(1) event notification). Recommended for production.
macOS/Windows: Falls back to Java NIO (poll-based). Suitable for development only.
Netty version: Pinned to 4.1.53.Final. Upgrading requires testing RPC compatibility.
gRPC: DolphinScheduler also includes gRPC 1.41.0 and Protobuf 3.17.2 for some communication paths.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment