Implementation:ClickHouse ClickHouse Poco HTTPClientSession Impl

Source File	base/poco/Net/src/HTTPClientSession.cpp
Lines of Code	613
Principle	Principle:ClickHouse_ClickHouse_HTTP_Client_Communication
Domain	Networking, HTTP
Language	C++
Last Updated	2026-02-08 00:00 GMT

Overview

Implementation of HTTP client session management, providing connection pooling, keepalive support, proxy handling, and request/response transmission for HTTP client operations.

Purpose

Manages the lifecycle of HTTP client connections including connection establishment, request transmission, response reception, keepalive management, and proxy support.

Key Classes

HTTPClientSession: Main HTTP client session class extending `HTTPSession`
ProxyConfig: Proxy configuration structure

Core Functionality

Session Management

// Send HTTP request, return stream for writing body
std::ostream& sendRequest(HTTPRequest& request, uint64_t* connect_time = nullptr,
                         uint64_t* first_byte_time = nullptr);

// Flush request stream
void flushRequest();

// Receive HTTP response, return stream for reading body
std::istream& receiveResponse(HTTPResponse& response);

// Peek at response without consuming it (for 100 Continue)
bool peekResponse(HTTPResponse& response);

// Close and reset session
void reset();

Connection Management

// Set target host and port
void setHost(const std::string& host);
void setPort(Poco::UInt16 port);

// Check if session uses secure connection
bool secure() const;

// Reconnect to server
void reconnect(uint64_t* connect_time = nullptr);

// Assign socket from another session
void assign(HTTPClientSession& session);

Proxy Configuration

struct ProxyConfig {
    std::string host;
    Poco::UInt16 port;
    std::string protocol;      // "http" or "https"
    std::string username;
    std::string password;
    std::string nonProxyHosts; // Regex pattern
    bool tunnel;               // Use CONNECT tunneling
    std::string originalRequestProtocol;
};

void setProxy(const std::string& host, Poco::UInt16 port,
              const std::string& protocol = "http", bool tunnel = false);
void setProxyCredentials(const std::string& username, const std::string& password);

Keep-Alive Management

// Set keep-alive timeout
void setKeepAliveTimeout(const Poco::Timespan& timeout);

// Set maximum requests per connection
void setKeepAliveMaxRequests(int max_requests);

// Check if keep-alive expired
bool isKeepAliveExpired(double reliability = 0.9) const;

// Check if reconnection needed
bool mustReconnect() const;

Implementation Details

Request Transmission

Handles various request formats:

std::ostream& sendRequest(HTTPRequest& request, ...) {
    // Handle reconnection if needed
    if (connected() && !keepAlive || mustReconnect()) {
        close();
    }

    // Connect if not connected
    if (!connected()) {
        reconnect(connect_time);
    }

    // Set headers
    if (!request.has(HOST)) {
        request.setHost(_host, _port);
    }
    if (keepAlive) {
        request.setKeepAlive(true);
        request.setKeepAliveTimeout(timeout, max_requests);
    }

    // Add proxy prefix if needed
    if (using_proxy && !bypassProxy()) {
        request.setURI(proxyRequestPrefix() + request.getURI());
        proxyAuthenticate(request);
    }

    // Write request based on encoding
    if (chunked) {
        // Use HTTPChunkedOutputStream
    } else if (hasContentLength) {
        // Use HTTPFixedLengthOutputStream
    } else {
        // Use HTTPOutputStream
    }

    return *_pRequestStream;
}

Response Reception

std::istream& receiveResponse(HTTPResponse& response) {
    flushRequest();

    // Read response, handle 100 Continue
    do {
        response.clear();
        HTTPHeaderInputStream his(*this);
        response.read(his);
    } while (response.getStatus() == HTTP_CONTINUE);

    // Update keep-alive parameters from server
    if (response.hasKeepAlive()) {
        _keepAliveTimeout = min(_keepAliveTimeout, response.getKeepAliveTimeout());
        _keepAliveMaxRequests = min(_keepAliveMaxRequests, response.getKeepAliveMaxRequests());
    }

    // Create appropriate response stream
    if (no_content_expected) {
        _pResponseStream = new HTTPFixedLengthInputStream(*this, 0);
    } else if (chunked) {
        _pResponseStream = new HTTPChunkedInputStream(*this);
    } else if (hasContentLength) {
        _pResponseStream = new HTTPFixedLengthInputStream(*this, contentLength);
    } else {
        _pResponseStream = new HTTPInputStream(*this);
    }

    return *_pResponseStream;
}

Proxy Support

Three proxy modes:

1. Direct proxy: Request line includes full URL 2. CONNECT tunneling: Establish tunnel, then normal requests 3. No proxy: Direct connection to server

std::string proxyRequestPrefix() const {
    return _proxyConfig.originalRequestProtocol + "://" + _host + ":" + _port;
}

bool bypassProxy() const {
    return RegularExpression::match(_host, _proxyConfig.nonProxyHosts);
}

StreamSocket proxyConnect() {
    // Create connection to proxy
    // Send CONNECT request
    // Receive 200 OK
    // Return connected socket
}

void proxyAuthenticate(HTTPRequest& request) {
    if (!_proxyConfig.username.empty()) {
        HTTPBasicCredentials creds(_proxyConfig.username, _proxyConfig.password);
        creds.proxyAuthenticate(request);
    }
}

Keep-Alive Expiration

bool isKeepAliveExpired(double reliability) const {
    Poco::Timestamp now;
    return (now - _lastRequest) >= Timespan(reliability * _keepAliveTimeout.totalMicroseconds())
        || _keepAliveCurrentRequest > _keepAliveMaxRequests;
}

bool mustReconnect() const {
    if (_mustReconnect)
        return true;
    return isKeepAliveExpired(_defaultKeepAliveReliabilityLevel);  // 0.9
}

Uses 90% of server's timeout to avoid races.

Connection Retry

Automatic retry on first write failure:

int write(const char* buffer, std::streamsize length) {
    try {
        return HTTPSession::write(buffer, length);
    } catch (Poco::Exception&) {
        if (_reconnect) {
            close();
            reconnect();
            int rc = HTTPSession::write(buffer, length);
            clearException();
            return rc;
        } else {
            throw;
        }
    }
}

Enables transparent recovery from connection drops.

Global Proxy Configuration

static ProxyConfig _globalProxyConfig;

static void setGlobalProxyConfig(const ProxyConfig& config) {
    _globalProxyConfig = config;
}

Default proxy settings for all new sessions.

Usage Examples

Basic Request/Response

HTTPClientSession session("www.example.com", 80);
HTTPRequest request(HTTPRequest::HTTP_GET, "/index.html");

session.sendRequest(request);

HTTPResponse response;
std::istream& rs = session.receiveResponse(response);

std::string body;
std::getline(rs, body, '\0');

With Proxy

session.setProxy("proxy.example.com", 8080);
session.setProxyCredentials("user", "password");

// Requests automatically routed through proxy
session.sendRequest(request);

Keep-Alive

session.setKeepAlive(true);
session.setKeepAliveTimeout(Poco::Timespan(30, 0));  // 30 seconds
session.setKeepAliveMaxRequests(100);

// Multiple requests reuse connection
for (int i = 0; i < 10; i++) {
    session.sendRequest(request);
    session.receiveResponse(response);
}

Connection Timing

uint64_t connect_time = 0;
uint64_t first_byte_time = 0;

session.sendRequest(request, &connect_time, &first_byte_time);
// connect_time: microseconds to establish connection
// first_byte_time: microseconds to send first byte

Dependencies

`Poco::Net::HTTPSession`: Base session class
`Poco::Net::HTTPRequest`: Request representation
`Poco::Net::HTTPResponse`: Response representation
`Poco::Net::StreamSocket`: TCP socket wrapper
`Poco::Net::SocketAddress`: Address representation
Various HTTP streams: Chunked, FixedLength, Header

Performance Considerations

Keep-alive reduces connection overhead
Connection pooling via session reuse
Automatic retry on transient failures
Configurable timeouts
Efficient stream handling

Testing Considerations

Test with and without keep-alive
Verify proxy functionality
Test connection retry logic
Check timeout behavior
Test with chunked and fixed-length encoding
Verify 100 Continue handling
Test concurrent sessions

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment