Implementation:Duckdb Duckdb Httplib

Knowledge Sources	Duckdb_Duckdb cpp-httplib
Domains	Networking, HTTP, Third_Party
Last Updated	2026-02-07 12:00 GMT

Overview

cpp-httplib (v0.14.3) is a single-header C++ HTTP/HTTPS client and server library used by DuckDB for HTTP-based data access and extension downloading.

Description

DuckDB vendors the cpp-httplib library by Yuji Hirose as third_party/httplib/httplib.hpp. The library provides both Server and Client classes within the duckdb_httplib namespace (or duckdb_httplib_openssl when built with OpenSSL support). DuckDB's vendored copy replaces std::regex with RE2 for improved performance and security. The library supports:

• HTTP methods: GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS
• Content handling: multipart form data, chunked transfer encoding, content providers, and streaming receivers
• Connection management: keep-alive, configurable timeouts (connection, read, write, idle), and automatic redirect following
• TLS support: optional OpenSSL integration via the CPPHTTPLIB_OPENSSL_SUPPORT macro

Key configuration defaults include a 300-second connection timeout, 5-second read/write timeouts, 5-second keep-alive timeout with a maximum of 5 keep-alive requests, 8192-byte URI and header length limits, and a 20-redirect maximum.

Usage

DuckDB uses the duckdb_httplib::Client class for downloading extensions from remote repositories, fetching data from HTTP/HTTPS sources (e.g., for read_csv and read_parquet with HTTP URLs), and communicating with cloud storage APIs. The duckdb_httplib::Server class may be used in testing scenarios and for the optional HTTP API server interface.

Code Reference

Source Location

• Repository: Duckdb_Duckdb
• Files:
  • third_party/httplib/httplib.hpp

Signature

namespace duckdb_httplib {

// --- Request / Response structs ---

struct Request {
    std::string method;
    std::string path;
    Headers headers;
    std::string body;
    std::string remote_addr;
    int remote_port;
    Params params;
    MultipartFormDataMap files;

    bool has_header(const std::string &key) const;
    std::string get_header_value(const std::string &key, size_t id = 0) const;
    uint64_t get_header_value_u64(const std::string &key, size_t id = 0) const;
    void set_header(const std::string &key, const std::string &val);
    bool has_param(const std::string &key) const;
    std::string get_param_value(const std::string &key, size_t id = 0) const;
    bool is_multipart_form_data() const;
    bool has_file(const std::string &key) const;
    MultipartFormData get_file_value(const std::string &key) const;
};

struct Response {
    std::string version;
    int status;
    std::string reason;
    Headers headers;
    std::string body;
    std::string location;

    bool has_header(const std::string &key) const;
    std::string get_header_value(const std::string &key, size_t id = 0) const;
    void set_header(const std::string &key, const std::string &val);
    void set_redirect(const std::string &url, int status = 302);
    void set_content(const char *s, size_t n, const std::string &content_type);
    void set_content(const std::string &s, const std::string &content_type);
    void set_content_provider(size_t length, const std::string &content_type,
                              ContentProvider provider,
                              ContentProviderResourceReleaser releaser = nullptr);
    void set_chunked_content_provider(const std::string &content_type,
                                      ContentProviderWithoutLength provider,
                                      ContentProviderResourceReleaser releaser = nullptr);
};

// --- Server class ---

class Server {
public:
    using Handler = std::function<void(const Request &, Response &)>;
    using HandlerWithResponse = std::function<HandlerResponse(const Request &, Response &)>;
    using HandlerWithContentReader = std::function<void(
        const Request &, Response &, const ContentReader &)>;

    Server &Get(const std::string &pattern, Handler handler);
    Server &Post(const std::string &pattern, Handler handler);
    Server &Put(const std::string &pattern, Handler handler);
    Server &Patch(const std::string &pattern, Handler handler);
    Server &Delete(const std::string &pattern, Handler handler);
    Server &Options(const std::string &pattern, Handler handler);

    bool listen(const std::string &host, int port, int socket_flags = 0);
    bool is_running() const;
    void stop();

    Server &set_read_timeout(time_t sec, time_t usec = 0);
    Server &set_write_timeout(time_t sec, time_t usec = 0);
    Server &set_keep_alive_max_count(size_t count);
    Server &set_payload_max_length(size_t length);
    Server &set_error_handler(Handler handler);
    Server &set_logger(Logger logger);
};

// --- Client class ---

class Client {
public:
    explicit Client(const std::string &scheme_host_port);
    explicit Client(const std::string &host, int port);
    explicit Client(const std::string &scheme_host_port,
                    const std::string &client_cert_path,
                    const std::string &client_key_path);

    bool is_valid() const;

    Result Get(const std::string &path);
    Result Get(const std::string &path, const Headers &headers);
    Result Get(const std::string &path, ContentReceiver content_receiver);
    Result Get(const std::string &path, const Headers &headers,
               ContentReceiver content_receiver, Progress progress);
    Result Get(const std::string &path, const Params &params,
               const Headers &headers, Progress progress = nullptr);

    Result Head(const std::string &path);
    Result Head(const std::string &path, const Headers &headers);

    Result Post(const std::string &path);
    Result Post(const std::string &path, const std::string &body,
                const std::string &content_type);
    Result Post(const std::string &path, const Headers &headers,
                const std::string &body, const std::string &content_type);
    Result Post(const std::string &path, const Params &params);
    Result Post(const std::string &path, const MultipartFormDataItems &items);

    Result Put(const std::string &path, const std::string &body,
               const std::string &content_type);
    Result Delete(const std::string &path);

    void set_read_timeout(time_t sec, time_t usec = 0);
    void set_write_timeout(time_t sec, time_t usec = 0);
    void set_connection_timeout(time_t sec, time_t usec = 0);
    void set_keep_alive(bool on);
    void set_follow_location(bool on);
    void set_default_headers(Headers headers);
    void set_bearer_token_auth(const std::string &token);
    void set_basic_auth(const std::string &username, const std::string &password);
    void set_proxy(const std::string &host, int port);
};

} // namespace duckdb_httplib

Import

#include "httplib.hpp"

I/O Contract

Inputs

Name	Type	Required	Description
host	std::string	Yes	Hostname or scheme+host+port string (e.g., "https://api.example.com") for both Server and Client
port	int	No	Port number; can be embedded in the scheme_host_port string for Client
path	std::string	Yes	The URL path for HTTP requests (e.g., "/api/v1/data")
headers	Headers	No	HTTP headers as a multimap of string pairs
body	std::string	No	Request body for POST/PUT methods
content_type	std::string	No	MIME type of the request body (e.g., "application/json")
params	Params	No	URL query parameters as a multimap of string pairs
content_receiver	ContentReceiver	No	Callback function for streaming response body data
progress	Progress	No	Callback function for tracking download progress

Outputs

Name	Type	Description
Result	Result	Wrapper around a shared pointer to Response; truthy if the request succeeded
Response.status	int	HTTP status code (e.g., 200, 404, 500)
Response.headers	Headers	Response HTTP headers
Response.body	std::string	Response body content
Response.location	std::string	Redirect target URL (for 3xx responses)
Server::listen return	bool	Returns true if the server started successfully

Usage Examples

#include "httplib.hpp"

// --- Client: Simple GET request ---
duckdb_httplib::Client cli("https://example.com");
cli.set_connection_timeout(10, 0);
cli.set_read_timeout(30, 0);
cli.set_follow_location(true);

auto res = cli.Get("/api/data");
if (res && res->status == 200) {
    std::string data = res->body;
}

// --- Client: POST with JSON body ---
auto post_res = cli.Post("/api/submit",
    "{\"key\": \"value\"}", "application/json");

// --- Client: Streaming download with progress ---
cli.Get("/large-file.parquet",
    [](const char *data, size_t len) -> bool {
        // process chunk
        return true;  // return false to cancel
    },
    [](uint64_t current, uint64_t total) -> bool {
        // progress callback
        return true;  // return false to cancel
    }
);

// --- Server: Basic route handling ---
duckdb_httplib::Server svr;
svr.Get("/health", [](const duckdb_httplib::Request &req,
                       duckdb_httplib::Response &res) {
    res.set_content("OK", "text/plain");
});
svr.listen("0.0.0.0", 8080);

Related Pages

• Principle:Duckdb_Duckdb_HTTP_Communication