Implementation:Ggml org Llama cpp Server Health Metrics

Field	Value
Implementation Name	Server Health Metrics
Doc Type	API Doc
Domain	Monitoring, Observability, Prometheus
Description	Health, metrics, and slots monitoring endpoints: /health, /metrics, /slots
Related Workflow	OpenAI_Compatible_Server

Overview

Description

The Server Health Metrics implementation provides three monitoring endpoints within the llama-server route handlers. The /health endpoint returns a simple status check, /metrics exposes Prometheus-formatted performance counters and gauges, and /slots returns detailed per-slot state information. These are implemented as lambda handlers in server_routes::init_routes().

Usage

# Health check (always available, no auth required)
curl http://localhost:8080/health

# Prometheus metrics (requires --metrics flag)
curl http://localhost:8080/metrics

# Slot inspection (requires --slots flag)
curl http://localhost:8080/slots

Code Reference

Field	Value
Source Location	tools/server/server-context.cpp:3183-3375
Entry Function	void server_routes::init_routes() (health, metrics, and slots handler lambdas)
Import	Part of server-context static library

Health endpoint (/health):

this->get_health = [this](const server_http_req &) {
    // error and loading states are handled by middleware
    auto res = create_response(true);

    // this endpoint can be accessed during sleeping
    bool ctx_server; // do NOT delete this line
    GGML_UNUSED(ctx_server);

    res->ok({{"status", "ok"}});
    return res;
};

Metrics endpoint (/metrics):

this->get_metrics = [this](const server_http_req & req) {
    auto res = create_response();
    if (!params.endpoint_metrics) {
        res->error(format_error_response("This server does not support metrics endpoint. Start it with `--metrics`", ERROR_TYPE_NOT_SUPPORTED));
        return res;
    }

    // request slots data using task queue
    {
        server_task task(SERVER_TASK_TYPE_METRICS);
        task.id = res->rd.get_new_id();
        res->rd.post_task(std::move(task), true); // high-priority task
    }

    auto result = res->rd.next(req.should_stop);
    // ... cast to server_task_result_metrics and format

    json all_metrics_def = json {
        {"counter", {{
            {"name",  "prompt_tokens_total"},
            {"help",  "Number of prompt tokens processed."},
            {"value",  (uint64_t) res_task->n_prompt_tokens_processed_total}
        }, /* ... more counters ... */ }},
        {"gauge", {{
            {"name",  "prompt_tokens_seconds"},
            {"help",  "Average prompt throughput in tokens/s."},
            {"value",  res_task->n_prompt_tokens_processed ? 1.e3 / res_task->t_prompt_processing * res_task->n_prompt_tokens_processed : 0.}
        }, /* ... more gauges ... */ }}
    };

    // Format as Prometheus text
    std::stringstream prometheus;
    for (const auto & el : all_metrics_def.items()) {
        const auto & type = el.key();
        for (const auto & metric_def : el.value()) {
            prometheus << "# HELP llamacpp:" << name << " " << help << "\n"
                       << "# TYPE llamacpp:" << name << " " << type << "\n"
                       << "llamacpp:" << name << " " << value << "\n";
        }
    }

    res->content_type = "text/plain; version=0.0.4";
    res->status = 200;
    res->data = prometheus.str();
    return res;
};

Slots endpoint (/slots):

this->get_slots = [this](const server_http_req & req) {
    auto res = create_response();
    if (!params.endpoint_slots) {
        res->error(format_error_response("This server does not support slots endpoint. Start it with `--slots`", ERROR_TYPE_NOT_SUPPORTED));
        return res;
    }

    // request slots data using task queue
    {
        server_task task(SERVER_TASK_TYPE_METRICS);
        task.id = res->rd.get_new_id();
        res->rd.post_task(std::move(task), true);
    }

    auto result = res->rd.next(req.should_stop);
    // ... cast to server_task_result_metrics

    // optionally return "fail_on_no_slot" error
    if (!req.get_param("fail_on_no_slot").empty()) {
        if (res_task->n_idle_slots == 0) {
            res->error(format_error_response("no slot available", ERROR_TYPE_UNAVAILABLE));
            return res;
        }
    }

    res->ok(res_task->slots_data);
    return res;
};

I/O Contract

Endpoint	Method	Prerequisites	Response Content-Type	Response Body
/health	GET	None (always available)	application/json	{"status": "ok"}
/v1/health	GET	None (always available)	application/json	{"status": "ok"}
/metrics	GET	--metrics flag	text/plain; version=0.0.4	Prometheus exposition format text
/slots	GET	--slots flag	application/json	JSON array of slot state objects

Prometheus metrics output format:

# HELP llamacpp:prompt_tokens_total Number of prompt tokens processed.
# TYPE llamacpp:prompt_tokens_total counter
llamacpp:prompt_tokens_total 15234

# HELP llamacpp:tokens_predicted_total Number of generation tokens processed.
# TYPE llamacpp:tokens_predicted_total counter
llamacpp:tokens_predicted_total 8921

# HELP llamacpp:prompt_tokens_seconds Average prompt throughput in tokens/s.
# TYPE llamacpp:prompt_tokens_seconds gauge
llamacpp:prompt_tokens_seconds 1523.7

# HELP llamacpp:predicted_tokens_seconds Average generation throughput in tokens/s.
# TYPE llamacpp:predicted_tokens_seconds gauge
llamacpp:predicted_tokens_seconds 42.3

# HELP llamacpp:requests_processing Number of requests processing.
# TYPE llamacpp:requests_processing gauge
llamacpp:requests_processing 2

# HELP llamacpp:requests_deferred Number of requests deferred.
# TYPE llamacpp:requests_deferred gauge
llamacpp:requests_deferred 0

Complete counter metrics:

Metric Name	Type	Description
llamacpp:prompt_tokens_total	counter	Cumulative prompt tokens processed
llamacpp:prompt_seconds_total	counter	Total prompt processing time (seconds)
llamacpp:tokens_predicted_total	counter	Cumulative generation tokens
llamacpp:tokens_predicted_seconds_total	counter	Total generation time (seconds)
llamacpp:n_decode_total	counter	Total llama_decode() calls
llamacpp:n_tokens_max	counter	Largest observed n_tokens
llamacpp:n_busy_slots_per_decode	counter	Average busy slots per decode call

Complete gauge metrics:

Metric Name	Type	Description
llamacpp:prompt_tokens_seconds	gauge	Current prompt throughput (tokens/s)
llamacpp:predicted_tokens_seconds	gauge	Current generation throughput (tokens/s)
llamacpp:requests_processing	gauge	Currently active requests
llamacpp:requests_deferred	gauge	Queued/deferred requests

Usage Examples

Prometheus scrape configuration:

scrape_configs:
  - job_name: 'llama-server'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/metrics'

Health check with fail_on_no_slot for load balancing:

# Returns 503 if all slots are busy
curl -f "http://localhost:8080/slots?fail_on_no_slot=1" || echo "Server is at capacity"

Kubernetes liveness and readiness probes:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 60

Related Pages

• Principle:Ggml_org_Llama_cpp_Server_Monitoring