Principle:Ggml org Llama cpp Server Monitoring
| Field | Value |
|---|---|
| Principle Name | Server Monitoring |
| Domain | Observability, Production Operations |
| Description | Theory of health checking and metrics collection for production inference services |
| Related Workflow | OpenAI_Compatible_Server |
Overview
Description
The Server Monitoring principle defines the theory behind providing observability into the operational state of an inference server. Production deployments of language model inference services require real-time visibility into server health, resource utilization, throughput, and individual slot status. This principle covers three monitoring dimensions:
- Health checking: Binary up/down status indicating whether the server is operational, loading, or in an error state. The
/healthendpoint is designed to be lightweight and accessible even during model loading or sleep states. - Metrics collection: Quantitative performance measurements exposed in Prometheus-compatible text format, including token throughput (prompt and generation), processing times, decode call counts, and slot utilization statistics.
- Slot inspection: Detailed per-slot state information showing which inference slots are active, idle, or processing requests, enabling capacity planning and debugging.
Usage
Server monitoring endpoints serve different consumers:
- Load balancers and orchestrators poll
/healthto determine whether to route traffic to a given instance. - Prometheus/Grafana stacks scrape
/metricsat regular intervals to build dashboards and alerts for throughput degradation or slot exhaustion. - Operators and debuggers query
/slotsto inspect the current state of each inference slot, diagnose stuck requests, or verify capacity.
All monitoring endpoints are opt-in for security: --metrics enables the Prometheus endpoint, --slots enables slot inspection. Only /health is always available and requires no authentication.
Theoretical Basis
Prometheus exposition format is the standard adopted for metrics output. Each metric follows the Prometheus naming convention with a llamacpp: namespace prefix and includes HELP and TYPE annotations. Metrics are categorized as either counters (monotonically increasing values like total tokens processed) or gauges (point-in-time measurements like current throughput or active requests).
The key metrics tracked are:
llamacpp:prompt_tokens_total(counter): Cumulative prompt tokens processedllamacpp:tokens_predicted_total(counter): Cumulative generation tokens producedllamacpp:prompt_seconds_total(counter): Total prompt processing timellamacpp:tokens_predicted_seconds_total(counter): Total generation timellamacpp:n_decode_total(counter): Number ofllama_decode()invocationsllamacpp:prompt_tokens_seconds(gauge): Current prompt throughput in tokens/secondllamacpp:predicted_tokens_seconds(gauge): Current generation throughput in tokens/secondllamacpp:requests_processing(gauge): Number of currently active requestsllamacpp:requests_deferred(gauge): Number of queued/waiting requests
Health check semantics distinguish between three states: ok (model loaded, ready to serve), loading (HTTP server running but model not yet loaded), and error (a fatal condition has occurred). The health endpoint is intentionally minimal and stateless, accessing no server context during sleep mode to avoid waking the inference engine.
Task-based metrics collection routes metrics requests through the same task queue as inference requests, ensuring a consistent snapshot of server state. The SERVER_TASK_TYPE_METRICS task is posted as high-priority to minimize latency in metrics collection.