Principle:Ollama Ollama Model Loading And GPU Scheduling
| Knowledge Sources | |
|---|---|
| Domains | Systems, GPU_Computing, Model_Serving |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
A resource-aware scheduling mechanism that loads LLM models into GPU/CPU memory on demand, managing concurrent model instances with memory-based eviction and device assignment.
Description
Model Loading and GPU Scheduling addresses the challenge of serving multiple large language models on limited hardware. When an inference request arrives, the scheduler must determine whether the requested model is already loaded, whether sufficient GPU memory exists to load it, or whether an existing model must be evicted to make room.
The scheduler maintains a map of loaded model runners, tracks GPU memory usage per device, and uses a queuing system to serialize model loading operations. It supports multi-GPU systems by distributing model layers across available GPUs based on their VRAM capacity.
This principle is critical for production LLM serving where multiple models may be requested concurrently and GPU memory is the primary bottleneck.
Usage
Use this principle when designing an inference serving system that must dynamically load and unload models based on request patterns, manage GPU memory across multiple devices, and handle concurrent inference requests with different model requirements.
Theoretical Basis
The scheduling algorithm follows these steps:
- Request Arrival: A request specifies a model name and runtime options (context length, GPU layers, etc.).
- Cache Lookup: Check if the model is already loaded and meets the request requirements (context size, adapter compatibility).
- Fast Path: If loaded and compatible, return the existing runner immediately.
- Queue Path: Otherwise, enqueue a load request to the scheduler.
- Memory Assessment: The scheduler evaluates available GPU memory across all devices.
- Eviction (if needed): If insufficient memory, evict the least-recently-used model runner.
- Model Loading: Launch a new inference runner (llama.cpp process or Go-native model), distributing layers across GPUs.
- Runner Registration: Register the runner and return it to the requesting handler.
Pseudo-code:
// Abstract scheduling algorithm
func schedule(request) runner {
if runner = cache.get(request.model); runner != nil && runner.fits(request) {
return runner // fast path
}
while memoryAvailable() < request.memoryNeeded() {
evict(leastRecentlyUsed())
}
runner = loadModel(request.model, assignGPUs(request))
cache.put(request.model, runner)
return runner
}