The trend of Large Language Model (LLM) based applications has prompted an increasing number of developers to run their own AI models on fully controlled infrastructure. This is often driven by cost considerations, data privacy requirements, or simply the desire to eliminate reliance on third-party API rate limits. However, AI models present resource demands that differ significantly from conventional web applications: image sizes can easily swell to tens of gigabytes, memory consumption is high, and performance often depends entirely on GPU access, which Docker does not expose to containers by default. This chapter covers how to containerize LLM-based applications using Ollama, ranging from basic setup and enabling NVIDIA GPU access via nvidia-container-toolkit, to configuring reasonable resource limits and deploying an API gateway in front of the model serving infrastructure for integration with other applications.
35.1 Containerizing LLM-Based Applications with Ollama
Ollama is an open-source runtime designed for running large language models locally. It packages model downloading, model version management, and an inference server into a single, straightforward command-line interface. Ollama provides an official image on Docker Hub under the name ollama/ollama, eliminating the need to write a custom Dockerfile from scratch when deploying ready-to-use models such as Llama or Mistral.
35.1.1 Running an Ollama Server with Docker Compose
Developers new to Ollama typically start with a single-line docker run command. However, as requirements expand to include model persistence and integration with other services, compose.yaml becomes a significantly cleaner approach for managing configurations. Create the following compose.yaml file to run the Ollama server:
services:
ollama:
image: ollama/ollama:0.34.0
container_name: ollama
ports:
- "11434:11434"
volumes:
- ollama-data:/root/.ollama
restart: unless-stopped
volumes:
ollama-data:The volume ollama-data is mapped to /root/.ollama, which is the directory where Ollama stores pulled models. Without this volume, every container re-creation forces the re-download of gigabyte-sized models from scratch—an inefficient use of bandwidth and time in production environments. The version tag 0.34.0 is explicitly used instead of latest, adhering to standard image tagging practices discussed in previous chapters to ensure inference behavior does not unexpectedly shift due to automatic updates.
Start the stack, then pull the model using docker exec within the running container:
docker compose up -d
docker exec -it ollama ollama pull llama3.2Once the model download completes, test its inference endpoint via curl to verify that the Ollama API responds correctly:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Jelaskan apa itu container dalam satu kalimat",
"stream": false
}'A JSON response containing a response field indicates that the model is active and ready to process inference requests. If an error such as model not found occurs, it typically indicates a typographical error in the model name or an incomplete pull operation from the preceding step.
35.1.2 Connecting a Backend Application to Ollama
A backend application that needs to invoke the LLM only requires connectivity to the ollama service over the shared Compose network, without needing to expose port 11434 to the host unless external access is required. Add the following app service to the existing compose.yaml file:
services:
app:
build: ./app
environment:
OLLAMA_HOST: http://ollama:11434
depends_on:
- ollama
networks:
- default
ollama:
image: ollama/ollama:0.34.0
volumes:
- ollama-data:/root/.ollama
networks:
- default
networks:
default:
volumes:
ollama-data:The environment variable OLLAMA_HOST used here represents a custom naming convention within the application codebase rather than an internal Ollama variable; ensure it matches the variable declared in the backend implementation. The service name ollama serves directly as the hostname because Compose automatically places both services on the same internal network, following standard inter-container DNS resolution patterns. While depends_on guarantees that the ollama container starts first, note that this option only waits for the container to reach a running state—it does not confirm that the model API is ready to accept incoming requests. Implementing basic retry logic within the application code is recommended to handle potential startup race conditions.
35.2 GPU Access in Containers with nvidia-container-toolkit
Executing LLM models solely on a CPU remains feasible for small models, but latency is substantially higher compared to GPU execution, particularly for models with billions of parameters. Because Docker isolates containers from host hardware by default—including GPUs—an additional component called the NVIDIA Container Toolkit is required to enable container access to host-attached NVIDIA GPUs.
35.2.1 Installing the NVIDIA Container Toolkit
This installation takes place on the host machine rather than within a Dockerfile, as the toolkit functions by modifying the Docker runtime on the host to map GPU devices into containers. The corresponding NVIDIA driver for the installed graphics hardware must be configured on the host prior to installing this toolkit, following official NVIDIA Container Toolkit documentation prerequisites. On Debian or Ubuntu-based Linux distributions, first add the official NVIDIA repository:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.listUpdate the package index and install the toolkit:
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkitFor RHEL or Fedora-based distributions such as CentOS or Rocky Linux, use dnf after adding the appropriate repository via dnf config-manager, following official NVIDIA guidelines for yum/dnf environments. Once the toolkit is installed, configure the Docker daemon to recognize the NVIDIA runtime, then restart the Docker service to apply the configuration changes:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerThe command nvidia-ctk runtime configure updates the NVIDIA runtime configuration inside /etc/docker/daemon.json automatically, eliminating manual file modifications. Windows environments running WSL2 with Docker Desktop use a distinct installation path that leverages NVIDIA drivers integrated directly into WSL2 without requiring a separate toolkit setup inside the Linux distribution. Systems administrators and DevOps engineers should verify the specific setup requirements corresponding to their installed version of Docker Desktop.
35.2.2 Verifying GPU Access inside a Container
Before attaching GPU resources to Ollama, verify the toolkit installation by running NVIDIA's official nvidia-smi test image:
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smiThe --gpus all flag has been available since Docker Engine 19.03 and serves as the standard mechanism to expose all detected GPUs to a container. Output displaying GPU metrics (such as device names, memory utilization, and driver versions) confirms a successful installation. An error reading could not select device driver "" with capabilities: [[gpu]] indicates that the Docker runtime configuration has not applied the updates from nvidia-ctk runtime configure, which can typically be resolved by restarting the Docker service.
After successful verification, pass GPU access to the ollama service via docker run:
docker run -d --gpus=all -v ollama-data:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:0.34.0When utilizing Docker Compose, GPU access is configured under the deploy.resources.reservations.devices block per the Compose Specification for device requests:
services:
ollama:
image: ollama/ollama:0.34.0
volumes:
- ollama-data:/root/.ollama
ports:
- "11434:11434"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: ["gpu"]
volumes:
ollama-data:The capabilities: ["gpu"] field is mandatory; leaving it empty causes Compose to reject the configuration with an error. Setting count: all allocates all host-detected GPUs to the container. To limit GPU usage to a specific number of devices on multi-GPU hosts, replace this value with an integer (for example, count: 1), or use device_ids to target specific GPUs by index. The count and device_ids fields are mutually exclusive and should not be declared simultaneously.
Relaunch the stack and access the container to confirm that Ollama detects the GPU correctly:
docker compose up -d
docker exec -it ollama nvidia-smiIf this command fails or displays no GPU devices, Ollama falls back silently to CPU mode without raising an explicit error, leading to degraded inference performance. Common root causes include host NVIDIA driver incompatibilities with the installed toolkit version, or a failure to restart the Docker daemon following runtime configuration.
35.3 Resource Management for AI Workloads
LLM workloads exhibit resource utilization patterns distinct from traditional web applications: memory demands are large and remain static while a model is loaded, whereas CPU and GPU utilization spike sharply during active inference processing. Without explicit resource bounds, an unconstrained AI container experiencing memory exhaustion can trigger the kernel Out Of Memory (OOM) killer, potentially disrupting neighboring containers on the host.
35.3.1 Restricting Container CPU and Memory Allocation
Apply resource constraints using the deploy.resources.limits block inside compose.yaml according to the Compose Specification for resource limits:
services:
ollama:
image: ollama/ollama:0.34.0
volumes:
- ollama-data:/root/.ollama
deploy:
resources:
limits:
cpus: "4"
memory: 16G
reservations:
cpus: "2"
memory: 8G
devices:
- driver: nvidia
count: all
capabilities: ["gpu"]
volumes:
ollama-data:The limits block defines the maximum resource ceiling allowed for the container, whereas reservations defines the minimum reserved allocation. Note that deploy.resources settings are primarily intended for orchestrators like Docker Swarm; when executed using standalone docker compose up without Swarm mode enabled, only limits are enforced on the container, while non-GPU device reservations are generally ignored. System administrators and DevOps engineers deploying Ollama via standalone Docker Compose should rely primarily on limits for resource management.
Additionally, host-level container memory limits do not govern the internal concurrency of models loaded into memory by Ollama. Configure the environment variable OLLAMA_MAX_LOADED_MODELS to restrict the number of concurrently loaded models, and OLLAMA_NUM_PARALLEL to restrict parallel request processing limits per model, as detailed in the official Ollama server environment variable documentation:
services:
ollama:
image: ollama/ollama:0.34.0
environment:
OLLAMA_MAX_LOADED_MODELS: "1"
OLLAMA_NUM_PARALLEL: "2"
deploy:
resources:
limits:
memory: 16GCombining container-level constraints with internal application runtime variables prevents severe performance degradation, high queuing latencies, or timeouts caused by attempting to load multiple large models into restricted memory space.
35.3.2 Monitoring Resource Utilization
Monitor real-time resource usage for the Ollama container using docker stats to ensure assigned limits match operational workloads:
docker stats ollamaIf the MEM USAGE / LIMIT column consistently approaches the configured upper boundary, memory limits should be adjusted upward or the number of concurrently loaded models reduced. For long-term monitoring, telemetry aggregation, and tracking GPU utilization alongside inference latency, integrate the container into a Prometheus and Grafana observability pipeline using cadvisor to collect container metrics.
35.4 API Gateway for Model Serving
By default, Ollama exposes a native API structure that differs from the OpenAI API standard widely adopted across LLM client libraries. Connecting multiple downstream services directly to an Ollama instance also prevents central management of authentication, request logging, or load balancing across multiple backend endpoints. Deploying an API gateway in front of Ollama addresses these architecture requirements.
35.4.1 Using Ollama as an OpenAI-Compatible Endpoint
Ollama supports an integrated endpoint compatible with OpenAI's Chat Completions format via the /v1/chat/completions path. This native compatibility simplifies client integration by allowing developers to standard OpenAI client SDKs simply by altering the base_url configuration:
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.2",
"messages": [{"role": "user", "content": "Halo, siapa kamu?"}]
}'While adequate for simple architectures interacting with a single Ollama instance, complex environments requiring multi-model routing, per-client virtual API keys, or usage accounting benefit from a dedicated gateway layer positioned between client applications and underlying models.
35.4.2 Deploying LiteLLM Proxy as a Gateway
LiteLLM Proxy is an open-source API gateway that standardizes access across numerous LLM providers (including Ollama) into a unified OpenAI-style API, while offering virtual key management and tracking metrics. Create a litellm-config.yaml file mapping model aliases to the running Ollama backend:
model_list:
- model_name: llama3.2
litellm_params:
model: ollama_chat/llama3.2
api_base: http://ollama:11434
general_settings:
master_key: os.environ/LITELLM_MASTER_KEYThe ollama_chat/ prefix in the model parameter instructs LiteLLM to route requests to Ollama's /api/chat endpoint rather than /api/generate, which yields better conversational output formatting. Next, add the litellm service to the compose.yaml configuration:
services:
ollama:
image: ollama/ollama:0.34.0
volumes:
- ollama-data:/root/.ollama
litellm:
image: ghcr.io/berriai/litellm:v1.99.1
ports:
- "4000:4000"
volumes:
- ./litellm-config.yaml:/app/config.yaml:ro
environment:
LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
command: ["--config", "/app/config.yaml", "--port", "4000"]
depends_on:
- ollama
volumes:
ollama-data:The LITELLM_MASTER_KEY variable serves as the primary authentication key for the gateway. It should be defined inside an uncommitted .env file to safeguard secret management. Launch the stack and query the gateway using standard OpenAI client formatting:
docker compose up -d
curl http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{
"model": "llama3.2",
"messages": [{"role": "user", "content": "Halo, siapa kamu?"}]
}'Backend services connecting to the gateway only need to reference the LiteLLM network address and pass a valid API key, decoupling client applications from specific underlying model providers.
35.4.3 Gateway Verification and Troubleshooting
If the gateway returns a 401 Unauthorized response, confirm that the Authorization header adheres to the Bearer <key> format matching the active LITELLM_MASTER_KEY environment variable. A model not found error from the gateway generally indicates a mismatch between the model_name string inside litellm-config.yaml and the payload model identifier, as LiteLLM matches model names case-sensitively.
If the gateway forwards requests but returns downstream backend errors, inspect the container logs for litellm to review raw response details from Ollama:
docker compose logs -f litellmCommon deployment issues include incorrect service name references within api_base, or attempting to invoke models that have not been pulled via docker exec ollama ollama pull. Additionally, ensure proper startup ordering: if LiteLLM attempts to query backend models while Ollama is still initializing, implement container healthchecks alongside depends_on conditions to defer gateway initialization until Ollama is fully operational.