Fix Ollama CUDA Out of Memory Errors on NVIDIA RTX GPUs

To fix Ollama CUDA Out of Memory (OOM) errors on NVIDIA RTX graphics cards, reduce your context window size with num_ctx 4096, offload excess layers to system RAM using num_gpu, and set NVIDIA CUDA Sysmem Fallback to “Prefer No Sysmem Fallback”. These adjustments prevent VRAM exhaustion and eliminate single-token inference freezes.

Running local language models like Llama 3.1, Qwen 2.5, or DeepSeek through Ollama gives our engineering team complete privacy and zero API costs. But when testing models on our developer workstations equipped with consumer GPUs (8GB RTX 4060s, 12GB RTX 3060s, and 16GB RTX 4070 Ti Supers), the dreaded CUDA out of memory or ggml_cuda_host_malloc: failed to allocate memory error appears the moment conversations extend past a few prompts.

Below is our team workbench runbook to calculate exact memory footprints, configure clean layer offloading, and prevent driver-level slowdowns.

+-------------------------------------------------------------------------+
|                  NVIDIA VRAM ALLOCATION PIPELINE                        |
+-------------------------------------------------------------------------+
| [Base Model Weights]     -> 4-bit Quantized Tensors (~4.8 GB for 8B)   |
| [KV Cache Dynamic]       -> Token Context History (1.2 GB - 3.5 GB)    |
| [CUDA Runtime Overhead]  -> Context Buffers & Workspace (~500 MB)       |
|                                                                         |
| Total Required: > 8.0 GB Physical VRAM Limit                           |
| CRASH: CUDA out of memory / Sysmem PCIe bus bottleneck                 |
+-------------------------------------------------------------------------+

What Triggers CUDA OOM During Local Model Inference

CUDA out of memory errors occur when the combined sum of model weights, KV cache memory, and CUDA runtime context buffers exceeds physical GPU VRAM.

When loading an 8-billion parameter model, developers often assume an 8GB GPU has sufficient headroom because the GGUF file is only 4.7 GB on disk. However, live inference requires substantial dynamic memory allocation across three primary zones:

1. Static Model Weights: The quantized weights loaded permanently into VRAM. An 8B Q4_K_M model requires roughly 4.8 GB of contiguous VRAM.
2. Key-Value (KV) Cache: Every token processed and retained in conversation memory allocates key and value vectors in VRAM. At an 8,192 token window, the KV cache demands between 1.2 GB and 2.2 GB depending on attention head dimensions.
3. CUDA Context Overhead: Initializing the CUDA driver context and allocating scratch workspaces consumes 450 MB to 700 MB of VRAM before processing a single prompt.

If background desktop apps (such as browsers, Discord, or IDEs) consume 1.5 GB of display VRAM, an 8GB card only has 6.5 GB of usable space. Loading a 4.8 GB model with a 2.0 GB context window pushes total demand to 6.8 GB, immediately triggering a CUDA OOM crash.

To verify your system requirements before running models, test your hardware allocations with our free VRAM Calculator.

Monitor Real-Time VRAM Usage with nvidia-smi

Tracking baseline VRAM consumption using nvidia-smi identifies rogue background processes consuming GPU memory before launching Ollama.

Before tuning configuration parameters, check how much unreserved VRAM is actually available on your primary GPU:

nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv -l 1

If your idle system shows over 1.2 GB of VRAM in use, close hardware-accelerated background applications or configure Windows Graphics Settings to run desktop monitors through integrated CPU graphics (iGPU), reserving your discrete RTX card entirely for compute.

Reduce Context Window Size in Custom Modelfiles

Restricting the num_ctx parameter to 2048 or 4096 tokens cuts KV cache memory consumption by up to 75 percent without changing the model weights.

Many popular frontends like Open-WebUI or Continue default to requesting 8,192 or 16,384 tokens. For an 8GB card, this causes instant memory exhaustion. You can permanently lock the context window by building a tuned Modelfile.

Create a plain text file named Modelfile on your workstation:

FROM llama3.1:8b

# Limit context window to 4K tokens to save ~1.5GB VRAM
PARAMETER num_ctx 4096

# Set temperature and system prompt
PARAMETER temperature 0.7
SYSTEM "You are an expert IT systems engineer providing concise answers."

Build your optimized model instance in your terminal:

ollama create llama3-4k -f ./Modelfile

Launch the model:

ollama run llama3-4k

By capping context at 4,096 tokens, you retain enough history for technical troubleshooting while freeing up 1.5 GB of precious VRAM.

Split Model Layers Across GPU and CPU Memory

Using the num_gpu parameter offloads a calculated number of model layers to system RAM, keeping inference active without exhausting VRAM.

When a model is slightly too large for your graphics card (such as a 14B parameter model on a 12GB GPU), you do not have to abandon local execution. Ollama allows partial offloading:

FROM qwen2.5:14b

# Total layers = 48. Put 34 on GPU, offload 14 to system RAM
PARAMETER num_gpu 34
PARAMETER num_ctx 4096

ollama create qwen-hybrid -f ./Modelfile
ollama run qwen-hybrid

While offloading layers across the PCIe bus to DDR4 or DDR5 system RAM slightly lowers generation speed (tokens per second), it guarantees 100% stability and prevents CUDA allocation failures.

If you are running Ollama inside Linux on Windows, optimize your virtual machine memory limits using our WSL Config Generator.

Disable NVIDIA CUDA Sysmem Fallback Policy

Disabling CUDA Sysmem Fallback forces clean memory management and prevents extreme inference slowdowns when VRAM approaches capacity.

Starting with NVIDIA Driver Version 536, NVIDIA enabled automatic memory fallback to system RAM when VRAM overflows. While this prevents games from crashing, it causes local LLMs to freeze, dropping generation speeds from 45 tokens per second down to 0.4 tokens per second.

To disable this behavior on Windows:

1. Right-click the desktop and open NVIDIA Control Panel.
2. Navigate to 3D Settings > Manage 3D Settings > Program Settings.
3. Click Add and select ollama_llama_server.exe (typically located at C:\Users\<Username>\AppData\Local\Programs\Ollama).
4. Locate CUDA - Sysmem Fallback Policy in the settings list.
5. Change the setting from Driver Default to Prefer No Sysmem Fallback.
6. Click Apply in the bottom right corner.

+-------------------------------------------------------------------------+
|                  NVIDIA CONTROL PANEL CONFIGURATION                     |
+-------------------------------------------------------------------------+
| Program:        ollama_llama_server.exe                                 |
| Setting:        CUDA - Sysmem Fallback Policy                           |
| Value:          [ Prefer No Sysmem Fallback ]                           |
| Result:         Fails fast into Ollama layer manager instead of hanging |
+-------------------------------------------------------------------------+

Configure Windows Environment Variables for Ollama

Setting OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS prevents multiple concurrent requests from triggering secondary VRAM allocations.

By default, Ollama may attempt to keep multiple models loaded or handle parallel prompt slots, multiplying memory demand. On consumer hardware, restrict Ollama to a single active model:

Open PowerShell as Administrator and run:

[System.Environment]::SetEnvironmentVariable('OLLAMA_MAX_LOADED_MODELS', '1', 'Machine')
[System.Environment]::SetEnvironmentVariable('OLLAMA_NUM_PARALLEL', '1', 'Machine')
Restart-Service -Name "Ollama" -ErrorAction SilentlyContinue

These variables ensure Ollama fully unloads old model weights from VRAM before loading a new model, eliminating memory leaks between session transitions.

Hardware Sizing Guide for Local LLM Deployments

Matching your model parameter scale and quantization level to your physical VRAM ensures sustainable local inference without crashes.

By following this sizing matrix and enforcing explicit context limits, your local Ollama server will maintain peak tokens-per-second performance without crashing your workstation.