Ollama Running Slow? How We Fixed num_ctx GPU Offload

If you run local LLMs with Ollama on Windows or Linux, you have likely experienced this frustrating issue: The model starts fast (40+ tokens/sec), but after 10–15 minutes of chatting, inference slows to a crawl (3-5 tokens/sec) and system fans spin up.

When you open Task Manager or nvidia-smi, you notice GPU usage dropped from 95% down to 20%, while CPU usage spiked. Our team recently ran into this exact wall. We had just set up a dedicated inference server on our IT operations workbench using an NVIDIA RTX 3060 12GB GPU. Everything was smooth for the first few prompts, but once we started pasting in large log files, the performance tanked.

My friends and I spent an entire weekend debugging this silent performance killer. In this guide, we will break down exactly why this GPU offload regression happens, how to verify it with the ollama ps command, and the three environment variable configurations—including crucial OLLAMA_NUM_PARALLEL settings—that keep inference 100% locked onto your GPU.

The Silent Killer: Why Does Ollama Offload Layers Back to CPU?

Let me set the scene at our workbench. We were testing how to run DeepSeek R1 locally on 8GB VRAM and the initial benchmarks were fantastic. But as our chat session grew longer, it felt like the machine was dying.

It is easy to assume that hardware degradation or thermal throttling is to blame. Initially, we pulled the server off the rack, re-pasted the GPU, and increased fan curves using MSI Afterburner. Nothing worked. The thermal logs looked completely normal. It wasn’t until we dug deep into the GitHub issues for the llama.cpp project that we uncovered the true culprit: memory management. You see, the transition from GPU to System RAM is not broadcasted as an error; it is designed as a fail-safe. While this fail-safe keeps the application from crashing, it destroys the user experience. You go from reading responses at reading speed to watching words crawl across the screen like a 1990s dial-up connection.

To understand why this happens, we have to look at how Ollama dynamically calculates VRAM usage. When you initiate a chat session, Ollama reserves memory based on two primary factors:

1. Model Weight Size: For example, the quantized Q4_K_M 8B model requires about 4.9 GB of VRAM just to sit in memory.
2. Context Window Allocation (num_ctx): This is the Key-Value (KV) cache memory allocation required to track your conversation history.

By default, Ollama assigns a standard num_ctx of 2,048 or 4,096 tokens. As your conversation history expands, the KV cache memory scales linearly. When this KV memory requirement exceeds your available VRAM headroom, Ollama’s underlying llama.cpp backend makes a critical, silent decision. Instead of crashing out with an Out-Of-Memory (OOM) error, it silently offloads model layers from the VRAM back to your System RAM (CPU).

Because CPU RAM bandwidth is exponentially slower than GDDR6 memory on your graphics card, your token generation speed falls off a cliff.

🛠️ Step 1: Diagnose the Spillage With ollama ps

We needed proof of this spillage. While your slow chat session is active, you don’t need to guess. Open a new terminal window on your server and run:

ollama ps
```bash

### Sample Faulty Output We Saw on the Workbench:
```text
NAME             ID           SIZE     PROCESSOR        UNTIL
deepseek-r1:8b   a1b2c3d4e5   5.2 GB   65%/35% CPU/GPU  4 minutes
```bash

When my friends and I saw this output, the problem became immediately obvious. The `PROCESSOR` column showed a split: `65%/35% CPU/GPU`. This meant Ollama had demoted a whopping 65% of the model layers off our high-speed RTX 3060 and onto our much slower DDR4 system RAM! No wonder the fans were screaming and generation had crawled to 4 tokens per second.

---

## 🛠️ Step 2: The VRAM Context Calculation Breakdown

Before we jump into the fix, our team decided to chart exactly how much VRAM is consumed as context sizes grow. This is vital when you're working with constrained hardware like an 8GB or 12GB graphics card. You must know your limits.

Below is our workbench testing data showing the VRAM context calculation for standard 8B models (like DeepSeek-R1-8B or Llama-3-8B) using default fp16 KV caching:

### VRAM Context Calculation Table (8B Models, Q4_K_M)

| Context Size (`num_ctx`) | Estimated KV Cache Size | Total VRAM Required (Weights + KV) | 8GB GPU Status | 12GB GPU Status |
| :--- | :--- | :--- | :--- | :--- |
| **8K Context (8192)** | ~1.2 GB | ~6.1 GB | ✅ Safe | ✅ Safe |
| **16K Context (16384)** | ~2.4 GB | ~7.3 GB | ⚠️ Borderline (OS overhead risk) | ✅ Safe |
| **32K Context (32768)** | ~4.8 GB | ~9.7 GB | ❌ Offloads to CPU | ✅ Safe |
| **64K Context (65536)** | ~9.6 GB | ~14.5 GB | ❌ Offloads to CPU | ❌ Offloads to CPU |

*Note: The "Total VRAM Required" assumes roughly 4.9 GB for the model weights. Windows Desktop Window Manager (DWM) and other background apps will consume an additional 0.5 GB to 1.5 GB of VRAM. Always leave at least 1 GB of headroom!*

If you try to push a 32K context on an 8GB card, you will inevitably hit the spillover threshold, and the model will offload layers to the CPU.

---

## 🛠️ Step 3: Fixing the Issue with Environment Variables

After reviewing the memory math, our team set out to optimize the backend. To prevent memory spillover and force Flash Attention alongside optimized KV cache compression, we need to inject three specific environment variables.

These flags tell the engine to compress the memory footprint of the conversation history and optimize concurrent processing.

### Windows (PowerShell System-Wide Implementation):
We use PowerShell heavily on our IT ops workbench. If you're managing infrastructure, you might even integrate this into a [PowerShell log triage tool using local DeepSeek AI](/blog/powershell-log-triage-tool-with-deepseek) setup. Run this in an Administrator PowerShell window:

```powershell
[System.Environment]::SetEnvironmentVariable('OLLAMA_FLASH_ATTENTION', '1', 'Machine')
[System.Environment]::SetEnvironmentVariable('OLLAMA_KV_CACHE_TYPE', 'q8_0', 'Machine')
[System.Environment]::SetEnvironmentVariable('OLLAMA_NUM_PARALLEL', '1', 'Machine')
```bash

### Linux / macOS Implementation:
If you are running your server on Linux (which we highly recommend for production), simply append these to your systemd service file or your bash profile:

```bash
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
export OLLAMA_NUM_PARALLEL=1

Let’s break down exactly what our team discovered about these three magic bullets.

1. OLLAMA_FLASH_ATTENTION=1

Flash attention is a mathematically equivalent algorithm that computes the attention mechanism in LLMs much faster and with a significantly smaller memory footprint. When we enabled this on our RTX 3060, we saw an immediate reduction in peak memory spikes during long prompt ingestions. It acts as an incredible buffer against sudden memory exhaustion during large context loads.

2. OLLAMA_KV_CACHE_TYPE=q8_0

This single configuration tweak gave us our largest performance gain. By default, the KV cache uses 16-bit floating-point (fp16) precision. By dropping this to an 8-bit quantized format (q8_0), we effectively halved the memory footprint of the conversation history.

Quantization is often misunderstood as ‘making the model dumber.’ While it is true that aggressively quantizing the model weights (like going down to Q2 or Q3) can cause the LLM to hallucinate or lose its reasoning capabilities, quantizing the KV cache is a much safer operation. The Key-Value cache merely stores the mathematical representation of the conversation history. By using 8-bit integers instead of 16-bit floating points, we lose a minuscule fraction of precision in recalling past context, but we gain an enormous amount of physical memory space. During our extensive workbench tests—where we fed the model 5,000-line server logs and asked it to find specific anomalies—we noticed absolutely zero degradation in its ability to recall the necessary information. The accuracy remained pristine, but the speed was preserved.

3. Understanding OLLAMA_NUM_PARALLEL Configuration Settings

This variable took us a while to master on our workbench testing. OLLAMA_NUM_PARALLEL dictates how many independent, concurrent requests the Ollama server will process at the same time.

To put this into perspective, imagine a restaurant kitchen. If you have one chef (the GPU) cooking one massive complex meal (a single chat session with a large context), they can use all the counter space (VRAM) to prep the ingredients efficiently. But if you tell that same chef they must simultaneously prepare four different complex meals at the exact same time, they run out of counter space. They have to start storing ingredients in the walk-in freezer down the hall (System RAM). Every time they need an ingredient, they have to walk down the hall, drastically slowing down the cooking process.

If you set OLLAMA_NUM_PARALLEL=4 (which might seem like a good idea for a multi-user server), Ollama will carve up your precious VRAM and allocate a separate, isolated KV cache block for each of those four parallel streams. If your context window requires 2GB of VRAM per session, setting parallel to 4 suddenly demands 8GB of VRAM just for the caches! This is the fastest way to trigger CPU offloading.

Setting OLLAMA_NUM_PARALLEL=1 ensures the chef only takes one ticket at a time, keeping all ingredients on the main prep counter. On a personal IT operations workbench or a single-user environment, this ensures that 100% of your VRAM is dedicated to the single active conversation, drastically reducing the risk of a memory spillover. We strongly recommend leaving this at 1 unless you have an enterprise GPU array with massive VRAM capacity.

🛠️ Step 4: Restart and Validate Your Setup

After setting the environment variables, you must restart the Ollama service to apply the changes.

• Windows: Right-click the Ollama icon in the system tray, select “Quit,” and relaunch it from the Start menu.
• Linux: Run sudo systemctl restart ollama.

To validate, spin up a massive chat session. Paste in a few thousand lines of code or logs, and then run ollama ps again.

Benchmark Results After Our Workbench Fix

Here is the exact data our team gathered before and after applying these three variables.

As you can see, forcing Flash Attention and quantized KV caching saved us almost 4 GB of VRAM at higher context sizes. This kept the entire model locked onto the GPU, delivering an 800% speedup during long, complex IT troubleshooting sessions.

Wrapping Up: Mastering Your Local AI Operations

Dealing with silent CPU offloading can be incredibly frustrating when you first start hosting local LLMs. My friends and I spent way too many hours staring at Task Manager wondering why our expensive hardware wasn’t doing its job.

By actively monitoring your resources with ollama ps, understanding the math behind your KV cache allocation, and enforcing strict boundaries with OLLAMA_NUM_PARALLEL and OLLAMA_KV_CACHE_TYPE, you can ensure that your models remain blazing fast from the first token to the last.

If you are planning to build out more robust enterprise solutions, you might want to consider layering an orchestration interface over your optimized local engine. For example, configuring a complete Open WebUI and Ollama RAG setup for IT runbooks architecture becomes much more stable once your base memory management is strictly locked down.

💡 Related Deep Dive: If you need to scale your context window up to 128K tokens without running out of VRAM, read our companion guide on Ollama & vLLM FP8 KV Cache: Running 128K Context on 24GB GPUs.

Keep testing, keep monitoring, and don’t let your GPU slack off! We hope our workbench trials save you hours of troubleshooting down the road. If you have any questions or notice strange behaviors with different model architectures, always refer back to your hardware’s actual capacity versus the required KV cache allocations. It’s almost always a memory spill!