DeepSeek-R1 FP8 vs Q4: VRAM Benchmarks & Speed Guide (2026)

Choosing a DeepSeek R1 quantization before you download it prevents a common local-inference failure: the model fits on disk but not in GPU memory once the runtime allocates the KV cache. In this guide, I separate model-weight estimates from runtime memory, explain what FP8, Q4_K_M, and Q8_0 mean, and show the Ollama and llama.cpp settings that control the tradeoff.

Note on scope: DeepSeek R1 comes in two forms: the full 671B MoE model (requires a multi-GPU server) and the Distill models (1.5B to 70B, designed for local consumer hardware). This guide focuses on the distill models since those are what most people actually run locally.

What Quantization Actually Does

When a language model trains, its weights are stored as 32-bit or 16-bit floating point numbers. Quantization compresses each weight into fewer bits to reduce memory footprint.

The tradeoff is precision loss: the fewer bits you use, the more rounding error accumulates across billions of parameters — and the more the model outputs can drift from the full-precision reference.

The GGUF format (used by llama.cpp and Ollama) supports many quantization levels. For DeepSeek R1 distill models, the most relevant are:

Q4_K_M vs Q4_0: The _K_M suffix means k-means quantization with a medium calibration dataset. It is meaningfully better quality than the older Q4_0 at the same VRAM cost. Always prefer Q4_K_M over Q4_0.

⚡ Try the Interactive Tool: Don’t want to calculate KV cache and weight memory by hand? Use our free Local AI VRAM & Quantization Calculator to select your exact GPU (RTX 4060, 4090, Apple Silicon M-series) and get real-time memory headroom plus copy-paste ollama commands.

Estimated VRAM Table: DeepSeek R1 Distill Models

These are planning estimates based on parameter count and common GGUF weight sizes, cross-checked against the Ollama DeepSeek R1 model card. They are model weights only. Add runtime overhead and KV-cache memory; a 1–3 GB allowance is a starting estimate, not a guarantee for every context length or backend.

Full 671B MoE model: FP8 requires approximately 700 GB of VRAM, Q8_0 requires 350–400 GB, Q4_K_M requires 200–220 GB. This is a multi-GPU server workload. Use the Distill models above for local inference.

Quality vs Speed Tradeoffs: What the Numbers Predict

Quantization changes memory traffic and the amount of rounding error, but it does not produce one universal tokens-per-second result. Backend, GPU architecture, context length, batch size, offload layers, and thermal limits all change the measured speed. Treat any benchmark that omits those conditions as a rough comparison rather than a promise.

• Q4_K_M usually gives the smallest weight footprint and leaves the most room for KV cache. It is the practical starting point for an 8 GB card, but long contexts can still spill to system RAM.
• Q5_K_M trades some of that headroom for a smaller precision loss. Use it when the model fits with comfortable memory margin and you care about preserving more detail.
• Q8_0 keeps more information per weight but can consume almost twice the weight memory of Q4. It is useful when the GPU has the capacity or when a quality comparison justifies the cost.
• FP8 uses floating-point scaling and needs a backend with compatible CUDA support. It is not automatically faster than Q8_0 on consumer hardware.

For a reproducible local comparison, record the GPU model, driver, backend version, model file hash, context length, prompt, generated token count, and peak VRAM. Run the same prompt three times after the model is warm, then report the median tokens per second. Without those details, a claimed “28 tokens per second” is not portable evidence.

Bottom line: For coding, summarization, and general Q&A, Q4_K_M is the right default. Move to Q5_K_M or Q8_0 only if you need tighter math/reasoning accuracy and have the VRAM.

Running DeepSeek R1 Distill with Ollama

Ollama selects the best GGUF quantization it can fit in your available VRAM automatically. You can override it explicitly.

Pull a specific model and quantization

# 7B distill, default (Ollama picks best fit)
ollama pull deepseek-r1:7b-qwen-distill-q4_K_M

# 8B distill, explicit Q4_K_M
ollama pull deepseek-r1:8b-0528-qwen3-q4_K_M

# 14B distill, Q8_0 (requires ~17 GB VRAM)
ollama pull deepseek-r1:14b-qwen-distill-q8_0

Run interactively

ollama run deepseek-r1:8b-0528-qwen3-q4_K_M

Cap context to reduce KV cache pressure on 8 GB cards

ollama run deepseek-r1:7b-qwen-distill-q4_K_M --num-ctx 4096

Ollama’s default context for DeepSeek R1 can be very large. On 8 GB cards, setting --num-ctx 4096 or --num-ctx 8192 prevents out-of-memory crashes during long conversations. See our full guide on running DeepSeek R1 on 8GB VRAM for Modelfile tuning options.

Running DeepSeek R1 Distill with llama.cpp

Download a GGUF file from HuggingFace (for example, bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF).

Download via HuggingFace CLI

pip install huggingface_hub
huggingface-cli download bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF \
  DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf \
  --local-dir ./models

Run inference

./llama-cli \
  -m ./models/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf \
  -n 512 \
  -ngl 35 \
  --ctx-size 4096 \
  -p "Explain the difference between Q4_K_M and Q8_0 quantization."

Key flags:

• -ngl 35 — number of layers offloaded to GPU. For a 7B Q4_K_M model on 8 GB, -ngl 35 offloads almost all layers.
• --ctx-size 4096 — context window. Keep low on 8 GB cards.
• -n 512 — max tokens to generate.

Verify GPU offloading

./llama-cli -m ./models/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf \
  --n-gpu-layers 999 --ctx-size 2048 -p "test" 2>&1 | grep "offloaded"

If you see offloaded 0/32 layers, your llama.cpp build was not compiled with CUDA support.

Fixing Tool Calling Issues

If you are using the distill models for agentic tasks, the thinking tokens (<think>...</think>) in DeepSeek R1 outputs can break JSON tool-call parsers. We have a dedicated guide: Fix DeepSeek R1 Tool Calling in Ollama (Thinking Budget).

For a second local-inference comparison, see the Ollama GPU-offload and context-size guide. It covers the same failure mode from the runtime side: a model can load successfully, then exhaust VRAM when the context grows.

Quick Reference: Which Quantization for Your GPU

When in doubt, start with Q4_K_M and leave at least 1.5 GB of VRAM headroom after loading — that buffer is used by the KV cache and prevents out-of-memory errors during generation.