Troubleshooting

Local LLM Slow? Diagnose Prefill, Decode, GPU Offload, and Memory

Find why a local LLM is slow by separating prompt processing from token generation, checking GPU offload, memory pressure, context growth, and CPU bottlenecks.

Approximately 5 min read

If your local LLM feels slow, do not start by changing ten settings. First answer two questions:

  1. Is the long wait before the first token, or is every generated token slow?
  2. Is the runtime actually using the GPU or accelerator you think it is?

Those two checks eliminate most blind tuning.

Start here: four checks

1. Confirm the device

With llama.cpp:

llama-cli --list-devices

On NVIDIA systems, also check the active GPU processes:

nvidia-smi

A GPU appearing in the operating system does not prove the model is actually placed there. Read the model-loading log and confirm which tensors or layers are being offloaded.

2. Measure prompt processing and generation separately

Run a repeatable baseline:

llama-bench -m model.gguf -p 512 -n 128 -r 5

llama-bench reports prompt processing and text generation separately. That matters because a machine can process a prompt quickly but generate each new token slowly, or the reverse.

The benchmark does not include tokenization or sampling time. If llama-bench is fast but the chat application still feels slow, the delay may be elsewhere in the stack.

3. Compare one controlled change

Keep the model, prompt size and generation size fixed. Change only one thing: offload, model size, quantization, batch size, context, CPU threads, or competing workloads.

4. Use the symptom to choose the next experiment

What you observe Most useful next check
Long delay before first token, but generation is fine Prompt length, prompt-processing speed, batch settings, context and cache state
Prompt processing is fine, but generation is slow Model size, GPU offload, host-memory traffic, CPU decode and memory bandwidth
Both are slow Backend, device placement, model size, power/thermal limits or a generally overloaded system
First run is slow, later runs improve Model loading, warm-up, cache state or compilation
Long chats get progressively slower Growing context and KV-cache pressure
Performance collapses when other apps run VRAM, RAM, memory bandwidth or GPU contention

If generation is slow, check model placement first

A common failure mode is a model that technically runs but leaves too much work on the CPU.

Check the runtime logs after model load. If only part of the model is offloaded, compare several GPU-layer settings that are valid for your model. llama-bench supports repeated offload comparisons, for example:

llama-bench -m model.gguf -ngl 0,16,32 -p 512 -n 128 -r 5

Replace the layer counts with values appropriate for the model. The point is not that a particular number is universally best; it is to see whether performance changes sharply as more of the working set moves to the accelerator.

If more offload helps but the model no longer fits comfortably, compare a smaller quantization or a smaller model that still completes the task.

Do not judge GPU use from utilization percentage alone. Decode can be limited by memory movement or host/device traffic even when utilization is not pinned near 100%.

If prompt processing is slow, test batch and context

Prompt processing handles many input tokens at once, so it responds differently to batch settings and context size than token-by-token generation.

A batch-size comparison can be run with:

llama-bench -m model.gguf -n 0 -p 1024 -b 256,512,1024 -r 5

Use the same model and device placement while comparing. A larger batch can improve throughput on some hardware, but it also consumes more working memory. If the system starts paging or pushes other data out of VRAM, the apparent optimization can reverse.

Also test a short fresh prompt against the long conversation that feels slow. If the short prompt is fast and the long one is not, context growth is part of the problem.

If CPU generation is slow, do not assume more threads are better

CPU decode often becomes limited by memory bandwidth rather than raw core count. Test thread counts instead of automatically using every logical CPU:

llama-bench -m model.gguf -p 0 -n 128 -t 4,8,16 -r 5

Use values that make sense for your CPU. If performance plateaus or falls as threads increase, the machine may already be saturating its useful memory path.

This is especially important on systems with heterogeneous cores, NUMA layouts, or other workloads competing for bandwidth.

Check RAM and swapping when the model spills out of VRAM

For Linux hosts:

free -h
vmstat 1

If the model or cache spills into host memory, generation can become much more dependent on CPU and system-memory bandwidth. Sustained swapping is an even stronger warning sign.

Storage speed matters most during loading and paging. Once the working set is resident, decode speed is usually dominated by compute placement and memory movement rather than disk throughput.

Treat long context as a separate workload

Do not benchmark a 512-token prompt and assume the result predicts a long-running 32K or 128K conversation.

Longer context changes cache use, attention work and available memory. Compare at least two context depths that resemble the real workload. If the slowdown appears only after the conversation grows, shrinking the model may not be the only answer; retained context and cache configuration may be the actual constraint.

Keep the before/after test honest

For every change, keep these fixed where possible:

Record prompt-processing speed and text-generation speed separately.

The useful result is not “this setting is faster.” It is “this setting improved the slow phase under the workload I actually care about.”

Fast decision guide

If you want the shortest path:

  1. Run llama-cli --list-devices.
  2. Run llama-bench -m model.gguf -p 512 -n 128 -r 5.
  3. Decide whether prompt processing, generation, or both are slow.
  4. Confirm model placement.
  5. Change one variable and repeat the same benchmark.

If memory is the limiting factor, continue with How to Reduce LLM VRAM Usage. If you are still building the first working llama.cpp setup, use llama.cpp for Beginners. For quantization choices, see GGUF Quantization Levels Explained.

Sources and further reading