llama.cpp Errors and Fixes: A Practical Troubleshooting Guide

A surprising number of llama.cpp troubleshooting threads eventually receive the same advice:

git pull
cmake --build build --config Release -j

That advice is often correct. It is also incomplete.

llama.cpp moves quickly. New model architectures, GGUF metadata, multimodal projectors, quantization formats, CUDA kernels, schedulers, and command-line behavior can land within days of a model release. A binary that is only a few weeks old can genuinely be too old.

But not every llama.cpp error is an old-build problem. Updating will not repair a malformed GGUF, fix the wrong model file, create missing upstream architecture support, make an impossible VRAM allocation fit, or correct a conversion whose tensor dimensions disagree with the architecture metadata.

This guide is a diagnostic map rather than a list of magic commands.

The 60-second triage

Before changing parameters, record exactly what you are running:

./build/bin/llama-cli --version
git rev-parse HEAD

Then preserve the complete command and the first meaningful error. If you are using a packaged application, Python binding, Docker image, Ollama/LM Studio integration, or another wrapper, record its embedded llama.cpp version if available. Updating a separate source checkout does not update the backend embedded in another application.

Now classify the failure:

That distinction saves more time than blindly changing ten flags.

1. unknown model architecture: '...'

Typical error:

llama_model_load: error loading model: unknown model architecture: 'gemma4'

or:

unknown model architecture: 'glm5next'

This is one of the cases where update first is good advice.

The loader reads the GGUF architecture identifier and maps it to an architecture llama.cpp knows how to instantiate. Current source explicitly throws an unknown model architecture runtime error when that mapping resolves to LLM_ARCH_UNKNOWN.

Historically, new model families have produced exactly this failure before support landed. Gemma 4 had an upstream feature request reporting unknown model architecture: 'gemma4'. GLM-5.3 Flash users later reported unknown model architecture: 'glm5next' while requesting support.

Fix path

First update and rebuild:

git pull
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
./build/bin/llama-cli --version

Then search the exact architecture string in llama.cpp issues and pull requests.

Fixed by updating? SOMETIMES.

If support has already merged and your binary predates it, yes. If the architecture is still an open feature request or unmerged PR, git pull cannot give you code that is not on the main branch.

Also verify that you are loading the correct file. An architecture error can be misleading when a projector or other auxiliary GGUF is accidentally supplied as the main language model.

2. unknown model architecture: 'clip' can mean you loaded the wrong file

An instructive historical example is llama.cpp issue #7799. The command supplied a multimodal projector GGUF as both --model and --mmproj:

--model ...mmproj-f16.gguf
--mmproj ...mmproj-f16.gguf

The resulting error was:

error loading model architecture: unknown model architecture: 'clip'

That is not the same diagnosis as “llama.cpp does not support my language model.”

A multimodal setup normally has distinct artifacts: the language model and the projector. If the projector is passed where the language model belongs, the loader is being asked to interpret the wrong kind of GGUF.

Fix path

Check filenames and roles before rebuilding anything:

--model   -> language-model GGUF
--mmproj  -> multimodal projector GGUF

Fixed by updating? NO, if the command points at the wrong artifact.

This is why the exact command belongs in every useful bug report.

3. tensor '...' has wrong shape

Examples from historical reports include:

create_tensor: tensor 'blk.0.attn_k.weight' has wrong shape

and:

check_tensor_dims: tensor 'blk.0.ffn_gate.weight' has wrong shape

This error is fundamentally different from an unknown architecture. llama.cpp has progressed far enough to know what tensor it expects, but the dimensions in the loaded GGUF do not match that expectation.

Possible causes include:

• a custom model whose architecture differs from what its metadata claims;
• a converter that did not understand the modified architecture;
• a GGUF converted with incompatible assumptions;
• a runtime/converter version mismatch;
• incorrect model metadata;
• an actual llama.cpp regression.

Issue #9235, for example, reported an expected FFN gate shape of 768, 2048 but a converted tensor with 768, 3072. Issue #5367 described a custom TinyLlama-derived model that began failing after a llama-cpp-python update.

Fix path

Do not immediately quantize the file again. First compare:

1. the original model config.json;
2. the converter version/commit;
3. the GGUF metadata;
4. the llama.cpp commit used to load it;
5. whether an unmodified upstream checkpoint of the same family loads.

For custom architectures, reconvert from the original weights using a converter that explicitly supports the architecture.

Fixed by updating? SOMETIMES, but often NO.

If the GGUF contains dimensions that genuinely disagree with the model definition llama.cpp implements, a newer binary cannot rewrite those tensors while loading them.

4. tensor ... data is not within the file bounds

This error sounds almost self-explanatory:

tensor '...' data is not within the file bounds, model is corrupted or incomplete

The obvious suspects are an incomplete download, truncated shard, bad merge, or damaged file. Check file sizes and hashes against the publisher’s values when available. For split GGUFs, make sure every required shard is present.

But do not treat the text as infallible proof of a corrupt download.

llama.cpp issue #9770 documented a Windows/MSVC code bug where two different definitions of a file structure could cause the runtime to read the wrong file size and then emit this same corruption-style message for a model that was not necessarily corrupt.

Fix path

Start with the artifact:

ls -lh model*.gguf
sha256sum model*.gguf

If the files match known-good hashes, reproduce on current llama.cpp before downloading hundreds of gigabytes again.

Fixed by updating? SOMETIMES.

A truncated file requires a new file. A loader bug requires a runtime fix.

5. cudaMalloc failed: out of memory

Typical output:

ggml_backend_cuda_buffer_type_alloc_buffer: ...
cudaMalloc failed: out of memory

This is where git pull is least useful as a default response.

Model weights are only part of the memory budget. llama.cpp may also need memory for KV cache, compute buffers, CUDA graphs, temporary workspaces, output buffers, and other backend allocations. Context length, batch sizes, parallel slots, KV types, GPU-layer placement and model architecture all matter.

Issue #11435 is a useful example: the model loaded across RPC/CUDA devices but later failed while trying to reserve an additional compute buffer. Issue #10052 showed a configuration where enabling Flash Attention changed memory behavior enough for a small model to run on an 8 GiB RTX 2080.

Fix path

First inspect actual free VRAM immediately before launch:

nvidia-smi

Then reduce one variable at a time. Depending on the current llama.cpp CLI and model, useful levers can include:

• fewer GPU-offloaded layers;
• shorter context;
• smaller batch / micro-batch;
• fewer parallel sequences;
• lower-memory KV-cache types where supported;
• Flash Attention where supported and appropriate;
• changing tensor/device split on multi-GPU systems.

Do not copy flags from an old forum post without checking --help; llama.cpp CLI options evolve.

Fixed by updating? USUALLY NO.

There are real memory regressions and backend fixes, but ordinary capacity pressure is a configuration problem.

6. OOM that grows during inference is not the same as OOM at startup

The timing of the failure matters.

If allocation fails immediately while the model initializes, investigate static memory requirements and configuration first.

If memory usage climbs continuously token after token, request after request, or only under sustained server load, suspect a leak, cache-growth behavior, or backend regression.

Issue #20315 reported an RPC/CUDA case where memory grew during generation until OOM, with the reporter tracing it to CUDA graph cache entries. That is qualitatively different from simply choosing too many GPU layers.

Fix path

Measure memory over time and build a minimal reproducer. Record whether the growth is proportional to:

• generated tokens;
• requests;
• distinct prompt shapes;
• context checkpoints;
• slots;
• speculative graph shapes.

Fixed by updating? YES if the regression has subsequently been fixed; otherwise NO.

Do not hide a reproducible leak by permanently shrinking the model until it stops crashing.

7. failed to allocate compute buffers

A model can appear to fit and still fail here:

ggml_gallocr_reserve_n: failed to allocate CUDA... buffer
llama_init_from_model: failed to allocate compute buffers

The important lesson is that “the GGUF is smaller than my VRAM” is not a sufficient capacity calculation.

A quantized 20 GiB file does not imply that a 24 GiB GPU can run every context/batch/offload configuration. Runtime state consumes additional memory, and allocation may need contiguous capacity on a particular device.

Fix path

Treat model weights, KV cache and compute buffers as separate budget lines. If multi-GPU placement leaves one device tight while another has free memory, inspect the actual split rather than only total aggregate VRAM.

Fixed by updating? USUALLY NO.

8. Multimodal failures: diagnose the language model and projector separately

Modern llama.cpp multimodal paths add another compatibility surface. A log can mention:

Failed to load CLIP model
failed to get memory usage of mmproj
unknown model architecture

Do not collapse all three into “the model doesn’t work.”

Ask separately:

1. Does the language-model GGUF load without multimodal input?
2. Is this projector built for this exact model family/version?
3. Is the runtime new enough for the model architecture?
4. Is the projector being passed through the correct option?
5. Did the model publisher specify a particular llama.cpp commit or PR?

Fixed by updating? SOMETIMES.

New multimodal families frequently need runtime support, but mismatched projectors and wrong file selection survive every git pull.

9. Assertions and speculative decoding: configuration can expose a real runtime limit

A current example is draft PR #28972. The reported scheduler hash set was sized once, while speculative-decoding graphs created by draft-mtp, draft-dspark, or draft-eagle3 can grow with --spec-draft-n-max.

The upstream reproduction cited in the PR showed:

n_max=8: hash=2053, nodes+leafs=2000 -> fits
n_max=9: hash=2053, nodes+leafs=2054 -> overflow by 1

The result is an assertion, not a graceful “your speculative setting is too high” error. The proposed fix grows the scheduler structures rather than asserting.

The important troubleshooting lesson is broader than this one PR: if changing a performance parameter predictably crosses a threshold and trips an internal assertion, do not assume the parameter itself is invalid. You may have exposed a capacity assumption inside the runtime.

Fixed by updating? NOT YET if the relevant fix is still an unmerged draft PR.

That distinction matters. “There is a fix on GitHub” does not mean “current master contains the fix.”

10. A crash that begins after git pull is evidence too

Updating is an experiment, not a ritual.

If yesterday’s commit works and today’s does not, preserve both SHAs. Do not immediately destroy the known-good build.

A useful local workflow is:

git rev-parse HEAD
./build/bin/llama-cli --version

before updating, then build into a clean directory. If the regression is stable, git bisect can turn “new llama.cpp broke my model” into a specific first-bad commit.

For serious local-AI work, keep the following with benchmark and reproduction results:

llama.cpp commit
build options
compiler/CUDA version
GPU + driver
model URL/revision
GGUF filename(s)
exact command
complete relevant log

Fixed by updating? Obviously not when the update caused it.

11. Python bindings, desktop apps and containers can be version traps

One of the easiest mistakes is:

cd ~/llama.cpp
git pull

followed by rerunning a completely different binary.

llama-cpp-python, LM Studio, packaged binaries, Docker images and other applications can embed or ship their own llama.cpp revision. Your fresh source checkout may have no relationship to the executable producing the error.

Always identify the binary first.

On Linux:

which llama-cli
readlink -f "$(which llama-cli)"

For containers, record the image tag or digest. For Python, record the package version and build provenance. For desktop applications, use the application’s own runtime/version information.

Fixed by updating? YES only if you update the thing you are actually running.

12. GGUF provenance matters as much as llama.cpp provenance

When a GGUF fails, record where it came from.

There is a major diagnostic difference between:

• an official/publisher conversion;
• a community quantization of known upstream weights;
• your own conversion;
• a custom fine-tune with architecture changes;
• a file converted months ago with an older converter;
• a split model with a missing shard.

A useful report identifies both ends:

runtime: llama.cpp <commit>
model:   <repository + revision + exact GGUF filename>

Without that pair, “latest llama.cpp” and “the Q4 model” are not reproducible descriptions.

The update-first rule, refined

Here is the rule I actually use:

Update first when

• the model family was released recently;
• the error is unknown model architecture;
• an upstream issue says support has already merged;
• you are several weeks or months behind master;
• the failure is known to be a runtime regression that has been fixed.

Do not expect an update alone to fix

• a wrong --model / --mmproj file;
• missing or truncated GGUF shards;
• tensor dimensions that are genuinely wrong;
• a context/offload configuration that exceeds memory;
• a custom architecture the converter does not understand;
• support that exists only in an open PR;
• a regression introduced by the newer build.

That turns git pull from cargo cult into a diagnostic step.

A reproducible troubleshooting template

Before opening an issue, collect this:

llama.cpp commit/version:
OS:
CPU:
GPU(s):
GPU driver:
CUDA/ROCm/Metal/Vulkan version:
build command/options:
model repository:
model revision:
GGUF filename(s):
GGUF source/converter:
exact llama.cpp command:
first meaningful error:
full relevant log:
works on older commit? yes/no/unknown
CPU-only reproduction? yes/no/not tested

Then make the reproduction smaller. Reduce optional flags. Try the smallest context that still fails. If a backend-specific failure is suspected and the model is small enough, compare CPU-only behavior. If the failure appears at a specific speculative-decoding value, test immediately below and above the boundary.

This is much more valuable to maintainers than “latest version, model doesn’t work.”

Decision tree

Model fails
|
+-- unknown architecture?
|   +-- old build -> update/rebuild
|   +-- support merged -> verify you are running rebuilt binary
|   +-- support only in open PR -> master cannot run it yet
|   +-- architecture says clip/projector -> verify file roles
|
+-- wrong tensor shape?
|   +-- verify original config
|   +-- verify converter + runtime revisions
|   +-- reconvert known-good source weights
|   +-- custom architecture -> converter/runtime support may be missing
|
+-- file bounds / corrupted?
|   +-- verify all shards, size, checksum
|   +-- known-good file still fails -> test current runtime / loader regression
|
+-- CUDA OOM?
|   +-- startup -> memory budget, context, batch, offload, KV, split
|   +-- grows over time -> leak/cache/regression investigation
|
+-- assertion/crash?
    +-- reproduce with minimal flags
    +-- identify threshold
    +-- compare known-good commit
    +-- search exact assertion in upstream issues/PRs

Why this guide will keep changing

llama.cpp is unusually fast-moving infrastructure. That is a feature: support for new models and hardware can arrive rapidly. It also means troubleshooting advice decays rapidly.

So this page should not become a museum of obsolete command lines. The durable layer is the classification method:

artifact problem?
runtime-support problem?
conversion/metadata problem?
memory/configuration problem?
backend problem?
regression?

Once you know which layer failed, git pull becomes either an excellent first fix or an obvious waste of time.

RAMGPT will keep the narrow forensic investigations in separate articles and use this guide as the troubleshooting hub. When a new exact error deserves source-level analysis, the detailed article can change; the diagnostic map above should remain useful.