llama.cpp Learns to Borrow Idle VRAM for Vision

A 16 GB GPU can have enough memory to run a vision encoder and still be unable to keep that encoder resident beside a long-context language model.

That sounds contradictory until you stop treating VRAM as one static capacity number.

A fresh llama.cpp pull request, #28320, proposes an unusually pragmatic solution: while the server is encoding an image, temporarily release the language model’s idle compute scratch buffers, load the multimodal projector into that space, perform the GPU encode, free the projector, and let the normal decode path reserve its scratch buffers again.

The interesting part is not merely the reported speedup. It is the resource-management model behind it.

For production inference, this PR is a useful example of why VRAM residency and VRAM demand are not the same thing.

The Constraint Was Temporal, Not Absolute

The conventional choice in a memory-constrained multimodal deployment is straightforward. If the text model plus its runtime state leaves insufficient room for the vision projector, run the projector on CPU with --no-mmproj-offload.

That avoids out-of-memory failures, but it can turn image ingestion into the slowest stage of the request.

PR #28320 argues that this framing misses an important detail: some of the memory preventing the projector from fitting is allocated for compute scratch that is idle during image encoding.

The contributor measured a configuration using two Tesla V100-SXM2 16 GB GPUs, Qwen3.8-27B Q6_K, an 885 MiB F16 multimodal projector, tensor split across both GPUs, Flash Attention, and a 150,016-token context. For a 1,048-token image prompt, the reported prompt-evaluation time was:

That is a contributor-reported measurement from the open PR, not a RAMGPT benchmark. It is also a specific hardware and model configuration, so the ratio should not be generalized to every GPU.

But the mechanism deserves attention even if your hardware produces a much smaller gain.

Borrow, Use, Return

The proposed lifecycle is approximately:

text model resident
KV cache resident
compute scratch reserved
        |
image arrives
        |
release idle compute scratch
        |
load temporary GPU projector
        |
encode image
        |
copy resulting embeddings to host-owned batch
        |
free projector
        |
decode begins
        |
compute scratch reserved again

The key safety property is that the model weights and KV cache are not released. Only scheduler/graph-allocation buffers that are not needed while the projector runs are surrendered.

According to the PR, the tested configuration released about 2,401 MiB of Flash Attention F16 workspace plus 632 MiB of activation storage. During the temporary swap, reported VRAM usage peaked at 16,071 MiB out of 16,384 MiB per GPU, with a low-water mark of 13,969 MiB.

This is much closer to memory scheduling than traditional model offload.

Nothing needs to migrate the LLM’s weights out to system RAM. The server instead exploits the fact that two consumers need the same physical capacity at different times.

Why This Matters Operationally

Inference stacks often report memory requirements as though every allocation were simultaneously live:

model weights
+ KV cache
+ compute workspace
+ multimodal projector
= required VRAM

That equation is safe, but it can be unnecessarily restrictive.

A more useful production model distinguishes persistent and phase-specific allocations:

persistent:
  model weights
  conversation state / KV cache

text-compute phase:
  compute scratch
  activations

vision phase:
  multimodal projector
  vision working state

If the text-compute and vision phases do not overlap, their transient allocations can potentially share a memory budget.

This is the same reason sophisticated schedulers care about resource lifetimes rather than only maximum declared resource sizes. Capacity planning becomes a scheduling problem.

For local inference, that distinction is particularly valuable because users frequently operate exactly at the VRAM boundary. Moving even one more component to CPU can change latency dramatically.

Long Context Makes the Technique More Interesting

The test case is notable for its 150K-token context.

Long-context inference consumes substantial memory in state and runtime workspace. A simplistic response to an out-of-memory projector is to reduce context length or move more model components away from the GPU.

The PR claims neither was necessary in its test: context remained unchanged and no model layers were moved.

That makes this technique qualitatively different from ordinary offload tuning.

It is attempting to preserve three things simultaneously:

1. model placement;
2. long context;
3. GPU-accelerated vision encoding.

The trade is additional lifecycle complexity rather than permanent loss of GPU residency for another component.

The Remaining 1.5-Second Problem

The contributor also decomposed the 3,619 ms path.

Creating the transient multimodal context with mtmd_init_from_file reportedly consumed 1,504 ms. The remaining 2,115 ms included model prefill of the 1,048 embedding positions.

The PR author estimates that transferring the 885 MiB projector over PCIe 3.0 x16 should account for only a small fraction of that 1.5 seconds, suggesting GGUF opening, parsing and buffer allocation are now significant overheads.

That exposes the next optimization frontier.

Once CPU vision encoding is removed from the critical path, initialization becomes visible.

A future implementation could potentially retain parsed projector metadata or otherwise reduce repeated setup while still allocating its large GPU buffers only during the vision phase. That is explicitly outside the current PR, but the progression is familiar in production optimization:

remove 20-second bottleneck
        |
1.5-second setup becomes visible
        |
optimize lifecycle/setup
        |
approach unavoidable prefill floor

Optimization changes what the bottleneck is.

There Is a Reliability Cost

Dynamic memory reuse is not free engineering-wise.

Static allocation is easy to reason about. Allocate everything once, prove that it fits, and keep addresses and lifetimes stable.

A phase-aware allocator introduces more transitions:

• release scheduler resources;
• instantiate another GPU context;
• handle allocation failure;
• preserve host-owned embeddings after the transient context disappears;
• restore compute reservations before decode;
• ensure prompt caching and text-only turns are unaffected.

PR #28320 therefore includes an important fallback: if the temporary GPU projector cannot be loaded, image encoding falls back to the host path. The proposal is also opt-in via --mmproj-vram-swap.

Those details matter more to production engineering than the headline benchmark. A fast path that turns memory pressure into request failure would be difficult to operate safely.

A Better Way to Think About Local-AI VRAM

Local inference optimization is often framed as packing:

How many gigabytes can I squeeze onto this GPU?

This PR suggests a more useful question:

Which gigabytes actually need to coexist at the same instant?

That shift matters as inference stacks become more heterogeneous. A single request may invoke a language model, vision encoder, audio encoder, speculative draft model, reranker, grammar/sampling graph, and increasingly large context state.

Keeping all of those components permanently resident is the simplest architecture, but not necessarily the best architecture for constrained hardware.

The server can instead treat VRAM as a scheduled resource with lifetimes.

In infrastructure terms, the difference resembles static capacity reservation versus workload-aware resource reuse.

What to Watch Upstream

PR #28320 is new and should be treated as a proposal, not an established llama.cpp feature. The benchmark numbers and memory observations above come from its contributor.

The questions worth watching during review are therefore less about whether 22.4 seconds becomes exactly 3.6 seconds elsewhere and more about the lifecycle contract:

• Is scheduler release safe across supported backends and server paths?
• Can projector initialization overhead be reduced without keeping its GPU allocation resident?
• How robust is fallback under fragmented VRAM rather than simple capacity pressure?
• Does concurrency complicate the assumption that compute scratch is idle during image encoding?
• Can the same lifetime-aware technique safely support other mutually exclusive inference components?

That last question is the architectural one.

Bottom Line

The most interesting idea in PR #28320 is not “GPU vision is faster than CPU vision.” We already know that.

It is that a memory allocation can block a workload even when the allocation is idle during the exact phase that needs the memory.

By temporarily lending text-inference scratch space to the vision projector, llama.cpp may be able to preserve long context, model placement, and GPU multimodal acceleration on hardware where the static sum of those allocations appears not to fit.

For production inference engineering, that is a useful principle beyond this particular patch:

Optimize memory by lifetime, not only by size.

As local models accumulate more modalities and auxiliary inference components, the next major VRAM optimization may come less from making every tensor smaller and more from making the runtime smarter about which tensors and workspaces truly need to coexist.