llama.cpp Vision Latency: The Hidden Scheduler Reserve Cost

A multimodal inference slowdown can look like a vision-model problem even when the expensive work is happening in runtime memory planning.

A fresh llama.cpp pull request isolates such a case. The proposed patch removes a forced scheduler reservation whenever the runtime changes its causal-attention mode around image chunks.

For image-heavy prompts, the contributor reports a large effect. With Gemma 4 26B A4B, 24 images, a 32768 context setting, ubatch 2048, and an RTX 4090, median prompt time reportedly falls from 13,377 ms to 1,759 ms. On an H200, the reported change is 5,379 ms to 1,584 ms.

These are upstream contributor measurements, not RAMGPT benchmarks. The pull request remains open as of September 13, 2026.

Why image processing changes attention mode

Autoregressive text normally uses causal attention. Vision chunks can need different attention behavior. In llama.cpp’s multimodal path, the mode changes around non-causal image chunks for Gemma 3, Gemma 4, and DeepSeek V4 vision paths.

Conceptually, the runtime moves from causal text processing to a non-causal image chunk and then back to causal text processing.

According to the pull request, each change currently marks the scheduler as needing a full reserve pass. One image chunk can therefore cause two reservations, and many images repeat the cost.

Graph rebuild and buffer reservation are different

A graph scheduler needs enough backend memory for the graph it executes. Reserving again is useful when graph shapes or required capacities change.

For the multimodal architectures that actually toggle this mode at runtime, the patch argues that causal attention normally changes mask values rather than tensor dimensions. The graph still rebuilds because attention mode is part of the graph-reuse key, but a full memory reservation is usually redundant.

A graph identity change does not automatically mean tensor sizes or required buffer capacity changed.

Why long context amplifies the cost

The contributor says reservation cost scales with context and ubatch configuration. The reported RTX 4090 measurements show the pattern.

One image at context 8192 and ubatch 512 changes from 201 ms to 119 ms. Twenty-four images at the same settings change from 3,559 ms to 1,748 ms. With 24 images at context 32768 and ubatch 2048, the reported result becomes 13,377 ms to 1,759 ms.

The vision encoder did not suddenly become dramatically cheaper. The avoidable scheduler work became more expensive as configured buffer requirements grew.

That gives troubleshooting a useful clue: latency that scales with context size is not necessarily attention compute or KV-cache compute. Runtime memory planning can scale with those settings too.

The Qwen3.8-Flash-Next exception

Review found an important complication. The simplest claim would be that changing causal attention never changes tensor sizes. That is not true for every model.

The contributor traced the exception to the qwen4exp architecture used for Qwen3.8-Flash-Next. There, causal attention affects a bias tensor whose size can change.

The contributor argues that normal scheduler allocation can grow the compute buffer when required and that llama.cpp does not currently toggle this mode at runtime for qwen4exp in the multimodal path. They also report a manual runtime test with output matching master.

A reviewer nevertheless identified a stricter issue: configurations that prohibit scheduler reallocation could expose the shape dependency. The suggested direction is to fix that model dependency explicitly before relying on later reallocation.

That is why this should be described as an open optimization, not a performance change already available in upstream releases.

The useful systems invariant

The deeper question is what a runtime parameter actually changes: values only, graph shape, or required buffer capacity.

Those are different kinds of state. If a parameter changes values but not capacity, rebuilding the graph may be enough. If it changes tensor dimensions, the scheduler may genuinely need more memory.

Treating every graph-key change as a capacity change is conservative but can be expensive. Treating every change as value-only is fast but unsafe.

The proposed change is an attempt to separate those cases for causal-attention changes while accounting for the model-specific exception.

Why multi-image workloads expose it

A fixed runtime overhead can disappear inside a one-image benchmark. Repeating it around every image makes it visible.

If each non-causal image chunk causes two unnecessary reserve passes, 24 images can create many repeated reserve operations. Video-style workloads can magnify the same pattern because they contain many visual chunks.

This is a common inference-systems failure mode: a harmless-looking operation becomes dominant because it sits inside a repeated multimodal loop.

What to compare when diagnosing it

Useful comparisons for unexpectedly high multimodal prompt latency include one image versus many images, small versus large context, small versus large ubatch, and text-only versus image prompts on the same model and backend.

If prompt time grows much faster than visual content alone suggests, scheduler allocation is worth inspecting before blaming the vision encoder.

For this patch, the most valuable next evidence is correctness coverage across architectures whose graph shapes can depend on causal attention, including configurations that prohibit reallocation.

Bottom line

The new pull request identifies a hidden potential tax in llama.cpp multimodal inference: causal-attention changes around image chunks can trigger full scheduler reservation even when compute-buffer sizes have not changed.

The contributor reports the penalty becoming dramatic at large context and ubatch settings. But review also found the important boundary condition in Qwen3.8-Flash-Next’s qwen4exp graph.

The broader lesson is durable: graph state, tensor shape, and scheduler capacity are different kinds of state. Conflating them can turn a tiny flag change into substantial avoidable multimodal latency.