Local AI

llama.cpp Vision Latency: The Hidden Scheduler Reserve Cost

PR 28751 traces multi-image llama.cpp latency to repeated scheduler reservation when causal attention changes around vision chunks.

Approximately 3 min read

With Gemma 4 26B A4B, 24 images, context 32,768, and ubatch 2,048, a llama.cpp contributor reported median prompt time falling from 13,377 ms to 1,759 ms on an RTX 4090. The change addressed repeated scheduler reservation around vision chunks rather than the vision encoder itself.

PR #28751 merged on September 28, 2026. Its contributor measurements show how the overhead changes with workload size:

GPU Images Context Ubatch Before After
RTX 4090 1 8,192 512 201 ms 119 ms
RTX 4090 24 8,192 512 3,559 ms 1,748 ms
RTX 4090 24 32,768 2,048 13,377 ms 1,759 ms
H200 24 32,768 2,048 5,379 ms 1,584 ms

Two reservations around an image chunk

Text generation normally uses causal attention. Vision chunks in the relevant Gemma 3, Gemma 4, and DeepSeek V4 multimodal paths switch to non-causal attention and then return to causal processing.

Before the change, each switch marked the scheduler for a full reservation pass. An image chunk could therefore trigger reservation twice. Repeating that work across many images made a small-looking state change expensive.

Longer context and larger ubatch increased the reservation cost in the reported tests. That explains why the largest improvement appears in the 24-image, 32K-context case.

A new graph does not always need new capacity

Attention mode participates in the graph-reuse key, so changing it still requires rebuilding the graph. The optimization separates that rebuild from reserving backend memory again.

For the multimodal paths under discussion, changing causal attention normally changes mask values while leaving tensor dimensions and required capacity unchanged. Reusing the existing reservation avoids repeating memory-planning work that has no additional capacity to provide.

This distinction gives a useful diagnostic question: did a runtime setting change tensor values, graph identity, or allocation size? Those changes require different amounts of work.

Review found a shape-dependent exception

The qwen4exp architecture used by Qwen3.8-Flash-Next complicates the general rule. Its causal-attention setting can change the size of a bias tensor.

The contributor argued that ordinary scheduler allocation could grow the compute buffer when needed, and that the multimodal path did not toggle causal attention for qwen4exp. They also reported a manual runtime test whose output matched master.

Review raised the case where scheduler reallocation is prohibited. In that configuration, relying on a later buffer resize would not be sufficient. This is why a model-specific shape dependency matters even when the measured optimization targets another architecture.

Diagnose an image-heavy slowdown

For a packaged build, first check whether its revision includes the merged change. Then compare one image with many images, and vary context and ubatch independently while keeping the model and backend fixed.

A prompt-time increase much larger than the additional visual content warrants investigation of scheduler reservation and allocation. Compare text-only and image prompts as another control.

The reported timings identify a repeated runtime cost; they do not establish the same speedup for every vision model. The investigation is most useful when the slowdown follows image count, context, and ubatch together—the conditions that amplified reservation work in the upstream tests.

Sources and further reading