llama.cpp ARM BF16 Slow: Check the Scalar Fallback

If a BF16 model component is far slower than the same workload in F16 or Q8_0 on an ARM CPU, do not assume BF16 itself is inherently that slow.

Check whether llama.cpp is using a vectorized BF16 dot-product path at all.

Upstream issue #29917 remains open as of October 17, 2026 and identifies a specific case:

ggml_vec_dot_bf16 has optimized paths for several CPU families, but on some ARMv8.2 systems it can fall through to a scalar loop.

That can turn a format choice into a large performance cliff.

The symptom

The reported workload used a BF16 multimodal projector on a Helio G99 system with Cortex-A76/A55 cores.

The reporter measured roughly:

BF16: ~400 ms/token
F16:  ~35 ms/token
Q8_0: ~60 ms/token

Those numbers are from the upstream issue, not a RAMGPT reproduction.

The useful diagnostic signal is the relative gap:

BF16 dramatically slower than F16 and Q8_0
on the same ARM device

If you see that pattern, inspect the CPU implementation before spending time tuning thread counts or context settings.

The root cause reported upstream

The function is:

ggml_vec_dot_bf16

In current source, architecture-specific implementations exist for several instruction sets.

Issue #29917 points out that the affected ARM configuration lacks a matching NEON branch for this BF16 dot product and does not expose the newer native BF16 feature required for a more direct instruction path.

Execution therefore reaches the scalar tail loop.

Conceptually:

for every element:
    convert BF16 -> FP32
    multiply
    accumulate

That does useful work, but it does not exploit the SIMD width available to normal NEON vector code.

Why F16 can beat BF16 on the same CPU

“BF16 uses 16 bits” does not imply “BF16 runs like FP16.”

Storage width and arithmetic implementation are different questions.

A CPU may have an efficient vector path for F16 while a particular BF16 kernel still has to:

1. load BF16 values;
2. convert them;
3. execute scalar FP32 math;
4. repeat for every element.

So this comparison is possible:

same 16-bit storage width
different kernel support
very different runtime

That is why hardware feature support has to be evaluated at the operation level, not just by datatype name.

First confirm that the slowdown is format-specific

Run the same model component or closest comparable workload with another supported weight type.

Useful comparisons are:

BF16 vs F16
BF16 vs Q8_0

Keep the rest of the setup as constant as practical:

• same device;
• same llama.cpp build;
• same thread settings;
• same input;
• same model architecture;
• same offload choices.

If all formats are equally slow, this article is probably not your root cause.

If BF16 alone collapses, the kernel path becomes a strong suspect.

Check the CPU features

On Linux, start with:

lscpu
cat /proc/cpuinfo

You are looking for the actual ARM ISA/features exposed by the kernel and toolchain.

Do not assume that because the CPU supports NEON/ASIMD it automatically supports native BF16 dot-product instructions.

ARM BF16 extensions arrived later than baseline NEON.

Check how llama.cpp was built

A generic binary may target a conservative architecture.

If you build locally, verify the compiler target and the features reported in llama.cpp startup output.

The important distinction is:

CPU supports feature
vs
binary was compiled to use feature
vs
kernel has an implementation for feature

All three have to line up.

A rebuild cannot invent a kernel that does not exist in the source path, but it can matter when a vectorized implementation exists and the binary was compiled too generically to select it.

Do not “fix” this first with more threads

Thread count can hide the diagnosis.

If each thread is doing a slow scalar inner loop, adding threads may improve throughput somewhat while leaving the fundamental per-core inefficiency unchanged.

Before tuning:

threads
batch size
context
NUMA
affinity

first establish whether the BF16 vector dot path exists for your target.

A practical workaround today

If BF16 on your ARM target is hitting this scalar path, use a format with a proven optimized kernel for that device when model quality and compatibility allow it.

The upstream report found F16 and Q8_0 much faster on its test device.

That does not mean those exact ratios apply to every ARM CPU.

It means format selection should follow the kernels your actual hardware executes.

What an upstream fix would look like

Issue #29917 proposes adding an ARM NEON implementation that converts BF16 pairs to FP32 vectors and accumulates with vector FMA, with a possible native BF16 path on newer ARM versions.

Until such a change lands and is validated, users should treat the current behavior as architecture-dependent.

Fast diagnosis checklist

If llama.cpp BF16 is unexpectedly slow on ARM:

1. compare BF16 with F16 or Q8_0 on the same workload;
2. verify CPU ISA features with lscpu or /proc/cpuinfo;
3. inspect the llama.cpp build target and startup feature report;
4. confirm whether ggml_vec_dot_bf16 has a vectorized path for your target;
5. use an optimized alternative format if BF16 is falling back to scalar execution;
6. re-test after an upstream NEON/BF16 kernel lands.

The key lesson is simple:

a compact datatype is not automatically a fast datatype when the runtime lacks the matching vector kernel.