When a 10GbE Model Server Crawls Below 1 Token/s

A 10GbE link, NVMe-backed client cache, and a server-side SSD cache sound like enough storage infrastructure for local inference. Yet a new llama.cpp report describes Qwen4 experimental inference falling below one token per second on exactly that kind of setup.

The interesting part is not that network storage can be slower than RAM. Everyone knows that. The interesting part is that the reporter says the filesystem cache is hitting 100%, yet performance still collapses until the model is forced into memory with vmtouch.

That points away from raw bandwidth and toward a more subtle systems problem: I/O granularity and access pattern can matter more than headline storage throughput.

The Reported Setup Should Not Look Slow on Paper

Issue #28256 was opened in llama.cpp on September 2. The reporter stores models on a remote NFS server connected over 10GbE. The client uses NVMe-backed FS-Cache, while the server stores the models on an HDD RAID array fronted by an SSD cache.

For ordinary large sequential model reads, this is a fairly serious storage stack:

llama.cpp
   |
   v
NFS client
   |
NVMe FS-Cache
   |
10GbE
   |
server SSD cache
   |
HDD RAID

But with a model using the Qwen4 experimental n-gram embedding mechanism, the reporter observes “pathologically small” reads into the model file during inference. Performance drops to far below 1 token/s even when FS-Cache reports a 100% hit rate.

Forcing the GGUF into RAM with vmtouch removes the problem.

This is an upstream user report, not a RAMGPT benchmark, and the issue remains open. We should therefore treat the diagnosis as an important lead rather than a proven universal llama.cpp defect.

A Cache Hit Is Not the Same Thing as a Cheap Read

The phrase “100% cache hit” is easy to misread as “storage is no longer relevant.”

That is not necessarily true.

A cache can eliminate expensive trips to the backing HDD array while still requiring the application to perform a huge number of tiny operations through the filesystem, page cache, network filesystem machinery, or block-cache layer.

A simplified comparison looks like this:

Pattern A:
read 64 MB
read 64 MB
read 64 MB

Pattern B:
read 4 KB
read 4 KB
read 4 KB
... thousands or millions of times

Both patterns can move the same number of bytes. They do not impose the same cost.

With sufficiently small accesses, fixed per-operation overhead becomes dominant. Latency, syscall handling, metadata bookkeeping, cache lookup, page-fault behavior, and remote-filesystem semantics can overwhelm the time spent transferring the actual bytes.

That means a 10GbE link can be mostly idle while inference remains I/O-bound.

Why N-Gram Embeddings Change the Storage Question

Traditional dense transformer inference encourages a fairly simple mental model of model weights: load large tensors, keep the hot working set resident, repeatedly run matrix operations over them.

The experimental Qwen4 path complicates that model. An n-gram embedding table can be extremely large, while each inference step may need only selected entries from it.

That creates an attractive memory-saving possibility: leave the large table file-backed and touch only the pieces required for current tokens.

Conceptually:

huge embedding table on storage
          |
          +--> token/ngram selects a small region
          |
          +--> fetch selected entry
          |
          +--> continue inference

This is elegant when the memory hierarchy serves those accesses cheaply.

It can be disastrous when the access pattern becomes a long sequence of fine-grained reads whose fixed costs exceed the useful computation between them.

The architecture has effectively transformed storage from a startup concern into part of the token-generation critical path.

mmap Does Not Magically Remove I/O

Local inference discussions often treat memory mapping as though it converts a file into RAM. It does not.

mmap gives the process a virtual-memory view of a file. The operating system still has to make the required pages resident. If the working set is not already in RAM, page faults and storage activity occur as the model touches new regions.

For a conventional weight tensor traversed predictably, the kernel’s readahead behavior and page cache can work well.

Sparse or irregular table lookups are different. The system may have much less opportunity to turn demand into efficient sequential I/O.

This explains why the vmtouch observation is so important. If forcing the whole GGUF resident makes the problem disappear, the compute kernels themselves become less convincing as the primary bottleneck.

The bottleneck is somewhere in the path between a model’s logical lookup and a resident physical page.

The Infrastructure Lesson: Bandwidth Is the Wrong First Metric

Platform engineers naturally describe storage with bandwidth numbers:

• 10GbE networking
• multi-GB/s NVMe
• SSD caching
• RAID throughput

Those numbers matter, but they describe the system best when workloads can generate large enough requests and enough concurrency to use the available bandwidth.

For sparse inference-time access, I would want a different dashboard:

I/O operations per generated token
average request size
page faults per token
cache lookup latency
storage wait per token
resident pages of the embedding table

The useful unit is no longer MB/s. It may be storage transactions per token.

That is a significant shift in how we should benchmark storage-backed model architectures.

Why This Matters Beyond NFS

It would be easy to classify #28256 as an NFS problem and move on. That would miss the larger point.

If an architecture depends on sparse accesses into a huge file-backed table, the same sensitivity can appear at several levels:

remote NFS
local SATA SSD
local NVMe
OS page cache
compressed filesystem
virtualized storage
container volume

The threshold will differ dramatically, but the underlying question is the same: does the runtime request data in a way that the memory and storage hierarchy can serve efficiently?

A local NVMe drive may hide the problem well enough that nobody notices. A remote filesystem can amplify it until the access pattern becomes obvious.

In that sense, NFS is useful here as a diagnostic magnifying glass.

What I Would Want llama.cpp to Expose

For models with giant auxiliary tables, inference telemetry should distinguish compute stalls from residency stalls.

At minimum, operators should be able to answer:

1. How much of the model file is resident?
2. Which tensor or table is faulting pages during generation?
3. How many page faults or read operations occur per token?
4. What is the average read size?
5. Does prefetching the relevant table change token throughput?

Without those signals, a user can easily blame GPU utilization, quantization, networking, or the model architecture while the real problem is thousands of tiny data fetches.

A future optimization might involve explicit prefetch, batching adjacent n-gram lookups, changing table layout, using a more storage-friendly index, or deliberately pinning part of the table in RAM. The current issue does not establish which solution is correct, so claiming one would be premature.

The Bigger Architectural Trade-Off

Large sparse structures are appealing because they let model designers buy capacity without requiring every parameter to participate in every token.

But sparsity moves complexity into routing and memory access.

For MoE, we already think about expert routing and whether selected experts live on GPU, CPU, or slower memory. Huge n-gram embedding tables introduce a related question at a different granularity:

How cheap is it to reach the parameter you decided not to keep resident?

A parameter that is mathematically inactive costs nothing. A parameter that is active but physically distant can cost more than expected.

That means the practical performance of future hybrid models may increasingly depend on data locality, not simply FLOPs or total parameter count.

Bottom Line

Issue #28256 is still an open report, so it is too early to declare a confirmed llama.cpp root cause. But its symptoms are unusually instructive: a sophisticated cached NFS setup, 100% reported cache hits, sub-1-token/s performance, and immediate relief when the GGUF is forced into RAM.

The lesson is broader than Qwen4 experimental support.

When model architecture turns a giant table into a sparse inference-time lookup structure, storage stops being something that merely loads the model. It becomes part of the execution engine.

And once storage enters the per-token critical path, request granularity, locality, and residency can matter far more than the bandwidth number printed on the box.