Qwen3.8's 38GB PLE: Only 222MB Resident at 8K Context

A 38GB table sounds like a 38GB memory problem.

That is not what I saw with Qwen3.8-Flash-Next.

I reproduced the AtomicChat PLE setup on an RTX 4090 Linux box, isolated the model’s dedicated PLE shard, and watched how many of its pages actually became resident as prompt length increased.

The file is 38,400,184,512 bytes: about 38.4GB decimal / 35.8GiB.

After a progressive high-diversity prompt sweep all the way to 8,192 tokens, Linux had only 221.7MiB of that PLE resident in page cache.

That is roughly 0.60% of the full table.

The more interesting result, though, came from comparing different 512-token workloads. A repetitive natural-language prompt left the PLE at about 2MiB resident, while a separate high-diversity synthetic 512-token test reached 25.3MiB.

Same nominal prompt length, more than an order-of-magnitude difference in resident PLE.

That makes the PLE look much less like conventional model weights being “offloaded to SSD” and much more like a giant sparse, file-backed lookup structure whose physical working set depends on the token/ngram access pattern.

Test machine

This was not a huge-memory server.

The server flags for the initial 8K-context run were:

-ngl 16 -fit off --tensor-read-lazy on -c 8192 -fa on --parallel 1

For the 512-to-8K scaling sweep I restarted with a 16,384-token context so the 8,192-token prompt would not sit directly on the context boundary. The other relevant flags stayed the same.

Normal mmap behavior was left enabled.

The RTX 3060 also installed in the machine was not used for model compute; CUDA_VISIBLE_DEVICES=0 isolated the llama.cpp process to the physical RTX 4090.

First check: does loading the model materialize the PLE?

Before loading the model I dropped Linux page cache and verified the PLE shard was effectively cold.

Once llama.cpp had loaded the model, the PLE was still almost completely untouched.

In one clean post-load measurement:

• PLE file: 35.8GiB
• PLE resident: 272KiB
• RTX 4090 VRAM: about 18.9GB
• llama-server PSS: about 35.7GiB
• swap: 0

So model startup did not materialize the 35.8GiB PLE into physical memory.

That distinction matters because mmap gives the process an address range for the file without requiring every mapped page to be resident at once.

The logical size of the table and its physical working set are separate quantities.

A 512-token prompt initially touched almost nothing

For the first inference test I built an exact 512-token prompt from repeated technical natural-language text.

Cold run:

Then I ran the exact same prompt again without dropping page cache.

The warm run was about 9.7% faster, but the important observation for this experiment was simpler: the second pass did not need to grow the PLE working set.

At first glance, ~2MiB resident from a 38.4GB table looked almost too good.

It was.

Not because the measurement was wrong, but because the workload was unusually repetitive.

Making the prompt deliberately harder on locality

PLE access is driven by token history, so repeating similar language can repeatedly hit the same region of the lookup structure.

For the next test I generated a deterministic synthetic corpus designed to create more varied histories. It mixed:

• technical vocabulary,
• changing record IDs,
• random numeric values,
• random word combinations,
• and unique short hashes.

The random seed was fixed so the source stream was reproducible.

I tokenized that corpus once and then used prefixes from the exact same token stream:

512 -> 1024 -> 2048 -> 4096 -> 8192 tokens

I intentionally did not drop page cache between these steps.

Because each larger test contains the previous prefix, the experiment measures the growth of the resident union of PLE pages as the accessible token history expands.

The 512-to-8K PLE residency curve

Here are the measured results:

After the full sweep, fincore reported:

RES     PAGES   SIZE
221.7M  56750   35.8G

The full PLE file was 35.8GiB, but only 221.7MiB was resident.

Using the binary units reported by the tooling, that is about 0.60% of the table.

Context grew 16x; resident PLE grew about 8.8x

From the first synthetic point to the last:

• context increased from 512 to 8,192 tokens: 16x
• resident PLE increased from 25.3MiB to 221.7MiB: ~8.8x

The incremental resident growth between measured points was:

I would not generalize that curve into a universal scaling law from one synthetic stream. The test is too small for that.

What it does establish is that an 8K token history did not require anything close to 35.8GiB of resident PLE on this workload.

Prompt diversity matters more than I expected

The most useful comparison may actually be the two separate 512-token experiments.

One repetitive natural-language workload ended around:

~2MiB resident PLE

The higher-diversity synthetic workload ended at:

25.3MiB resident PLE

These runs were performed separately, and the server context setting differed between the initial test and the later sweep, so I am not treating the ratio as a controlled performance benchmark.

But the page-residency difference is large enough to motivate a clear next question:

What properties of a prompt make the PLE working set large?

Context length alone is not a sufficient description of this system.

Token/ngram locality appears to matter substantially.

Why “SSD offload” is an incomplete mental model

A conventional offload description suggests a hierarchy like this:

GPU -> RAM -> SSD

That framing makes it sound as if a 38GB table is fundamentally 38GB of model state that merely happens to be stored on a slower tier.

The observed behavior is different.

The PLE looks more like a large file-backed lookup address space:

1. llama.cpp maps the file.
2. Linux provides virtual addresses covering the table.
3. Inference touches only some of those addresses.
4. The kernel faults the required pages into page cache.
5. Untouched regions consume essentially no physical RAM.
6. Repeated accesses can reuse already resident pages.

The more useful capacity question is therefore not simply:

How big is the PLE file?

It is:

How large does the active PLE working set become for the workloads I care about, and can my storage path service misses quickly enough?

Those are very different questions.

The OS pager is now part of the inference system

This architecture pushes something normally hidden in the operating system directly into inference performance.

For conventional dense weights, model placement is usually discussed in terms of VRAM and RAM capacity.

For a large sparse lookup table, the kernel’s virtual-memory behavior becomes part of the runtime:

• page-cache residency,
• major page faults,
• filesystem readahead,
• NVMe latency and queue behavior,
• access locality,
• and cache eviction pressure.

That means two machines with the same GPU and nominal RAM capacity could behave quite differently depending on storage and memory pressure.

It also means a benchmark that reports only “model size” and “RAM used” can miss the mechanism that actually matters.

A llama.cpp CUDA detour

The reproduction also exposed an unrelated CUDA problem.

My first 512-token inference attempt crashed with:

CUDA error: an illegal memory access was encountered

The kernel log reported NVIDIA Xid 31 with an MMU virtual-read fault.

This matched open llama.cpp issue #27792, which describes a CUDA MMQ MUL_MAT_ID tail-padding problem where some ubatch layouts can read beyond the allocated src1_q8_1 buffer.

The issue proposes changing the padding calculation from ne11 to the actual row count the kernel may touch:

- ggml_cuda_mmq_get_J_max(src0->type, fallback, cc, ne11)
+ ggml_cuda_mmq_get_J_max(src0->type, fallback, cc,
+                         std::max<int64_t>(ne_get_rows, 128))

I applied that one-line patch and rebuilt the same llama.cpp commit before collecting the successful inference measurements reported here.

That is worth documenting because otherwise someone reproducing the same setup could mistake the CUDA crash for a PLE or SSD-paging failure.

This experiment does not prove that #27792 is the only possible cause of every similar CUDA crash. It is simply the patch used in this reproduction, and it eliminated the failure I encountered.

What this experiment does and does not show

It does show

• Loading the model does not require materializing the full 35.8GiB PLE.
• PLE residency grows on demand during inference.
• A high-diversity 8K prefix sweep reached only 221.7MiB resident PLE on this machine.
• Two different 512-token workloads produced dramatically different resident working sets.
• Repeating the same initial 512-token prompt did not grow the PLE further in the warm run.

It does not show

• that every 8K prompt will stay near 222MiB,
• that 222MiB is a worst case,
• that NVMe I/O has negligible performance cost,
• that the measured curve generalizes to other PLE configurations,
• or that file residency alone explains end-to-end throughput.

Those require additional experiments.

In particular, fincore reports system-wide page-cache residency for the file. It is direct evidence about which file pages are resident, but it is not the same metric as process RSS.

Similarly, whole-device NVMe counters can be contaminated by unrelated I/O, so attributing exact SSD bytes to a particular PLE fault needs more careful tracing.

The next benchmark should measure misses, not just residency

The obvious follow-up is a matched cold/warm experiment that records four things together:

1. PLE residency,
2. major page-fault deltas,
3. NVMe read traffic,
4. prefill throughput.

Then repeat it across different access patterns:

• normal prose,
• code,
• multilingual text,
• random tokens,
• deliberately high-diversity token histories.

That would tell us whether the generic Linux pager is merely making the model fit, or whether it is also making SSD-backed PLE access fast enough to be a practical inference strategy.

The takeaway

The headline number is easy:

38.4GB addressable PLE, 221.7MiB resident after this 8K high-diversity sweep.

But the more important result is the workload dependence.

A PLE should not be evaluated like a normal block of model weights. Its logical size can be enormous while its physical working set remains much smaller, and that working set changes with the token histories being queried.

For this kind of model, the interesting question is no longer just:

Does the table fit in RAM?

It is:

How good is the pager, and how much of the table does my workload actually touch?