Qwen3.8-27B at 200K: The Performance Cliff Is VRAM, Not IQ3

A fresh LocalLLaMA post reported Qwen3.8-27B running above 200K context on a 16GB RTX 5060 Ti eGPU using UD-IQ3_XXS, llama.cpp, and q5_1 KV cache.

The headline was impressive, but the more interesting observation was the performance tradeoff. The author described moving from UD-Q3_K_XL at roughly 140K context with about 700-800 tok/s prompt processing to UD-IQ3_XXS above 200K context at roughly 400 tok/s.

That comparison changes two variables at once:

1. the model quantization changes from Q3_K_XL to IQ3_XXS;
2. the context grows substantially.

It also pushes a 16GB GPU very close to its memory limit.

So I reproduced the comparison on an RTX 4090 and separated those effects.

The result is cleaner than the original headline suggests:

IQ3_XXS itself is not responsible for the prompt-processing collapse. At matched context it was only about 1.5-2% slower than Q3_K_XL. The sharp slowdown appeared when the 200K workload could no longer remain inside the GPU memory budget.

On this machine, 200K prompt processing stayed effectively flat down to a 16 GiB GPU budget. Reducing that budget by only one more GiB cut throughput by almost 19%.

Test system

The RTX 3060 installed in the same machine was excluded from the benchmark.

No MTP model and no vision/mmproj component were used.

First control: does IQ3_XXS itself make prompt processing slow?

The first experiment kept context fixed and changed only the model quantization.

At 64K context:

IQ3_XXS was only 1.92% slower.

That is already enough to reject a simple explanation in which the IQ3 quant kernel itself causes anything close to the reported 40-50% drop.

The smaller quant reduces the model file by about 2.21 GB, or roughly 16.8%, while giving up less than 2% prompt-processing throughput at this context length.

The same result holds at 128K

At 128K context:

The relative difference is almost identical to the 64K result.

Even more striking, both quantizations lose essentially the same fraction of throughput when context doubles from 64K to 128K:

There is no sign here of an IQ3-specific long-context penalty.

At 200K, IQ3 still tracks Q3

The decisive matched-context test was 200K.

Again, IQ3_XXS is only slightly slower.

Across the entire controlled range:

That is unusually consistent.

A useful approximation for this RTX 4090 configuration is simply:

PP_IQ3(context) ≈ 0.98 × PP_Q3(context)

The large slowdown in the original 16GB report therefore needs another explanation.

The next variable: GPU memory residency

At 200K, the IQ3_XXS workload used roughly 16.3 GB of VRAM in the unrestricted run.

That is comfortable on a 24GB RTX 4090 but extremely close to the physical capacity of a 16GB card once CUDA context, compute buffers, and other allocations are included.

To isolate that effect without changing GPU architecture, I used llama.cpp’s GPU memory fitting control to progressively reduce the effective GPU memory budget while keeping all other benchmark settings fixed.

The command shape was:

CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-bench \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ3_XXS \
  -p 200000 \
  -n 0 \
  -ctk q5_1 \
  -ctv q5_1 \
  -fa on \
  -b 2048 \
  -ub 512 \
  -fitt <margin-MiB> \
  -r 1 \
  -o json

-fitt specifies a device-memory margin. On a 24,082 MiB GPU, increasing the margin progressively reduces the memory budget available to the workload.

The VRAM residency cliff

This is the main result.

The shape matters more than any individual number.

From 19 GiB down through 16 GiB, prompt processing is effectively flat:

19 GiB   1572.96 tok/s
17 GiB   1571.36 tok/s
16 GiB   1570.57 tok/s

Then one additional GiB produces a discontinuity:

16 GiB   1570.57 tok/s
15 GiB   1274.51 tok/s
          -18.85%

Below that threshold, throughput keeps falling as more of the execution path is forced outside the fast all-resident regime.

By a 12 GiB-equivalent budget, PP has fallen to 848.47 tok/s, almost 46% below the 16 GiB point.

This is much closer to the scale of the slowdown that motivated the reproduction.

The 16 GiB point is still full speed

One of the most useful observations is that the nominal 16 GiB point itself is not slower on this setup.

The unrestricted IQ3_XXS 200K result was:

1570.97 tok/s

The ~16 GiB-budget result was:

1570.57 tok/s

The difference is only about 0.03%.

So the threshold is not a smooth degradation beginning far above 16GB. The workload remains essentially at full speed until the effective budget falls below the amount required to keep the critical working set resident.

That distinction is important when interpreting claims such as “this model fits on 16GB.”

Two systems can both technically run the same model and context while sitting on opposite sides of a residency threshold because the remaining memory available after runtime overhead is different.

GPU telemetry shows the execution path changing

A repeated ~15 GiB-budget run produced 1281.51 tok/s, within about 0.6% of the first 1274.51 tok/s measurement.

During that run, GPU0 remained around 15,449 MiB used, confirming that the fitter was actually constraining the device-memory footprint.

The unrestricted run showed the 4090 near 99-100% utilization and roughly 448-450 W for long stretches.

Under the tighter memory budget, utilization repeatedly fell into the high-70s and 80s before returning to 100%, while power frequently dropped into roughly the 375-405 W range.

The verbose teardown also reported:

CUDA0 compute buffer size:       1379.8599 MiB
CUDA_Host compute buffer size:    225.9004 MiB

Those observations are consistent with the GPU spending less time continuously compute-bound once the workload is forced out of the fully resident regime.

They do not, by themselves, identify exactly which tensor transfers or host-side operations account for every lost token per second. Kernel-level tracing or explicit tensor-placement logging would be required for that attribution.

Separating the three penalties

The controlled experiments make it possible to decompose the original comparison into three distinct effects.

1. Quantization effect

At matched context, replacing Q3_K_XL with IQ3_XXS costs only about 1.5-2% PP on this RTX 4090.

2. Context-length effect

For IQ3_XXS:

128K: 1878.67 tok/s
200K: 1570.97 tok/s

That is a 16.38% reduction from the larger attention/KV workload alone.

3. Memory-residency effect

At 200K:

~16 GiB budget: 1570.57 tok/s
~15 GiB budget: 1274.51 tok/s
~12 GiB budget:  848.47 tok/s

The memory-budget effect is much larger than the intrinsic IQ3-vs-Q3 difference.

That changes the interpretation of the original observation.

Instead of:

IQ3 buys more context but is dramatically slower.

The controlled evidence supports a narrower statement:

IQ3 buys model-weight headroom with very little PP penalty. The large performance loss appears when the resulting long-context workload crosses a GPU-memory residency threshold.

Why the original 5060 Ti eGPU can still be much slower

This reproduction does not attempt to match the original hardware exactly.

The community report used a 16GB RTX 5060 Ti eGPU. This experiment uses a desktop RTX 4090 with a local PCIe connection.

That creates at least two major differences:

• the 4090 has substantially more compute throughput;
• any host-device traffic on an eGPU can face a very different interconnect bandwidth and latency profile than a desktop GPU.

The present experiment therefore does not prove that Thunderbolt or eGPU transfer overhead caused the original ~400 tok/s result.

What it does show is that a memory-residency cliff exists independently on a different system, and that crossing it can remove 19-46% of PP throughput even before reproducing the original eGPU topology.

A plausible hypothesis is that the original system combines the same basic residency pressure with slower GPU compute and a more constrained host-device path.

That should be tested directly rather than assumed.

What this reproduction establishes

It does show

• UD-IQ3_XXS is only about 1.5-2% slower than UD-Q3_K_XL at matched 64K, 128K, and 200K context on this RTX 4090.
• The smaller IQ3_XXS file saves about 2.21 GB versus Q3_K_XL.
• Both quantizations scale almost identically from 64K to 128K.
• IQ3_XXS still tracks Q3_K_XL closely at 200K.
• The 200K IQ3 workload remains essentially full speed down to an effective ~16 GiB GPU budget.
• Reducing that budget from ~16 to ~15 GiB cuts PP by 18.85%.
• At ~12 GiB, PP is down 45.98% relative to the 16 GiB point.
• The memory-budget penalty is far larger than the intrinsic IQ3 quantization penalty.

It does not show

• that every 16GB GPU will reach the same threshold at the same context;
• that -fitt perfectly emulates a physical lower-VRAM GPU;
• that eGPU interconnect overhead is the cause of the original post’s exact ~400 tok/s result;
• that IQ3_XXS preserves enough coding, tool-use, or reasoning quality to be the best practical quantization choice;
• or that the same residency curve applies to different KV types, batch sizes, llama.cpp builds, or model families.

The next question is quality, not another capacity headline

The systems result is now fairly clear: IQ3_XXS is an efficient way to recover memory headroom, and the dramatic slowdown is dominated by what happens when the long-context working set no longer stays resident on the GPU.

The remaining practical question is whether the smaller quant preserves enough model capability to justify using that extra context.

A useful follow-up would compare Q3_K_XL and IQ3_XXS on a small coding and tool-calling set at matched context, then test whether the additional context available to IQ3 improves end-to-end task success enough to offset any quantization-quality loss.

That is a better question than simply asking whether 200K technically fits.

Reproduction commands

Matched-context Q3_K_XL:

CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-bench \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q3_K_XL \
  -p 200000 \
  -n 0 \
  -ctk q5_1 \
  -ctv q5_1 \
  -fa on \
  -ngl 999 \
  -b 2048 \
  -ub 512 \
  -r 3 \
  -o json

Matched-context IQ3_XXS:

CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-bench \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ3_XXS \
  -p 200000 \
  -n 0 \
  -ctk q5_1 \
  -ctv q5_1 \
  -fa on \
  -ngl 999 \
  -b 2048 \
  -ub 512 \
  -r 3 \
  -o json

VRAM-pressure sweep:

for F in 4626 6674 7700 8722 9750 10770 11800; do
  CUDA_VISIBLE_DEVICES=0 ./build/bin/llama-bench \
    -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ3_XXS \
    -p 200000 \
    -n 0 \
    -ctk q5_1 \
    -ctv q5_1 \
    -fa on \
    -b 2048 \
    -ub 512 \
    -fitt "$F" \
    -r 1 \
    -o json
done

The main lesson from this reproduction is not that 200K context is free.

It is that model quantization, context scaling, and memory residency are separate costs, and combining them into one before/after number can point to the wrong bottleneck.