We Beat Unsloth UD-Q4_K_XL on Qwen3-4B at Smaller Size

I started this experiment with a narrow question:

Can a custom llama.cpp quantization recipe beat a strong public Q4 artifact without simply making the model larger?

The first target was Unsloth’s standard Qwen3-4B Q4_K_M GGUF.

That turned out to be only the warm-up.

After calibration work and a small tensor-allocation search, RAMGPT produced a Qwen3-4B artifact that beat the standard Unsloth Q4_K_M at essentially identical size. The obvious next question was harder: what happens against Unsloth Dynamic, specifically Qwen3-4B-UD-Q4_K_XL.gguf?

That is a much stronger comparison. Unsloth Dynamic 2.0 is explicitly model-specific: Unsloth says it changes quantization types across layers, uses a curated calibration dataset, and optimizes layer selection rather than applying one uniform recipe to every model.

On the final 65,536-position test, the custom RAMGPT recipe still won.

And it did so while producing a slightly smaller GGUF.

The RAMGPT artifact was 2,611,328 bytes smaller, a difference of about 0.10%, while reducing mean full-vocabulary KL by 4.72% relative to Unsloth UD-Q4_K_XL.

Across 128 complete segments, RAMGPT had lower mean KL on 86 and Unsloth UD on 42. A paired segment bootstrap gave a 95% interval of:

[0.0007964, 0.0024242]

for mean_KL_UD - mean_KL_RAMGPT, with an empirical bootstrap probability of RAMGPT being better of 0.9999.

That is the final result.

But the route to it is more interesting than the headline.

Why KL divergence was the primary metric

This experiment is about behavioral fidelity to the native BF16 model, not downstream benchmark accuracy.

For every scored position, we captured the entire next-token logit vector over the Qwen3-4B vocabulary and compared the quantized distribution against BF16.

The primary metric was:

KL(BF16 || quantized artifact)

Lower is better. Zero would mean the quantized next-token probability distribution exactly matches the BF16 reference at that position.

This is stricter than looking only at whether the top token changed. A quantized artifact can keep the same argmax while materially moving probability mass among the rest of the vocabulary. Full-vocabulary KL measures that distributional movement.

Top-1 flips were retained as a secondary metric.

The final RAMGPT v3 result improved both:

Mean KL:
Unsloth UD   0.0340240914
RAMGPT v3    0.0324186488

Top-1 flips:
Unsloth UD   5,293
RAMGPT v3    5,226

The median, P90 and P99 KL values also all moved in the same direction, so the mean improvement was not produced by a single isolated tail event.

The frozen execution contract

All quantization and evaluation work was pinned to the same llama.cpp revision:

70adb1b4cea5ee39f867792c78dc59320921eda7

The canonical BF16 GGUF was:

models/Qwen3-4B-BF16.gguf
bytes: 8,051,285,408
SHA256: 47d6654f04449d367b84da24187597a76cbf66a31e07a6665d29844f2161bf77

The logit-capture contract was held constant:

Each full RQL logit file was exactly:

39,829,111,105 bytes

That corresponds to 65,536 positions times 151,936 FP32 logits plus the fixed RQL v2 header.

Stage one: the obvious calibration was not enough

The first RAMGPT attempt used a simple WikiText-derived calibration corpus.

It produced a Q4_K_M artifact at essentially the same size as Unsloth’s standard Q4_K_M, but on an 8,192-position development run it was worse:

Unsloth standard Q4_K_M   0.03684897
RAMGPT fast-v0            0.03861684

The failure was useful because it was highly concentrated in particular domains: dialogue, structured data, Chinese text, mixed English/Chinese, code and math.

That pointed directly at the calibration distribution.

The next calibration set was deliberately broad. It kept the same 685 × 512 chunk geometry but mixed approximately:

• 30% English prose
• 20% dialogue and instruction data
• 15% Chinese
• 10% mixed English/Chinese
• 15% code and structured data
• 10% math

The resulting importance matrix contained 252 entries.

This alone moved the model dramatically closer to Unsloth at the same artifact size.

On 8,192 positions:

Unsloth standard Q4_K_M   0.03684897
RAMGPT fast-v1            0.03699087

The gap had fallen to only about 0.39%.

The important lesson was not that broad calibration automatically wins. It was that the original loss was largely a calibration-distribution problem, not an unavoidable property of llama.cpp Q4_K_M.

Stage two: tensor sensitivity exposed an unusually valuable target

The next step was a deliberately coarse sensitivity sweep.

Instead of tuning hundreds of individual tensors, I promoted whole tensor families from their baseline Q4 allocation to Q5 and measured KL improvement per added MiB.

The clean 4,096-position sweep looked like this:

attn_k was the obvious first move.

It cost only 11.25 MiB and produced the best improvement per added byte by a wide margin.

On the full 8,192-position development sample, promoting all 36 attn_k tensors to Q5 reduced mean KL to:

0.0345052456

That was 6.36% below the standard Unsloth Q4_K_M result on the same positions.

The problem was size.

A fair recipe needed to pay for those extra bits somewhere else.

Stage three: fund the useful bits instead of growing the model

llama.cpp’s importance-matrix statistics expose activation-derived information for each tensor. The statistic used for the quick layer ranking here was Σ(Act²), the sum of squared activations.

I used it only as a coarse ranking heuristic inside the attn_q family, then let KL decide whether the resulting artifact was actually good.

The nine lowest-ranked attn_q layers were:

0, 1, 2, 3, 4, 5, 6, 7, 13

Those nine tensors were demoted from Q4_K to Q3_K, funding the all-layer attn_k Q5 promotion.

The resulting RAMGPT v2 recipe was:

36 × attn_k: Q4_K → Q5_K
9 × low-activation attn_q: Q4_K → Q3_K
all remaining tensors: default Q4_K_M behavior

Its artifact size was:

RAMGPT v2             2,496,543,904 bytes
Unsloth standard Q4   2,497,281,312 bytes

RAMGPT v2 was 737,408 bytes smaller.

On the full 65,536-position run against the standard Unsloth Q4_K_M artifact:

Unsloth standard mean KL   0.0385423075
RAMGPT v2 mean KL           0.0364638984
relative reduction          5.3925%

Unsloth top-1 flips         5,699
RAMGPT v2 top-1 flips       5,525

At that point the original experiment had succeeded.

Then I asked the more important question: what about Unsloth UD?

Unsloth UD was a much stronger opponent

Unsloth’s Dynamic v2.0 methodology is explicitly designed around model-specific mixed precision. Unsloth describes revamped layer selection, custom quantization schemes for individual models, and a large curated calibration set.

The Qwen3-4B repository publishes:

Qwen3-4B-UD-Q4_K_XL.gguf

The exact artifact used in this comparison was:

2,546,341,152 bytes

On 8,192 positions, it immediately beat RAMGPT v2:

RAMGPT v2 was about 7.32% worse than UD on mean KL.

That was not a close result, and it should not be described as one.

But the size difference suggested a clean next experiment.

UD was 49,797,248 bytes larger than RAMGPT v2.

The earlier sensitivity sweep had already measured the cost of promoting all 36 attn_output tensors from Q4 to Q5:

+47,185,920 bytes

That was almost a perfect fit inside the UD size budget.

So rather than starting another broad search, I added exactly that promotion.

RAMGPT v3: the UD-budget recipe

The final recipe is surprisingly small:

36 × attn_k:
Q4_K → Q5_K

36 × attn_output:
Q4_K → Q5_K

9 × attn_q, layers 0,1,2,3,4,5,6,7,13:
Q4_K → Q3_K

Everything else:
default Q4_K_M heuristic

The resulting artifact was:

RAMGPT v3    2,543,729,824 bytes
Unsloth UD   2,546,341,152 bytes

RAMGPT remained smaller by:

2,611,328 bytes

or approximately 0.10%.

This is close enough that the comparison is not being driven by a materially larger bit budget on the RAMGPT side.

The development checks were already encouraging

At 4,096 scored positions:

Unsloth UD mean KL   0.0332628123
RAMGPT v3 mean KL    0.0306044112
RAMGPT reduction     7.99%

The paired segment bootstrap interval for UD - RAMGPT was already entirely above zero:

[0.0003245, 0.0052946]

At 8,192 positions:

Unsloth UD mean KL   0.0330798328
RAMGPT v3 mean KL    0.0310291405
RAMGPT reduction     6.20%

with:

95% CI: [0.0002702, 0.0039045]
P(RAMGPT better): 0.9877

That was strong enough to stop tuning and run the full position set.

Final 65,536-position result

The final run scored all 512 available positions in each of 128 segments.

The complete result was:

Mean KL fell by:

4.7185%

relative to Unsloth UD.

Segment head-to-head results were:

RAMGPT better: 86 / 128
Unsloth UD better: 42 / 128

The mean paired segment delta was:

mean_KL_UD - mean_KL_RAMGPT
= 0.0016054427

A 20,000-replicate paired bootstrap using complete segments as the resampling unit produced:

95% CI:
[0.0007963916, 0.0024242161]

P(RAMGPT better):
0.9999

The interval is entirely above zero.

Within this evaluation set and execution contract, the RAMGPT recipe therefore produced a lower mean BF16-to-artifact KL than Unsloth UD-Q4_K_XL at a slightly smaller whole-file size.

What I think the experiment actually shows

The important result is not that attn_k should always be Q5 or that early attn_q layers should always be Q3.

That would be the wrong generalization.

The useful result is that bit allocation can be treated as an optimization problem under a fixed artifact-size budget.

A nominal quantization label like Q4_K_M is only the starting point. Different tensor families can have radically different marginal value per byte.

In this Qwen3-4B experiment:

• attn_k was exceptionally cheap to promote and unusually valuable per added MiB.
• attn_output was less efficient than attn_k, but still valuable enough to consume the additional budget available when matching UD-Q4_K_XL.
• selected attn_q layers provided a way to recover bytes with an acceptable fidelity penalty.
• calibration quality mattered enough to move the baseline from clearly losing to nearly matching a strong public Q4 before any tensor-family optimization was added.

The recipe emerged from those measured trade-offs rather than from a fixed belief that one tensor class always deserves more bits.

That distinction matters.

A major limitation: this is not an untouched external holdout

There is an important methodological caveat.

The same 128 segment sources were used during recipe development.

The early sensitivity work scored only the first 32 or 64 positions per segment, while the final run expanded evaluation to all 512 scored positions per segment. That means most final positions were not directly used during the early tuning loops, but the documents and domains were already known.

So the 65,536-position result is a position-expanded evaluation on the development corpus, not an untouched external holdout.

That is why I am comfortable making the narrow claim that RAMGPT v3 wins this frozen comparison, but I am not treating the result as proof that the recipe will dominate Unsloth UD on every unseen corpus or downstream task.

The next scientifically stronger step would be to freeze v3 permanently and run both artifacts on a new corpus that played no role in calibration, tensor selection or recipe iteration.

No more tuning would be allowed after seeing that new corpus.

What this does not claim

This benchmark does not show that RAMGPT v3 is universally a better model.

It does not test:

• MMLU or MMLU-Pro
• coding benchmark accuracy
• instruction-following quality
• long-context degradation
• tool calling
• multilingual task scores
• subjective chat preference

It measures one thing precisely: next-token distributional fidelity to a frozen BF16 Qwen3-4B reference under a fixed llama.cpp execution contract.

That metric is useful because it exposes quantization damage directly, but it is not semantic ground truth.

A lower KL does not guarantee a higher score on every downstream benchmark.

Why the UD comparison matters

Beating a standard Q4_K_M artifact was useful, but it was not enough to establish that the tensor-aware strategy was competitive with more sophisticated public quantization work.

Unsloth Dynamic is specifically built around the same general idea that motivated this experiment: different parts of a model deserve different quantization treatment.

That makes UD-Q4_K_XL the more meaningful opponent.

The final comparison is therefore much more interesting than the original standard-Q4 result:

Unsloth UD-Q4_K_XL
2,546,341,152 bytes
mean KL 0.0340240914

RAMGPT v3
2,543,729,824 bytes
mean KL 0.0324186488

RAMGPT did not win by spending more bytes.

It won with a file that was fractionally smaller.

The practical takeaway

If you are building GGUFs, the experiment suggests a more useful workflow than asking only:

Which global quant type should I choose?

A better sequence is:

1. build a calibration set that resembles the real distribution you care about;
2. generate an importance matrix;
3. establish a frozen reference and a distributional fidelity metric;
4. measure coarse tensor-family sensitivity;
5. express every promotion in both KL gain and bytes added;
6. fund expensive promotions with the least damaging demotions;
7. keep the final artifact under a fixed size budget;
8. stop tuning before the evaluation corpus turns into a lookup table.

llama.cpp already exposes the machinery needed for much of this. Its quantizer supports an importance matrix and per-tensor type overrides, while llama-imatrix --show-statistics exposes activation-derived statistics that can help prioritize candidate layers.

The missing piece is the search discipline around those tools.

For Qwen3-4B, that search produced a three-line idea:

attn_k      gets more bits
attn_output gets more bits
selected attn_q layers pay for them

That was enough to move from losing to standard Q4, to beating standard Q4 at the same size, to finally beating Unsloth UD-Q4_K_XL at an almost identical size budget.

For the final measured result, RAMGPT v3 reduced mean full-vocabulary KL by 4.72%, reduced top-1 flips from 5,293 to 5,226, improved median/P90/P99 KL, and remained 0.10% smaller than the UD artifact.

The next test should not be another tweak to v3.

It should be a new, untouched corpus.