GSQ-RCO Beats Unsloth on WikiText-2 at ~8.4GB — But Not on Throughput

ISTA DASLab just released three Qwen3.8-27B GGUFs built with GSQ + RCO at 2.50, 2.75, and 3.00 bits per weight.

The headline claim is unusually strong for a 2.5-bpw model: at roughly the same 8.4GB file size as Unsloth’s UD-IQ2_S, the new GSQ-RCO-IQ2_XS quant reports substantially better reasoning and coding scores, while remaining a normal GGUF that runs unmodified in llama.cpp.

I wanted to test the part of that claim that can be reproduced quickly and cleanly on commodity hardware:

At nearly the same storage budget, does GSQ-RCO actually preserve more model quality than Unsloth Dynamic quantization?

On my RTX 4090 system, the answer is yes on WikiText-2 perplexity.

But the reproduction also exposed a more interesting tradeoff: GSQ-RCO is not simply a more aggressive low-bit quant. Its GGUF is a highly heterogeneous mixture of precisions, including IQ1, IQ2, IQ3, IQ4, BF16, and F32 tensors. That allocation improves perplexity in this test, but it is slightly slower than Unsloth on both prefill and decode.

The result looks less like a universal replacement for Unsloth Dynamic and more like a different point on the quality-size-speed Pareto frontier.

What GSQ and RCO are actually doing

There are two separate optimization problems here.

GSQ — Gumbel-Softmax Quantization — optimizes how an individual tensor is quantized at a chosen precision. Instead of treating the scalar quantization grid as fixed and merely rounding weights into it, GSQ jointly learns grid assignments and group scales through a Gumbel-Softmax relaxation.

The practical goal is important: recover more of the quality normally associated with vector or trellis quantization while still producing scalar formats that existing inference kernels can run.

RCO — Riemannian Constrained Optimization — operates one level above that. It decides which quantization type each tensor should receive while obeying a fixed total size budget.

Conceptually, the problem is not:

quantize the whole model to 2.5 bits

It is closer to:

fixed whole-model budget: ~2.5 bpw

Tensor A -> ~1-bit format
Tensor B -> ~2-bit format
Tensor C -> ~3-bit format
Tensor D -> ~4-bit format
Tensor E -> BF16
...

weighted average -> target file size

The ISTA model card describes a three-stage pipeline:

1. Quantize every weight tensor into multiple candidate GGUF formats using GSQ.
2. Let RCO search for one format per tensor under the global bit budget.
3. Stitch the selected tensors into a standard GGUF.

That last point matters. llama.cpp does not need a special GSQ or RCO runtime. The optimization happens offline. The final file contains ordinary GGUF tensor types that llama.cpp already understands.

The matched-size test

I compared the smallest GSQ-RCO release against the closest Unsloth Dynamic quant:

The GSQ-RCO file is 50,871,424 bytes larger, or about 0.61%.

This is close enough to be useful as a matched-storage comparison, although it is not literally byte-identical. That distinction is worth keeping explicit when interpreting small quality differences.

Test machine

The RTX 3060 installed in the same machine was excluded with CUDA_VISIBLE_DEVICES=0 so both models ran on the physical RTX 4090 only.

Perplexity: GSQ-RCO wins this reproduction

Using the same llama.cpp build, corpus, context, and GPU configuration, I measured:

That is an absolute reduction of 0.1315 PPL, or about 1.93% lower perplexity relative to the Unsloth result.

The direction matches ISTA’s published comparison. Their model card reports WikiText-2 perplexity of 7.69 for GSQ-RCO IQ2_XS and 8.02 for UD-IQ2_S.

My absolute values are lower for both models, so I would not claim an exact reproduction of ISTA’s reported numbers. The exact evaluation protocol and preprocessing details are not fully specified in the release material, and small differences in perplexity setup can move the absolute values.

The controlled result here is narrower and more defensible:

Under one identical llama.cpp WikiText-2 evaluation, the GSQ-RCO file achieved lower perplexity than the near-matched-size Unsloth Dynamic file.

The official advantage is roughly 4.1% on WikiText-2. This reproduction sees a smaller but still clear 1.93% advantage.

The GGUF dump explains why “2.5 bpw” is misleading

The most interesting part of the experiment was not the final PPL number. It was dumping the tensor types.

llama.cpp identifies the GSQ file’s coarse ftype as IQ2_XXS - 2.0625 bpw, yet the loader separately reports the actual file as 7.83 GiB / 2.50 BPW.

That apparent contradiction exists because the model is not uniformly IQ2_XXS. The file-type label does not describe the weighted precision of every tensor in a mixed-precision GGUF.

The loader counted all 851 tensors in the GSQ-RCO file as:

This is not a model where everything was simply pushed down one more quantization level.

It is a model where some tensors are crushed to around the 1-bit family while others are deliberately protected with much higher precision.

A representative tensor-level comparison

The first block already shows the difference in allocation philosophy:

This is the key structural observation.

GSQ-RCO is more aggressive in some places and more conservative in others.

It spends fewer bits on the embedding and some FFN tensors, but then spends some of that recovered budget on attention tensors and even leaves selected SSM parameters at BF16.

Calling this merely “a more aggressive quant” misses the main idea.

The novelty is the allocation policy: precision itself becomes an optimization variable under a global storage constraint.

Unsloth Dynamic quantization also uses mixed precision and protects sensitive tensors. The difference is not “mixed precision versus uniform precision.” The difference is that RCO explicitly formulates per-tensor precision assignment as a constrained optimization problem and searches the allocation under an exact overall budget.

Throughput: the quality gain is not free

The same models were also tested with llama-bench on the RTX 4090 with full GPU offload.

On this build and GPU, GSQ-RCO was approximately:

• 5.23% slower on pp512 prefill
• 1.73% slower on tg128 decode

That is not a large decode penalty, but it matters when describing what GSQ-RCO improves.

The evidence so far points to quality per storage budget, not raw inference throughput.

One plausible explanation is the highly heterogeneous mix of tensor formats. A precision allocation chosen to minimize model loss under a size constraint is not necessarily the allocation that best matches GPU kernel throughput.

That is an inference from the structure and measurements, not something this small experiment proves. Profiling individual kernels would be needed to attribute the speed difference properly.

The result is more nuanced than the release headline

ISTA’s broader published table is interesting because GSQ-RCO does not dominate every perplexity metric.

At the ~8.4GB point, the model card reports:

That is exactly why a single perplexity reproduction should not be turned into “GSQ wins everything.”

The more interesting hypothesis is that RCO is learning a precision allocation that preserves task-relevant capability better than a more conventional dynamic quant at the same broad storage class, even if that allocation does not minimize perplexity on every corpus and does not maximize throughput.

That is a much stronger research question than simply asking which GGUF is smaller.

What this reproduction establishes

It does show

• The 2.50-bpw GSQ-RCO GGUF loads and runs unmodified in current llama.cpp.
• Its actual file size is within about 0.61% of the compared UD-IQ2_S file.
• Under the same 4K-context WikiText-2 run, GSQ-RCO achieved 1.93% lower perplexity.
• The GSQ-RCO file uses an unusually heterogeneous tensor-level precision allocation.
• Some tensors are quantized more aggressively than UD, while others receive substantially more precision.
• On this RTX 4090, GSQ-RCO was slightly slower than UD-IQ2_S in both pp512 and tg128.

It does not show

• that GSQ-RCO is universally better across corpora,
• that the official absolute perplexity values are exactly reproducible with this llama.cpp command,
• that the ~0.61% file-size difference is irrelevant in every comparison,
• that the heterogeneous tensor mix itself causes the throughput loss,
• or that perplexity alone predicts reasoning and coding quality.

Those require more experiments.

The next test should be C4, not another speed run

The obvious next reproduction is already suggested by ISTA’s own table.

Their WikiText-2 result favors GSQ-RCO, but their C4 perplexity favors UD-IQ2_S.

That gives us a falsifiable follow-up:

Can we reproduce the same corpus-dependent reversal on the same machine and llama.cpp build?

If WikiText-2 again favors GSQ-RCO while C4 favors Unsloth, that would be more informative than another round of throughput measurements. It would show that the RCO allocation is not simply reducing generic language-model loss uniformly across datasets.

After that, the most valuable step would be a small task benchmark — preferably one close to the release claims — to test whether the perplexity advantage translates into the reasoning and coding recovery that motivated the quant in the first place.

Takeaway

GSQ-RCO at 2.50 bpw is not interesting because it somehow invented a new GGUF runtime format.

It did not.

The interesting part is what happens before the GGUF is assembled.

GSQ improves the low-bit scalar quantization available for each tensor. RCO then treats precision allocation itself as an optimization problem under a fixed total budget.

The result is a strange-looking model: IQ1 in places where conventional intuition might hesitate, IQ3/IQ4 where the optimizer thinks sensitivity is higher, and even BF16 for selected tensors — all inside a roughly 2.5-bpw whole-file budget.

On this independent RTX 4090 test, that allocation bought a measurable WikiText-2 quality gain over Unsloth UD-IQ2_S at nearly the same file size:

6.6645 vs 6.7960 PPL.

It did not buy speed:

2728.8 vs 2879.2 tok/s pp512, and 79.61 vs 81.01 tok/s tg128.

So the early evidence is not “GSQ-RCO is faster” or even “GSQ-RCO always wins.”

It is more specific:

RCO appears to move the ~2.5-bpw operating point toward better quality per byte by spending precision much less uniformly, while accepting a small throughput penalty on this GPU.

That is worth reproducing further.