Mean KL can decide a fidelity rank. It is not the whole measurement system. QuantBench surrounds it with exact artifact identity, a native BF16 anchor, a pinned runtime, complete next-token distributions, reference-conditioned decision forensics, provenance-aware ranking classes, independent recomputation, and frozen evidence identities.
Do not ask only “did the score move?”Ask which exact model decisions moved, how far they were from the BF16 decision boundary, whether the artifact identity is actually distinct, and whether the result survives independent recomputation.
Seven gates before a result becomes a ranking
- 01
Artifact identity
Hash the exact GGUF and inspect tensor structure before benchmarking. Different filenames or file hashes do not automatically mean different quantized payloads.
- 02
Native BF16 anchor
Build the reference from the pinned native BF16 source instead of treating another quantized file as ground truth.
- 03
Runtime contract
Pin the llama.cpp revision, context geometry, batch geometry, KV representation, Flash Attention state, tokenizer behavior, and execution configuration.
- 04
Full-vocabulary capture
Compare the complete next-token distribution at every aligned scored position rather than judging only decoded text or one target probability.
- 05
Decision forensics
Measure not only whether the distribution moved, but where top-1 decisions flip relative to the BF16 decision margin.
- 06
Paired uncertainty
Resample independent segments as paired units so a ranking carries uncertainty rather than pretending a point estimate is self-validating.
- 07
Frozen evidence
Independently recompute the result from raw captures, freeze the release, bind it with SHA-256 identities, and publish the same machine-readable registry used by the site.
What this adds to a typical quant comparison
Perplexity, task scores, tokens per second, model size, bits per weight, and generated examples describe useful outcomes. QuantBench adds exact GGUF SHA-256 and tensor-payload preflight, full-vocabulary distributions at aligned positions, a native BF16 reference, versioned llama.cpp execution contracts, decision-flip forensics, paired uncertainty, and independently validated frozen releases. A filename or publisher alone is not an artifact identity.
Decision forensics: where the behavior changes
The fidelity layer asks where a quantized artifact changes a BF16 decision and conditions those failures on the BF16 decision margin.
High-Confidence Decision Flip Rate — HCDFRq(τ)
P(top1q ≠ top1BF16 | BF16 margin ≥ τ)
BF16 margin is the native reference logit gap between its top-1 and top-2 tokens. A flip near a tie is structurally different from overturning a large-margin winner. The margin is not semantic confidence or correctness.
Reference-Conditioned Flip Curve — RCFC
flip rate × fixed BF16-margin bins
This shows how artifact instability changes as the native BF16 reference moves from ambiguous decisions toward large-margin decisions.
Exclusive Failure Forensics
BOTH_STABLE · A_ONLY_FLIP · B_ONLY_FLIP · BOTH_FLIP_SAME_ALTERNATE · BOTH_FLIP_DIFFERENT_ALTERNATE
Two artifacts can have similar aggregate scores while failing on different positions. QuantBench keeps those disagreements visible instead of averaging them away.
Tensor identity: why different files may contain the same weights
Different GGUF files do not automatically represent different quantizations. Whole-file SHA-256 identifies the download. QuantBench also inspects tensor topology and tensor-payload identity before spending compute on behavioral comparison.
File hash → tensor topology → payload identity → eligible comparison.
Two files can have different whole-file GGUF hashes while containing byte-identical tensor payloads. QuantBench keeps file identity and quantized-weight identity distinct.