We Tried to Break a 27B Q4 Model. The Benchmark Broke First.

I started RAMGPT ExactBench with a simple goal: build small, deterministic reasoning problems that could make a local 27B Q4 model finish naturally and still give the wrong answer.

That sounds easier than it is.

The target was a locally served Qwen3.8-27B-UD-Q4_K_XL.gguf running through llama.cpp with native reasoning_effort=low. I did not want to win by forcing a huge search, exhausting the context window, or giving the model an underspecified prompt. The useful failure case was deliberately narrow:

small problem
explicit semantics
cheap oracle
short certificate
natural completion
wrong final answer

After several generations of benchmark design, the surprising result was not a spectacular model failure. It was the opposite.

Across the three methodology-cleaner rounds described below, the model answered all 94 scored items correctly. The final two rounds alone produced 64/64 verified natural completions with no token-cap failures, no format failures, and no clean wrong answers.

The more important story, however, is what failed before the model did: several benchmark designs that initially looked like genuine reasoning failures turned out to contain semantic or measurement bugs.

This is a negative result, but it is a useful one.

What ExactBench was trying to measure

The benchmark was not designed to test obscure knowledge or brute-force search. The target region was intentionally constrained:

oracle computation       cheap
verification             cheap
correct certificate      short
search complexity        low
semantic specification   exact
grading                   deterministic
model inference           difficult

A request that consumes 8,192 tokens and never produces an answer is not the same thing as a reasoning error. It may simply be a resource-limit event.

So I separated a clean wrong answer from a censored run.

The useful event was:

answer = wrong
termination = natural
format = valid

In the later harnesses I called that a natural_W.

This distinction killed many apparently difficult tasks very quickly.

The first failure mode: making the problem larger only measured endurance

The earliest ExactBench versions used ordinary exact reasoning tasks:

• state tracking
• conditional routing
• Boolean satisfiability
• candidate verification
• proof verification
• assignment optimization
• exact knapsack
• maximum independent set
• set cover
• traveling-salesman variants
• recurrence enumeration
• counterexample search

Small versions were often solved almost perfectly. Increasing their size eventually made the model run for thousands of tokens and hit the output ceiling.

That is a real system limit, but it is not the failure mode I wanted.

The pattern was roughly:

small instance
model knows the algorithm
        ->
larger instance
more bookkeeping and search
        ->
large instance
reasoning budget or completion ceiling dominates

This led to the first ExactBench rule:

High inference difficulty should not be manufactured primarily through high computational burden.

A short certificate is not enough. The certificate may be short while discovery remains expensive. SAT is the obvious example: a satisfying assignment can be tiny, but finding it can still require substantial search.

The benchmark broke first: the hidden mod-10 rule

One adversarial miner initially produced what looked like an excellent set of failures.

The Python oracle applied modulo 10 after every register assignment. The model produced larger integer values and disagreed with the gold answer. Several items survived the first filter as normal-completion wrong answers.

Then I audited the prompt.

The prompt had explicitly defined a modulo-10 snapshot expression, but it had not clearly stated that every subsequent assignment was also reduced modulo 10.

The oracle knew a rule the model had never been told.

The apparent model failures were invalid.

The correct diagnosis was:

prompt semantics != oracle semantics

not:

model reasoning != correct reasoning

Every survivor from that family was discarded.

The second benchmark bug: defeasible rules and an unstated convention

A later priority_override family looked even more promising.

Out of 25 items, five finished normally with wrong answers at roughly 1,100 to 1,300 completion tokens. This looked close to the desired difficulty range.

The prompt stated that strict support could not be defeated by a defeasible rule. The oracle interpreted that narrowly: an undefeated defeasible argument for the opposite conclusion could still survive, leaving both conclusions warranted.

The model repeatedly imported another plausible convention: strict support also defeats an opposing defeasible rule.

Its visible analysis literally appealed to “standard defeasible logic.”

The problem was that the prompt had never defined the converse direction.

Again, the model and the oracle were operating under different but defensible semantics.

Those five wrong answers were also discarded.

This became the central benchmark-engineering rule:

Every semantic assumption used by the oracle must be represented explicitly in the item contract.

From handwritten prompts to semantic contracts

After those failures, the benchmark architecture changed.

Instead of maintaining one handwritten English problem and a separately handwritten Python oracle, later miners were based on a typed semantic object:

                 Semantic AST
                    |
          +---------+---------+
          |                   |
       renderer           interpreter A
          |                   |
        prompt                gold
          |
        parser
          |
       AST compare

plus:
interpreter B
certificate checker
mutation tests

The purpose of the second interpreter is important. A single shared implementation can make the prompt and the oracle agree on the same bug. Independent evaluation paths make that harder.

Mutation tests deliberately injected bad semantics such as witness merging or incorrect proof-edge scope and required the validation suite to detect them.

At that point the benchmark had become as much a software-verification problem as a question-writing problem.

llama.cpp taught another measurement lesson

The experiments also exposed a separate issue around reasoning controls.

At first, a total completion ceiling was being treated too casually as if it were a reasoning budget. That produced misleading transitions where a higher reasoning effort appeared “wrong” only because it never reached a final answer before the ceiling.

llama.cpp exposes separate reasoning controls. In a direct probe of request-level thinking_budget_tokens, the same simple prompt produced:

thinking budget 0   -> 25 completion tokens
thinking budget 32  -> 57 completion tokens
thinking budget 96  -> 121 completion tokens

The exact 32- and 96-token deltas were strong evidence that the request-level budget was being applied in the local server configuration.

But another wrinkle appeared: after the internal reasoning block ended, the model could still produce several hundred tokens of visible analysis in ordinary content before the final answer.

That means these three quantities must not be collapsed into one:

native reasoning effort
reasoning/thinking budget
total completion ceiling

For this reason, I now treat native effort comparison and fixed-budget comparison as different experimental protocols.

Semantic Miner v1: witness binding and proof-edge validity

The first cleaner semantic miner tested two mechanisms.

Witness binding

The key distinction was between:

exists x: P(x) AND Q(x)

and:

(exists x: P(x)) AND (exists y: Q(y))

The first expression requires one shared witness. The second does not.

Instances used only four to six named objects, a small number of explicit facts, and at most two dependency layers. A certificate was often just a witness object or an empty intersection between two support sets.

Result:

12 / 12 correct

Global truth versus cited-edge validity

The second family used short proof DAGs. A conclusion could be globally derivable while a specific proof step citing only selected parent lines was invalid.

The benchmark therefore distinguished:

global_truth(node)

from:

entailed_by(cited_parents, node)

The local oracle needed at most four Boolean atoms, so exhaustive entailment required at most 16 assignments.

Result:

12 / 12 correct

Six controls were also correct.

Semantic Miner v1 therefore ended at:

30 / 30 correct

Completion lengths ranged from 251 to 1,416 tokens.

One caveat matters: this first run did not include the later explicit unlimited-reasoning attestation, so the harness conservatively classified termination as unknown. That does not change the answer result—there were no wrong answers—but it is why I do not count these 30 as formally verified natural completions in the strongest aggregate claim below.

Representation Miner v2: could an equivalent representation cause a failure?

The next idea was to stop making the underlying problem harder.

Instead, keep the semantic problem tiny and change its representation.

One family defined a complete local operator table under a neutral symbol, then rendered the same semantics under familiar-looking symbols such as +, *, or max. The hypothesis was that pretrained semantic priors might override the explicit local table.

A second family used three-state mappings and varied composition orientation, input/output column direction, and relabeling while preserving a programmatically verified answer mapping.

The results were unambiguous:

composition-orientation family: 16 / 16 correct
operator-prior family:           16 / 16 correct

All 32 were verified natural completions.

natural wrongs          0
completion caps         0
format failures         0
metamorphic mismatches  0

Completion lengths were only 188 to 764 tokens.

The target model did not merely survive these transformations. It handled them cheaply.

Mechanism Tournament v3: four different failure hypotheses at once

At this point, testing one mechanism per iteration was becoming inefficient. The next pack used four deliberately different mechanisms, eight structural roots each.

The rule was simple: if a family scored 8/8, do not rescue it by increasing problem size.

Capture-avoiding substitution

A classic lambda-calculus trap is variable capture.

For example, substituting y for free x inside a term containing a lambda y binder cannot be implemented as naive textual replacement. The binder must first be alpha-renamed when necessary.

The wrong algorithm produces a highly plausible term, while the correct certificate is local and short.

Result:

8 / 8 correct

Completion range:

578-1484 tokens

Simultaneous substitution

The second family distinguished simultaneous replacement from sequential replacement.

For a mapping such as:

{x := y, y := z}

the inserted right-hand side of one substitution must not be recursively rewritten by another substitution from the same simultaneous mapping.

Result:

8 / 8 correct

Completion range:

404-1191 tokens

Lexical closures, mutable cells, and shadowing

The third family defined a tiny closure language with explicit semantics:

• lexical scope
• closures capture environment locations
• set mutates the currently referenced cell
• shadow creates a new cell
• calls use the closure’s captured environment plus a fresh parameter cell

Executable wrong semantics included dynamic scope and capture-by-value.

Result:

8 / 8 correct

Completion range:

386-706 tokens

Deterministic causal intervention

The final family used small binary structural causal models.

The contract explicitly stated that:

do(X=v)

replaces X’s structural equation and requires recomputing downstream variables in the modified system.

A deliberately wrong semantics instead solved the original system first and then clamped X afterward.

Result:

8 / 8 correct

Completion range:

289-404 tokens

The clean aggregate result

The methodology-cleaner rounds produced the following table:

Total answer accuracy across these rounds was:

94 / 94

The stronger termination-qualified subset is v2 plus v3:

64 / 64 verified natural correct
0 completion caps
0 format failures
0 clean wrong answers

This is not evidence that Qwen3.8-27B has “solved reasoning.” The sample is small, synthetic, and intentionally excludes many sources of real difficulty.

It does support a narrower claim:

On these small, explicit, low-search semantic reasoning mechanisms, this Q4 27B local model was extremely robust under low native reasoning effort.

The token counts matter. These were not 10,000-token rescues. Many items were completed in a few hundred tokens.

What this result does not cover

ExactBench deliberately excluded or constrained several things that can make models fail:

• large combinatorial search
• long-horizon planning
• very long proofs
• external or obscure knowledge
• noisy retrieval
• ambiguous natural language
• huge state spaces
• context-window pressure

Those are legitimate benchmark targets, but they answer different questions.

The problem here was narrower: can we find a small, self-contained reasoning error whose answer is cheap to verify and whose failure is not dominated by resources or ambiguity?

So far, the answer is: not with these mechanisms.

Ten rules that survived the experiment

The most reusable output of ExactBench so far is not a collection of difficult questions. It is a set of benchmark-engineering rules.

1. Token exhaustion is not a reasoning error. Record it as censoring or a resource-limit event.
2. Model-oracle disagreement is not automatically a model failure. Audit the semantic contract first.
3. Every oracle assumption must be explicit in the prompt contract.
4. Generate prompts and gold answers from a shared formal semantic object when practical.
5. Still use an independent second oracle. A shared bug can otherwise validate itself.
6. Short certificates do not imply cheap discovery. Keep search complexity separate.
7. Do not rescue a solved mechanism by only increasing scale. That often turns semantic testing into bookkeeping endurance.
8. A natural wrong answer is much more informative than a timeout.
9. A failure challenge set is not the same thing as a held-out evaluation set. Selecting items because one condition failed creates selection bias.
10. A negative experiment is still a result. Do not force a benchmark to produce a dramatic failure.

The statistical trap in mining low-effort failures

One methodological point deserves special attention.

Suppose a benchmark keeps only items that low answered incorrectly and then tests medium on those survivors. A high repair rate does not by itself prove that medium reasoning is better on the underlying distribution.

The set was selected conditional on a low-effort failure.

A better design separates two products:

Failure Challenge Set

Known failures discovered through adversarial mining.

Question answered:

Can another model, quantization, or reasoning condition repair this known weakness?

Held-Out Evaluation Set

Generated only after the distribution and generator are frozen, without filtering on low performance.

Question answered:

What is the average paired performance difference between conditions?

Both sets are useful. They simply support different claims.

Why this was harder than expected

The original goal sounded straightforward: make a 27B Q4 model fail.

That is easy if “failure” includes enormous search, missing context, ambiguous instructions, or exhausted output budgets.

The hard version is much more specific:

tiny
clear
cheap
verifiable
naturally completed
wrong

Once those constraints are enforced, benchmark construction becomes surprisingly unforgiving.

The two most convincing “failures” found early in this project turned out to be benchmark specification bugs. Later, once semantic contracts, dual oracles, termination states, and mutation tests were tightened, the target model simply kept answering correctly.

That is why the title of this note is literal.

We tried to break a 27B Q4 model.

The benchmark broke first.

Where ExactBench goes next

The next useful step is not to inflate the same mechanisms until the model runs out of tokens.

A better direction is to search for qualitatively different, reproducible semantic failure modes while preserving the same constraints: small instances, exact semantics, low search, independent oracles, and natural completion.

Quantization sensitivity may also become more interesting than raw difficulty. An item that is stable at Q8 but fails at Q4 would be valuable even if every reasoning-effort setting at Q4 made the same mistake.

Likewise, native-effort experiments should remain separate from fixed-budget experiments. llama.cpp exposes both kinds of control, but they represent different interventions and should not be collapsed into one axis.

For now, the honest result is simple:

We have not yet found a clean short-certificate reasoning weakness in this Qwen3.8-27B Q4 setup.

That is not the same as saying no such weakness exists.

It means the benchmark has finally become strict enough that a wrong answer has to earn the name.