Qwen3.8-27B on 24GB GPUs: The New Local LLM Sweet Spot?

A 27B dense model that can realistically live on a single 24 GB GPU is exactly the kind of release that makes the local-LLM crowd stop scrolling.

That is why Qwen3.8-27B matters.

The official model is live. Unsloth GGUFs are live. llama.cpp can run it. And the interesting question is no longer whether the model exists.

The question is:

Which quant should I download?
Can I keep it fully on a 24 GB GPU?
How much context can I afford?
Is Q5 worth the squeeze?
What do MTP and thinking mode change?

Here is the punchline first:

If you own a 24 GB-class GPU, Qwen3.8-27B in Q4 is one of the most compelling new local-model targets right now.

Not because 27B is magically perfect.

Because the numbers line up unusually well.

27B dense model
+
Q4 GGUF around 17 GB
+
24 GB GPU
=
real room to actually run the model

That is the sweet spot.

The catch is that 17 GB of model weights does not mean 17 GB total VRAM. Context, runtime buffers, vision, cache state, and backend overhead still have to fit.

So the goal of this guide is simple: figure out where Qwen3.8-27B feels comfortable, where it starts getting tight, and where you are just fighting physics.

The 30-second answer

Qwen3.8-27B is an official 27B dense multimodal model released under Apache 2.0.

The model card lists:

27B parameters
64 layers
native image and video understanding
thinking mode
reasoning_effort controls
preserve_thinking support
MTP training
262,144-token native context
up to 1,000,000 tokens with context extension

Unsloth already publishes llama.cpp-ready GGUFs.

For a single 24 GB GPU, these are the numbers that matter most:

So my simple recommendation is:

24 GB GPU
→ start with Q4

Want more fidelity and modest context?
→ try Q5

Trying Q6 on 24 GB?
→ expect compromises

Q8?
→ bring more VRAM or offload

This is why people are excited

There are plenty of local models that are impressive on paper but awkward in practice.

Some are too small to feel like a serious workstation model.

Others are excellent but want:

48 GB
80 GB
multiple GPUs
or a lot of CPU offload

Qwen3.8-27B lands in a much more interesting place.

It is large enough to target serious coding, reasoning, agentic work, multimodal tasks, and long-context workloads, while still being small enough that modern Q4 quantization can put the weights on one 24 GB consumer GPU.

That combination is what makes the release fun.

It is not just:

another model got 3 points higher on a benchmark.

It is:

a model in a capability class people care about may actually fit on hardware many local-AI users already own.

The Q4 numbers are the real story

Unsloth currently provides a broad set of Qwen3.8-27B GGUFs.

The most relevant 24 GB choices are:

Q4_K_S       16.1 GB
Q4_K_M       17.1 GB
UD-Q4_K_XL   17.9 GB

Now compare that with a nominal 24 GB GPU.

For UD-Q4_K_XL:

24.0 GB GPU
-17.9 GB weights
----------------
 6.1 GB remaining

That 6.1 GB is not a free bonus pool.

It has to absorb some mixture of:

context state
runtime buffers
temporary allocations
backend overhead
vision-related allocations
other GPU processes

But 6.1 GB is enough headroom to make the conversation interesting.

That is very different from trying to cram a 22.9 GB model file into the same card and pretending the remaining 1.1 GB will somehow handle everything else.

Q4_K_M or UD-Q4_K_XL?

If I wanted the safest first run, I would start with Q4_K_M.

Why?

Because the first objective is not to win a quantization beauty contest.

The first objective is:

load the model
keep useful context
avoid unnecessary offload
see how it behaves

Q4_K_M at 17.1 GB gives you a little more breathing room than UD-Q4_K_XL at 17.9 GB.

Once that baseline is stable, moving upward is easy.

If you already know your workload is modest and you want to try the larger dynamic quant, UD-Q4_K_XL is a very reasonable target.

For the general trade-offs between Q4, Q5, Q6 and Q8, see GGUF Quantization Levels Explained.

Q5 is where the temptation starts

The current Q5 files sit around:

Q5_K_S       19.3 GB
Q5_K_M       19.8 GB
UD-Q5_K_XL   20.2 GB

And this is where every 24 GB GPU owner has the same thought:

It is below 24 GB. Why not just use Q5?

Because your GPU is not a USB stick.

The model file is only one part of inference memory.

With a 20.2 GB model, you have only about 3.8 GB left before everything else starts asking for space.

That can still work.

For a moderate text workload, Q5 may be perfectly usable.

But now you are making a trade:

more weight precision
vs
less room for context and runtime state

So Q5 is not bad.

Q5 is spicy.

It is the quant I would test after Q4, not the one I would blindly recommend to every 24 GB user.

Q6 is where the math stops being friendly

Q6_K is currently about:

22.9 GB

That leaves roughly:

1.1 GB

on a nominal 24 GB card before runtime overhead.

That is not headroom.

That is a hostage situation.

UD-Q6_K_XL is even larger at about:

25.9 GB

So yes, you can still use CPU offload, partial GPU placement, or multiple GPUs.

But once you are doing that, the question has changed.

You are no longer asking:

What is the cleanest Qwen3.8-27B setup for one 24 GB GPU?

You are asking:

How much compromise am I willing to accept to run a larger quant?

For most single-card users, Q4 or Q5 is simply the more sensible fight.

Q8: stop pretending, you need more memory

Q8_0 is around:

29.0 GB

UD-Q8_K_XL is around:

31.5 GB

There is no clever interpretation here.

Those weights do not fully fit inside 24 GB VRAM.

You need:

CPU offload
multiple GPUs
or more VRAM

If your goal is simply to run Qwen3.8-27B locally and enjoy the model, Q8 is not the hill to die on.

llama.cpp support is already here

This release is not stuck in the awkward phase where weights exist but every local runtime is still catching up.

Unsloth already documents direct llama.cpp launch commands.

The shortest path is:

llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

With the traditional server binary:

./llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Or from a source build:

./build/bin/llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

That is the part I like most about this release.

You do not need a weekend of conversion scripts just to see whether the model works.

You can go from:

new model announced

to:

GGUF
→ llama.cpp
→ local server

very quickly.

If llama.cpp itself is new to you, start with How to Run a GGUF Model with llama.cpp.

Why does the GGUF say qwen35?

This is one of those details that can send people down the wrong rabbit hole.

You may inspect the GGUF metadata and see:

qwen35

Then think:

Wait. Did I download Qwen3.5 by mistake?

No.

Qwen describes Qwen3.8 as building on the architectural foundation of Qwen3.5, so the GGUF ecosystem can reuse the underlying architecture label.

The branding changed.

The low-level architecture identifier does not necessarily need to.

The practical evidence is what matters: the current Qwen3.8-27B GGUFs are documented for llama.cpp and are already being used by the local-model community.

The architecture is more interesting than the parameter count

Qwen3.8-27B is not just a conventional 64-layer full-attention transformer.

The official architecture uses a hybrid layout:

16 × (
  3 × (Gated DeltaNet → FFN)
  →
  1 × (Gated Attention → FFN)
)

That means most layers use Gated DeltaNet-style blocks, with full gated-attention blocks appearing periodically.

The full-attention portion uses:

24 Q heads
4 KV heads
256 head dimension

Why should a local user care?

Because it means a lazy VRAM calculation copied from a conventional transformer may be wrong.

The model’s long-context memory behavior is more nuanced than:

64 identical layers
×
standard KV cache formula

If someone gives you an exact 262K-context VRAM number without explaining the architecture assumptions, treat it carefully.

For the general cache concept, read What Is a KV Cache?.

262K context sounds amazing. Do not max it out on day one.

The official native context is:

262,144 tokens

Qwen also documents extension up to:

1,000,000 tokens

That is a huge capability.

It is also a perfect way to turn a successful 24 GB setup into an OOM experiment if you treat maximum context as the default setting.

Start smaller.

For example:

8K
16K
32K

Then watch memory use and prompt-processing behavior.

If you actually need more, increase it deliberately.

A model supporting 262K does not mean your 24 GB GPU owes you 262K for free.

That distinction is exactly why context length and VRAM planning matter.

MTP could be a big deal

Qwen3.8-27B was trained with Multi-Token Prediction, and Unsloth says MTP support is available for faster inference.

That is worth paying attention to.

But this is also where benchmark screenshots can become dangerous.

An MTP speed result depends on things like:

GPU
quantization
llama.cpp build
context length
MTP settings
flash attention
KV-cache type
prompt workload

So I would not ask:

How many tokens per second does Qwen3.8-27B get?

I would ask:

How many tokens per second does this exact Qwen3.8-27B configuration get on my hardware?

That is the benchmark that matters.

Thinking mode can make a fast GPU feel slow

Qwen3.8-27B supports thinking mode and controls such as:

reasoning_effort = xhigh
reasoning_effort = medium
reasoning_effort = low

It also supports preserve_thinking for carrying reasoning context across turns.

This creates an important practical distinction.

A model can generate tokens quickly and still feel slow because it spends a long time reasoning before producing the answer you actually wanted.

So when someone says:

Qwen3.8 feels slow

ask two different questions:

Is token generation slow?

or

Is the model simply thinking for a long time?

Those are not the same problem.

The benchmark numbers are strong — but they are Qwen’s numbers

Qwen reports large gains over Qwen3.6-27B across coding, agentic work, instruction following, reasoning and computer-use tasks.

Examples from Qwen’s release material include:

Those are attention-grabbing numbers.

They are also vendor-reported results.

RAMGPT has not independently reproduced that benchmark suite, so I would treat them as release claims rather than settled fact about real-world superiority.

The more interesting test is what happens when people run the same prompts, same quant class, and same hardware against Qwen3.6-27B and Qwen3.8-27B.

LocalLLaMA has already moved past the announcement phase

The useful signal from LocalLLaMA is not that people are excited.

LocalLLaMA is excited about everything for at least six hours.

The useful signal is that discussion has already moved into deployment details:

3090-class GPUs
dual-GPU setups
GGUF choice
context sizing
MTP
thinking length
chat-template behavior

That means this is already a real local-deployment story, not just a model-card story.

Community reports are excellent for finding weird edge cases quickly.

They are not controlled benchmarks.

If somebody posts one spectacular tokens-per-second number, you still need to know:

which GPU
which quant
which context
which build
which flags
which cache type

before treating it as useful evidence.

What should 16 GB users do?

Qwen3.8-27B is not completely off-limits below 24 GB.

Current smaller GGUFs include:

UD-IQ3_XXS   11.9 GB
UD-Q3_K_XL   13.4 GB
IQ4_XS       15.7 GB

There are even more aggressive 2-bit files below that.

So yes, you can try.

But the further you push below Q4-class territory, the more you are trading:

quantization fidelity
context headroom
runtime comfort

A 15.7 GB file on a 16 GB GPU is not the same thing as a comfortable 16 GB deployment.

32 GB and 48 GB-class systems get to relax

With roughly 32 GB of accelerator memory, Q6 becomes far more realistic.

With 40–48 GB, Q8 becomes much easier to justify while still leaving room for runtime state.

That is where the conversation changes from:

What can I squeeze in?

To:

Which quant actually gives me the quality/performance trade-off I want?

Multi-GPU setups add even more options, although two GPUs are not magically identical to one perfectly unified memory pool. Interconnect and split strategy still matter.

The multimodal part is easy to underestimate

Qwen3.8-27B is not text-only.

It supports images and video, and Qwen highlights workloads involving documents, STEM diagrams, computer-use perception, interfaces and long video understanding.

That is important because your real memory question may become:

model weights
+
context
+
vision path
+
runtime buffers

not simply:

Does the GGUF fit?

If you only care about coding, benchmark coding.

If you care about vision, test the full multimodal path.

Do not assume one workload tells you everything about the other.

What I would do first

For one 24 GB GPU, I would not overthink it.

I would start here:

llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

Then I would set a sane context, make sure the baseline works, and only then tune:

context
KV-cache precision
MTP
flash attention
thinking mode
sampling

Do not begin with a 15-flag command copied from Reddit and then spend two hours figuring out which flag broke the model.

Baseline first.

Optimization second.

The specifications and benchmark figures above come from Qwen; download sizes and run instructions come from the listed GGUF publisher. For measured runtime behavior, continue with the 200K-context residency investigation and the independent reproduction comparison.