Running Qwen3.8-27B on an RTX 4090 with vLLM + OpenWebUI

I wanted to see how far I could push a single 24 GB consumer GPU before deciding that local AI needed a much more expensive setup.

The result surprised me.

With one RTX 4090, I was able to run Qwen3.8-27B through a patched vLLM stack with a 64K context window, MTP speculative decoding, an OpenAI-compatible API, tool calling, and OpenWebUI on top.

This is not the usual “download a GGUF file and run llama.cpp” path. It is a more experimental stack built around quantization, custom patches, and speculative decoding:

Qwen3.8-27B
  ↓
W4A16 + int4/int8 requantization
  ↓
MTP speculative decoding
  ↓
patched vLLM 0.27.1
  ↓
OpenAI-compatible API
  ↓
OpenWebUI

That also means it is less forgiving than a stock setup. I would treat this as a high-performance local AI workstation experiment, not a production inference server.

The hardware I used

The machine had two GPUs installed:

CUDA 0: RTX 4090 24 GB
CUDA 1: RTX 3060

The Qwen service used only the 4090:

export CUDA_VISIBLE_DEVICES=0

The 3060 was left completely out of the LLM inference stack, which means it can stay available for something else such as speech-to-text, TTS, or another GPU workload.

In the working fast configuration, the model used a 65,536-token context window with FlashAttention, BF16 KV cache, and MTP-4 speculative decoding. During startup, the model weights were around 13.97 GiB and the KV cache around 5.23 GiB.

If VRAM budgeting is new to you, RAMGPT’s guides on LLM VRAM requirements and KV cache are useful background before trying this kind of setup.

1. Clone the project

I used the syv-ai/qwen38-27b-rtx3090 project:

cd ~
git clone https://github.com/syv-ai/qwen38-27b-rtx3090.git qwen38-serving
cd ~/qwen38-serving

This project is specifically optimized for 24 GB NVIDIA GPUs and includes several pieces that are not part of a normal vLLM install:

• W4A16 model weights;
• requantized lm_head and embeddings;
• MTP speculative decoding;
• a 40K draft vocabulary;
• a GPTQ int4 fast variant;
• patched FlashAttention behavior;
• patched vLLM sampling and Marlin kernels;
• FP16 DeltaNet recurrent state.

The most important lesson here is simple: do not casually upgrade the vLLM version. The patches are part of the stack.

The setup I tested used:

vLLM 0.27.1

2. Create the Python 3.12 environment

I created a dedicated virtual environment:

python3.12 -m venv venv

On Ubuntu, I initially hit:

ensurepip is not available

The fix was straightforward:

sudo apt update
sudo apt install -y python3.12-venv
rm -rf venv
python3.12 -m venv venv

Then I checked it:

venv/bin/python --version

3. Install vLLM and the supporting packages

venv/bin/pip install -U pip

venv/bin/pip install \
  vllm \
  huggingface_hub \
  hf_transfer \
  ninja

In my working environment, this ended up with:

vLLM 0.27.1
torch 2.13.0+cu130

The install pulled a large set of CUDA 13.x Python runtime packages. That looked intimidating the first time, but it was expected for this stack.

4. Lock the service to the RTX 4090

Because the machine had more than one GPU, I explicitly pinned CUDA ordering and visibility:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0

Using PCI_BUS_ID also makes GPU numbering less surprising on multi-GPU systems.

5. Download the Qwen3.8-27B W4A16 model

HF_HUB_ENABLE_HF_TRANSFER=1 \
venv/bin/hf download \
  dbirks/Qwen3.8-27B-W4A16-AutoRound \
  --local-dir models/Qwen3.8-27B-W4A16-AutoRound

The download was roughly 19.5 GB.

The final model directory was:

~/qwen38-serving/models/Qwen3.8-27B-W4A16-AutoRound

This is where the setup starts becoming different from a normal local model install. Downloading the model is only the beginning.

6. Requantize the model components

The next step was to process three parts of the model separately:

export CUDA_VISIBLE_DEVICES=0

venv/bin/python quant_lm_head.py \
  models/Qwen3.8-27B-W4A16-AutoRound

venv/bin/python quant_embed.py \
  models/Qwen3.8-27B-W4A16-AutoRound

venv/bin/python quant_mtp.py \
  models/Qwen3.8-27B-W4A16-AutoRound

These handle:

lm_head
embedding
MTP drafter

The round-trip relative error I saw was roughly in the 0.0056 - 0.0153 range.

Then I built the 40K-token draft vocabulary:

venv/bin/python build_draft_vocab.py \
  models/Qwen3.8-27B-W4A16-AutoRound \
  --ids draft_vocab_ids.json

That produced a 40,960-token draft head of about 213 MB.

If you want the intuition behind this, the main model is not the only thing being compressed. The fast path is trying to make the draft-and-verify cycle cheap enough that speculative decoding is actually useful on one GPU. RAMGPT’s LLM quantization guide explains the broader trade-off between model size, precision, and speed.

7. Download the fast variant

For single-user performance, this was one of the most important steps:

venv/bin/python fetch_fast_variant.py

It downloaded roughly 983 MB and created:

models/Qwen3.8-27B-W4A16-AutoRound-fast

The startup script automatically detects this directory and prefers it when available.

The fast variant uses:

GPTQ int4 lm_head
GPTQ int4 MTP module
40K own-output draft vocabulary

8. Patch vLLM

This project does not run on stock vLLM 0.27.1. The patches are not optional decoration; they are part of the design.

I applied them with:

for p in patches/*.patch; do
  patch -p1 \
    -d venv/lib/python3.12/site-packages/vllm \
    < "$p"
done

My run applied patches for Marlin, Qwen embedding quantization, MTP draft vocabulary handling, sampler behavior, speculative decoding attention, and related speed controls.

This is also the biggest reason I would not call this a beginner production stack. It works, but you are depending on a specific combination of code versions and patches.

9. Create an API key and verify everything

I generated a local API key:

openssl rand -hex 24 > api_key.txt

Then I ran the project’s verification script:

bash verify.sh --no-server

The target result is:

verify: OK (0 failures)

My checks passed for the virtual environment, RTX 4090, vLLM version, Triton, FlashInfer, compressed tensors, all required patches, requantized model components, the MTP draft head, the fast variant, and the API key.

I did see a warning that KVarN was not installed. That was fine for this configuration because I was targeting the stable 64K fast mode first rather than the experimental 200K+ context modes.

10. Start Qwen in fast mode

The basic launch command was:

cd ~/qwen38-serving

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0

CTX=fast bash single-user/start_qwen.sh

CTX=fast gave me:

64K context
FlashAttention
BF16 KV cache
4-token MTP speculative decoding

The logs confirmed:

model: Qwen3.8-27B-W4A16-AutoRound-fast
max_model_len: 65536
attention: FLASH_ATTN
kv_cache: bfloat16
speculative decoding: MTP, 4 tokens

The first startup was much slower because of torch.compile, Triton JIT compilation, CUDA graph capture, and warmup. My first initialization took about 94 seconds, with roughly 41 seconds spent compiling.

After the caches were built, later engine initialization dropped to around 9 seconds.

That difference made the setup feel much more practical than it did during the first launch.

11. Test the OpenAI-compatible API

The vLLM server listened on:

http://0.0.0.0:18020

The OpenAI-compatible base URL was:

http://localhost:18020/v1

I first checked the model list:

curl http://localhost:18020/v1/models \
  -H "Authorization: Bearer $(cat ~/qwen38-serving/api_key.txt)"

Then I tested chat completions:

curl http://localhost:18020/v1/chat/completions \
  -H "Authorization: Bearer $(cat ~/qwen38-serving/api_key.txt)" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [
      {
        "role": "user",
        "content": "Explain what speculative decoding does in simple terms."
      }
    ]
  }'

Once this worked, I knew the inference stack itself was healthy before adding a UI.

12. Connect OpenWebUI

In OpenWebUI I went to:

Admin Settings
→ Connections
→ OpenAI

Then I added:

API URL:
http://localhost:18020/v1

and used the value from:

cat ~/qwen38-serving/api_key.txt

The first connection returned:

GET /v1/models
401 Unauthorized

This was actually useful because it proved OpenWebUI could reach vLLM. The network path was fine; the authentication value was wrong.

After I entered the correct API key, the request immediately changed to:

GET /v1/models
200 OK

13. Fix OpenWebUI tool calling

After basic chat worked, I hit the second issue:

"auto" tool choice requires
--enable-auto-tool-choice
and
--tool-call-parser

OpenWebUI was sending tools with:

tool_choice = auto

but the original startup only had the reasoning parser enabled.

I restarted vLLM with:

cd ~/qwen38-serving

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0

EXTRA_ARGS="--enable-auto-tool-choice --tool-call-parser qwen3_coder" \
CTX=fast \
bash single-user/start_qwen.sh

The logs then confirmed:

enable_auto_tool_choice: True
tool_call_parser: qwen3_coder

and OpenWebUI chat requests started returning 200 OK with tool calling enabled.

My final launch command

After working through the setup, this became the command I actually wanted to keep:

cd ~/qwen38-serving

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0

EXTRA_ARGS="--enable-auto-tool-choice --tool-call-parser qwen3_coder" \
CTX=fast \
bash single-user/start_qwen.sh

That gave me this stack:

RTX 4090
Qwen3.8-27B W4A16
GPTQ int4 lm_head
GPTQ int4 MTP
64K context
FlashAttention
BF16 KV cache
MTP-4 speculative decoding
Qwen3 reasoning parser
Qwen3 coder tool-call parser
OpenAI-compatible API
OpenWebUI

What performance looked like in actual use

This is the part where it is easy to overstate the result, so I want to be careful.

The project README reports roughly 114 tok/s with default sampling and 124 tok/s greedy on an RTX 3090. On my RTX 4090, the vLLM runtime logs often showed generation throughput in the 100-180 tok/s range during normal OpenWebUI use.

I also saw short windows above 200 tok/s, including one around 213 tok/s.

That does not mean “Qwen3.8-27B runs at 213 tok/s on a 4090” as a benchmark result.

Those numbers came from vLLM’s rolling runtime statistics, and speculative decoding makes them highly dependent on how many draft tokens are accepted.

A more accurate summary is:

In actual single-user OpenWebUI sessions, generation throughput was often around 100-180 tok/s, with short bursts above 200 tok/s when draft acceptance was unusually high.

Why MTP speed changes so much

The logs made speculative decoding much easier for me to understand.

When the drafter predicted the target model well, I saw something like:

Mean acceptance length: 4.79
Draft acceptance rate: 94.7%
Generation throughput: 213 tok/s

But another generation could look like:

Mean acceptance length: 1.13
Draft acceptance rate: 3.1%
Generation throughput: 49.2 tok/s

The basic idea is simple:

Drafter predicts several tokens
        ↓
Target model verifies them
        ↓
More accepted tokens per verification step
        ↓
Higher effective decode speed

When the drafter keeps guessing wrong, most speculative tokens are rejected and the speed advantage shrinks.

That is why seeing throughput swing from roughly 50 tok/s to more than 200 tok/s was not necessarily a sign that the system was broken.

Prompt throughput is not generation speed

Another thing I had to learn was not to confuse prefill speed with decode speed.

The logs sometimes showed prompt throughput of:

600 tok/s
1500 tok/s
3000 tok/s

That is the model processing the input prompt. It is not the rate at which the answer appears on screen.

For interactive use, I care more about:

TTFT
+
generation throughput

For example:

Avg prompt throughput: 1579 tok/s
Avg generation throughput: 84.6 tok/s

The second number is much closer to what I actually feel while reading the response.

A few scary-looking logs were not fatal

I saw startup messages involving Qwen video processor parameters even though this was a text-only server. The logs then explicitly continued in text-only mode and the API started normally.

I also saw this once:

expandable_segments: memory mapping failed with OOM

The engine still finished loading and created about 5.23 GiB of KV cache, so in that run it was not a fatal serving error.

I would react differently to messages such as:

CUDA out of memory
EngineCore failed

Those would indicate that the engine itself did not start successfully.

Why I stayed at 64K context

The project also has longer context modes, including configurations that can go far beyond 64K with additional KV-cache compression techniques.

I decided not to start there.

The 65,536-token fast mode already gave me:

• a large enough context for daily chat;
• coding assistance;
• RAG experiments;
• medium-to-large document work;
• FlashAttention;
• strong MTP speculative-decoding performance.

For a first working build, I would rather have a fast 64K configuration I understand than a 200K+ configuration that adds another layer of experimental behavior.

Was it worth it?

For me, yes, but with one big condition: I knew I was choosing a patched experimental stack.

The interesting part is not a single benchmark number. It is that one consumer RTX 4090 can support a setup with:

27B model
+
24 GB GPU
+
64K context
+
OpenAI-compatible API
+
OpenWebUI
+
tool calling
+
fast interactive generation

That is already enough to feel like a real personal AI workstation instead of a demo.

The trade-off is maintenance. This setup depends on a specific vLLM version, several patches, custom quantization steps, and experimental kernel behavior. Upgrading it is more work than running stock vLLM or a normal GGUF model.

So I would recommend it to someone who enjoys understanding the stack and is comfortable fixing things when an update breaks assumptions.

If your priority is maximum simplicity, this is probably not the first local LLM setup I would choose.

If your priority is seeing how much performance you can squeeze out of one 24 GB GPU, it is a very interesting project.