Local AI

I Ran DeepSeek at 23 Tokens/s on a Single RTX 4090

DeepSeek V4.1 Flash reached 20–23 output tokens per second on one RTX 4090 with tandem.cpp. Bigger than VRAM, fast enough to use, and speculative decoding made it slower.

Approximately 3 min read

Results

What we found
DeepSeek V4.1 Flash produced 20.08–22.76 output tok/s in 400-token tests on one RTX 4090, with a 25.54 tok/s short-run peak.
Test conditions
Single RTX 4090 24GB; 128GB RAM; GSQ-RCO IQ2_XS; tandem.cpp; 16GB GPU expert cache; 16K configured context.
Single-GPU DeepSeek short-run peakHigher is faster
  1. 150-token test
    Peak generation speed25.54 tok/s

Measured short-run peak; longer 400-token tests reached 20.08–22.76 output tok/s. Data table ↓

A DeepSeek model with roughly 139 GiB of quantized weights. One RTX 4090 with 24GB VRAM. And it runs at 23 tokens per second.

I tested DeepSeek V4.1 Flash on my home workstation. No GPU cluster. No datacenter hardware. Just a single RTX 4090, 128GB RAM, and tandem.cpp.

The result surprised me.

23 tokens per second. On one 4090.

Here is what I actually saw:

Test Generation speed
400-token runs 20.08–22.76 tok/s
Best 150-token run 25.54 tok/s

That is not a slideshow. It is fast enough to chat with.

The setup was simple:

Component Configuration
GPU RTX 4090, 24GB
System RAM 128GB
Model DeepSeek V4.1 Flash
Quantization GSQ-RCO IQ2_XS
Runtime tandem.cpp
GPU expert cache 16GB
Configured context 16K

The model weights are far larger than the GPU memory. That is the whole point.

How is this possible?

DeepSeek is a Mixture-of-Experts (MoE) model. It does not need every expert for every generated token.

tandem.cpp keeps useful experts in a GPU cache while the CPU and system memory handle other work. The important optimization is not squeezing an enormous model entirely into VRAM. It is keeping the right experts close to the GPU when they are needed.

With a 16GB expert cache and six CPU expert-worker threads, the result was a surprisingly usable local setup.

139 GiB of weights does not mean you need 139 GiB of VRAM.

The funniest part: speculative decoding made it worse

I also tried adding DSpark speculative decoding. It sounds like an obvious performance upgrade.

Instead, it slowed the model down.

Mode Observed speed
Plain inference 20–23 tok/s
DSpark Q2_K 17–19 tok/s
DSpark Q8 14–17 tok/s

More machinery, less speed.

On this workload, the extra drafting and verification overhead was more expensive than simply generating tokens with the original setup.

I turned it off.

The takeaway

One RTX 4090. A DeepSeek model much bigger than its VRAM. More than 20 output tokens per second.

That is what I wanted to know: can a huge MoE model feel usable on hardware I already own?

Yes.

The future of local AI is not only about buying bigger GPUs. It is also about smarter expert caching, memory movement, and runtimes that know how to use an ordinary workstation.

And sometimes, the best optimization is switching an optimization off.


RAMGPT Local AI Lab · Tested on Ubuntu with RTX 4090, 128GB RAM, and tandem.cpp.

Sources and further reading