I Ran DeepSeek at 23 Tokens/s on a Single RTX 4090
DeepSeek V4.1 Flash reached 20–23 output tokens per second on one RTX 4090 with tandem.cpp. Bigger than VRAM, fast enough to use, and speculative decoding made it slower.
Approximately 3 min read
Results
- What we found
- DeepSeek V4.1 Flash produced 20.08–22.76 output tok/s in 400-token tests on one RTX 4090, with a 25.54 tok/s short-run peak.
- Test conditions
- Single RTX 4090 24GB; 128GB RAM; GSQ-RCO IQ2_XS; tandem.cpp; 16GB GPU expert cache; 16K configured context.
- 150-token testPeak generation speed25.54 tok/s
Measured short-run peak; longer 400-token tests reached 20.08–22.76 output tok/s. Data table ↓
A DeepSeek model with roughly 139 GiB of quantized weights. One RTX 4090 with 24GB VRAM. And it runs at 23 tokens per second.
I tested DeepSeek V4.1 Flash on my home workstation. No GPU cluster. No datacenter hardware. Just a single RTX 4090, 128GB RAM, and tandem.cpp.
The result surprised me.
23 tokens per second. On one 4090.
Here is what I actually saw:
| Test | Generation speed |
|---|---|
| 400-token runs | 20.08–22.76 tok/s |
| Best 150-token run | 25.54 tok/s |
That is not a slideshow. It is fast enough to chat with.
The setup was simple:
| Component | Configuration |
|---|---|
| GPU | RTX 4090, 24GB |
| System RAM | 128GB |
| Model | DeepSeek V4.1 Flash |
| Quantization | GSQ-RCO IQ2_XS |
| Runtime | tandem.cpp |
| GPU expert cache | 16GB |
| Configured context | 16K |
The model weights are far larger than the GPU memory. That is the whole point.
How is this possible?
DeepSeek is a Mixture-of-Experts (MoE) model. It does not need every expert for every generated token.
tandem.cpp keeps useful experts in a GPU cache while the CPU and system memory handle other work. The important optimization is not squeezing an enormous model entirely into VRAM. It is keeping the right experts close to the GPU when they are needed.
With a 16GB expert cache and six CPU expert-worker threads, the result was a surprisingly usable local setup.
139 GiB of weights does not mean you need 139 GiB of VRAM.
The funniest part: speculative decoding made it worse
I also tried adding DSpark speculative decoding. It sounds like an obvious performance upgrade.
Instead, it slowed the model down.
| Mode | Observed speed |
|---|---|
| Plain inference | 20–23 tok/s |
| DSpark Q2_K | 17–19 tok/s |
| DSpark Q8 | 14–17 tok/s |
More machinery, less speed.
On this workload, the extra drafting and verification overhead was more expensive than simply generating tokens with the original setup.
I turned it off.
The takeaway
One RTX 4090. A DeepSeek model much bigger than its VRAM. More than 20 output tokens per second.
That is what I wanted to know: can a huge MoE model feel usable on hardware I already own?
Yes.
The future of local AI is not only about buying bigger GPUs. It is also about smarter expert caching, memory movement, and runtimes that know how to use an ordinary workstation.
And sometimes, the best optimization is switching an optimization off.
RAMGPT Local AI Lab · Tested on Ubuntu with RTX 4090, 128GB RAM, and tandem.cpp.