I Ran DeepSeek at 23 Tokens/s on a Single RTX 4090
DeepSeek V4.1 Flash reached 20–23 output tokens per second on one RTX 4090 with tandem.cpp. Bigger than VRAM, fast enough to use, and speculative decoding made it slower.
Research · Guides · Explainers
Independent AI research & technical writing
Local AI Systems & Inference Contributor
bettercallcaleb writes about local AI inference, open-source runtimes, quantization, model behavior, and practical systems experimentation. His work emphasizes implementation details, reproducible methodology, and the gap between headline claims and what software actually does on real hardware.
Published work
43 published
DeepSeek V4.1 Flash reached 20–23 output tokens per second on one RTX 4090 with tandem.cpp. Bigger than VRAM, fast enough to use, and speculative decoding made it slower.
By bettercallcaleb ·
A 25-minute local coding failure turned into a verified patch after fixing OpenCode's working directory. Here is what our benchmark actually measured.
By bettercallcaleb ·
Our August 2026 OpenCode hardening analysis identified MCP OAuth, remote configuration, and credential custody as critical boundaries. October incidents tested that reasoning.
By bettercallcaleb ·
Fix vLLM DeepGEMM's Runtime is already registered ImportError by using a build containing merged PR #48962, which moved vendored DeepGEMM away from the duplicate pybind11 registration path.
By bettercallcaleb ·
A practical llama.cpp CUDA-vs-Vulkan guide: verify the backend actually loaded, compare device support and kernels, benchmark the same workload, and avoid mistaking CPU fallback for a backend result.
By bettercallcaleb ·
A local-deployment audit of BAAI AREX-2: its Qwen3.8 base, Qwen3.5-compatible model class, memory budget, multimodal conversion and reference-runtime checks.
By bettercallcaleb ·
A controlled RTX 4090 + RTX 3060 experiment traced a severe Strata Q2_0 multi-GPU prefill slowdown to the Linux expert-copy path. Allowing full CUDA host registration raised 32K prefill from 665 to 2,110 tok/s.
By bettercallcaleb ·
If you searched whether llama.cpp supports NVFP4, the answer depends on the checkpoint, converter, model architecture, backend, and quant-scale path. Use this checklist to identify the real blocker.
By bettercallcaleb ·
Independent EXL3 4.0 bpw measurements for Xiaomi MiMo-V2.6-Distill-Qwen-9B across RTX 4090, RTX 3060 12GB, mixed-GPU layer split, long context, TabbyAPI, vision, and tool calling.
By bettercallcaleb ·
Source analysis of a Sep 28 llama.cpp Vulkan bug where flash attention or large softmax can overwrite cached scratch data, causing a later matmul to reuse corrupted input.
By bettercallcaleb ·
Fix llama.cpp KV-cache allocation failures by lowering context or parallelism, leaving VRAM headroom, checking cache types, and reading the real allocator error.
By bettercallcaleb ·
Fix llama.cpp's unknown model architecture glm5next error by running a build that includes merged GLM-5.3-Flash support and verifying the actual binary or bundled backend.
By bettercallcaleb ·
If RX 7800 XT Vulkan MTP throughput drops after an update, pin the last known-good build and bisect the 146-commit regression window; issue #29410 remains open.
By bettercallcaleb ·
A local RTX 4090 agent benchmark shows Gemma 4 winning small tasks on speed while Qwen3.8 keeps agent turns flat by batching parallel tool calls.
By bettercallcaleb ·
A real AI-agent workflow audit shows how task routing, compact state checks, shared validation scripts, and smaller handoffs can reduce context and tool-call overhead.
By bettercallcaleb ·
Fix DeepSeek-V4 CUDA-graph startup assertions after max_model_len auto-fit by using vLLM 0.31.0+ with the ModelState fix or setting an explicit context that fits.
By bettercallcaleb ·
We tested public MiMo-V2.6-Distill-Qwen-9B GGUFs on one RTX 4090, comparing loadability, perplexity, speed, exact-task accuracy, and BF16 fidelity.
By bettercallcaleb ·
An independent Qwen3.8 Flash-Next benchmark on RTX 4090 + RTX 3060 showing how auto-fit, manual MoE placement, quantization, and MTP changed decode speed.
By bettercallcaleb ·
Diagnose vLLM speculative-decoding CUDA illegal-memory-access crashes by comparing target and draft context limits; route over-limit requests without speculation or align the limits.
By bettercallcaleb ·
Issue #29168 initially bisected an MTP acceptance drop to CUDA MoE fusion, but the reporter later found it was specific to one raw prompt and closed the issue.
By bettercallcaleb ·
Fix OpenVINO large-context allocation failures by using llama.cpp master with #29852, which reports the GPU's real single-allocation limit so GGML can split buffers.
By bettercallcaleb ·
Fix the Mistral-Large-3 YaRN accuracy regression by using vLLM 0.31.0+ or a build containing PR #57563, which restores the missing no-mscale attention flag.
By bettercallcaleb ·
Fix llama.cpp Vulkan wrong-result regressions from sliced cache views by updating past merged PR #28956, then compare the same GGUF and prompt against CPU.
By bettercallcaleb ·
Match common llama.cpp errors to the right fix: update stale runtimes, repair GGUF paths, isolate tensor mismatches, reduce KV or compute memory, and identify regressions.
By bettercallcaleb ·
Diagnose llama.cpp speculative-decoding assertions when --spec-draft-n-max grows the graph past scheduler hash capacity; lower n_max as a workaround while PR #28972 remains open.
By bettercallcaleb ·
PR #28898 proposes moving FP8 and NVFP4 quantization scales into GGML so more scaled safetensors checkpoints can convert directly instead of relying on model-specific handling.
By bettercallcaleb ·
Unsloth did not begin as a generic AI startup. It began with a narrow technical advantage, reproducible notebooks, and direct distribution to the local-LLM community.
By bettercallcaleb ·
Fix llama.cpp Metal corruption around the 2 GiB slice-offset boundary by updating to a build containing merged PR #28748, then retesting the same large workload.
By bettercallcaleb ·
A paired benchmark of refusal behavior, calibrated reasoning, open-ended response drift, and epistemic integrity in a 27B Base vs Heretic model comparison.
By bettercallcaleb ·
A new llama.cpp CUDA PR targets duplicate activation quantization and intermediate FP32 traffic in dense Q4_K MLP prefill, with contributor-reported gains from 4.1% to 14.7%.
By bettercallcaleb ·
A new llama.cpp prototype wires Qwen3.8-Flash-Next's built-in NextN/MTP head into speculative decoding. Here is what it changes, the reported speedup, and why mainline still cannot use it.
By bettercallcaleb ·
A RAMGPT ExactBench research note on Qwen3.8-27B Q4: 94 short deterministic reasoning items, zero wrong answers, and several benchmark-design failures uncovered along the way.
By bettercallcaleb ·
A fixed 16-row quantization batch can cross a Windows allocator boundary, making GGUF conversion time depend dramatically on tensor shape.
By bettercallcaleb ·
I asked ChatGPT how hard it would be to make a token and ended with one billion RAMG on Base Sepolia, backed by public source and an on-chain receipt.
By bettercallcaleb ·
Independent RTX 4090 testing shows Qwen3.8 Flash-Next decode falling from 21.00 to 6.93 tok/s under concurrent STREAM traffic, while matched CPU-only contention reduced throughput by only 5.4%.
By bettercallcaleb ·
RAMGPT's tensor-aware Qwen3-4B recipe cut mean full-vocabulary KL 4.72% vs Unsloth UD-Q4_K_XL while producing a 0.10% smaller GGUF.
By bettercallcaleb ·
Independent RTX 4090 CUDA testing of adaptive DFlash2 in llama.cpp: fixed n=7 reached 80.11 tok/s on structured JSON, versus 70.61 tok/s for adaptive 3–7.
By bettercallcaleb ·
Explore three published GGUF comparison groups, inspect KL and decision-flip results, and download the QuantBench forensic summaries and comparison registry.
By bettercallcaleb ·
A detailed look at llama.cpp v0.2.0: CUDA decode tuning, DSpark for LFM2, KV-cache work, lower-memory quantization, server hardening, and the new stable release track.
By bettercallcaleb ·
A reproducible guide to training LightGBM on Yahoo Finance data, ranking stocks before each week, buying the first session, and exiting the last.
By bettercallcaleb ·
What I learned hardening an open-source coding agent for enterprise use, including controls for MCP, shell execution, secrets, downloads, and runtime trust.
By bettercallcaleb ·
Qwen3.8-27B is here. See which GGUF fits 24GB GPUs, the best llama.cpp starting point, context limits, MTP, and what makes this 27B model exciting.
By bettercallcaleb ·
A practical first-person workflow for using ChatGPT to understand technical subjects, test your reasoning, create practice questions, and verify important answers.
By bettercallcaleb ·