I Ran DeepSeek at 23 Tokens/s on a Single RTX 4090
DeepSeek V4.1 Flash reached 20–23 output tokens per second on one RTX 4090 with tandem.cpp. Bigger than VRAM, fast enough to use, and speculative decoding made it slower.
Research · Guides · Explainers
Independent AI research & technical writing
The RAMGPT library / 58 articles
Practical guidance for running private AI tools on your own hardware.
DeepSeek V4.1 Flash reached 20–23 output tokens per second on one RTX 4090 with tandem.cpp. Bigger than VRAM, fast enough to use, and speculative decoding made it slower.
A source-led audit of vLLM v0.33.0 breaking changes: CUDA 12.9 artifacts, removed Medusa and MLP speculators, the Ray executor, NIXL 1.5 and stricter API behaviour.
A 25-minute local coding failure turned into a verified patch after fixing OpenCode's working directory. Here is what our benchmark actually measured.
A November 1 vLLM-Ascend bug shows how a removed environment variable, skipped validation and missing manager forwarding can silently disable hybrid KV-cache retention.
An October 31 llama.cpp patch removed letterbox padding from Gemma 4 vision preprocessing. Audit the source, the reported embedding-cosine results, and a controlled parity test.
An evidence-led analysis of the October 29 Cursor marketplace compromise: tampered VSIX provenance, JavaScript execution, SSH persistence, detection confidence and extension supply-chain controls.
A source-level audit of learned attention gates for prompt-injection defense: 18,432 parameters, adaptive-attack results, trust-channel risks, and a reproducible test plan.
GitHub's new local-and-cloud model routing separates inference location from routing control. Here is the actual trust boundary, hardware gate, billing model and a falsifiable evaluation plan.
How cross-repository Xet file deduplication works, why cache sizes can mislead, and how to delete models without breaking shared references.
A source-grounded guide to CUDA graph capture, persistent tensor addresses, graph-private memory pools, safe PyTorch replay, and honest inference latency comparisons.
Why changing batch sizes can trigger new torch.compile graphs, how to log guard failures, and how to evaluate dynamic-shape trade-offs without confusing compile time with inference speed.
Diagnose vLLM 0.31.0 concurrency using max_num_active_seqs, max_num_seqs, and queue admission: configuration, failure modes, and a controlled test plan.
A version-aware guide to vLLM import failures after PyTorch or CUDA changes, with environment checks, clean-install controls, and a recovery decision tree.
A guide to vLLM automatic prefix caching, the purpose of cache_salt, hash collision risks, and a controlled latency test that separates cache reuse from isolation.
A practical guide to vLLM response-level timing: distinguish queue wait, time to first token, decode speed, streaming usage and Prometheus aggregates.
An October 10 vLLM report traces a long-running DGX Spark inference freeze to a cuDNN FP8 GEMM path. Symptoms, control tests, and an attributed kernel-selection workaround.
llama.cpp v0.6.0 adds llama_batch_ext and llama_process for mixed token and embedding input. Here is what C API integrators need to test before upgrading.
A vLLM 0.30/0.31 report isolates output corruption to prefix-cache hits combined with hidden-state speculative drafters on Qwen3.8-27B NVFP4.
vLLM issue #60008 reports an 11–16% output-throughput cost from Mamba align-mode prefix caching with zero cache hits, exposing host-side launch and prefill-splitting overhead.
A source-level look at llama.cpp PR #29963, which explores pipeline parallelism when MoE expert weights remain in host RAM and must be uploaded to the GPU that owns each layer.
Merged llama.cpp PR 29958 stabilizes k-pool graph shapes for GLM-5.3 and Qwen4exp, addressing reallocation after mutable state changed the reserved graph.
An October 3 llama.cpp change rewrites Qwen4exp indexer scoring around the lightning indexer, reducing live score-buffer memory and extending backend handling across CUDA, Metal, and Vulkan.
A local-deployment audit of BAAI AREX-2: its Qwen3.8 base, Qwen3.5-compatible model class, memory budget, multimodal conversion and reference-runtime checks.
Independent EXL3 4.0 bpw measurements for Xiaomi MiMo-V2.6-Distill-Qwen-9B across RTX 4090, RTX 3060 12GB, mixed-GPU layer split, long context, TabbyAPI, vision, and tool calling.
Source analysis of a Sep 28 llama.cpp Vulkan bug where flash attention or large softmax can overwrite cached scratch data, causing a later matmul to reuse corrupted input.
An independent Qwen3.8 Flash-Next benchmark on RTX 4090 + RTX 3060 showing how auto-fit, manual MoE placement, quantization, and MTP changed decode speed.
PR #28898 proposes moving FP8 and NVFP4 quantization scales into GGML so more scaled safetensors checkpoints can convert directly instead of relying on model-specific handling.
Unsloth did not begin as a generic AI startup. It began with a narrow technical advantage, reproducible notebooks, and direct distribution to the local-LLM community.
A source-level look at llama.cpp PR #28809, which makes server router discovery prefer a GGUF whose filename matches its model directory instead of relying on directory iteration order.
PR 28751 traces multi-image llama.cpp latency to repeated scheduler reservation when causal attention changes around vision chunks.
A new llama.cpp prototype wires Qwen3.8-Flash-Next's built-in NextN/MTP head into speculative decoding. Here is what it changes, the reported speedup, and why mainline still cannot use it.
llama.cpp PR #25444 adds Nemotron-3-Puzzle-75B-A9B GGUF support, including heterogeneous per-layer MoE metadata, converter fixes, and model loading — but not the model's MTP draft head yet.
A RAMGPT ExactBench research note on Qwen3.8-27B Q4: 94 short deterministic reasoning items, zero wrong answers, and several benchmark-design failures uncovered along the way.
A source-level look at llama.cpp PR #28024: how fusing RMSNorm, residual math, RoPE, views, and row writes cuts Vulkan graph overhead in Gemma 4.
A source-driven history of the increasingly uncomfortable relationship between Ollama and llama.cpp: upstream engineering, attribution disputes, forks, compatibility failures, community backlash, and the cost of hiding the engine.
A fresh llama.cpp issue shows how MSVC feature detection can silently compile out AVX-VNNI paths even when the CPU supports them.
A fixed 16-row quantization batch can cross a Windows allocator boundary, making GGUF conversion time depend dramatically on tensor shape.
A llama.cpp Qwen4 experimental report exposes how tiny inference-time reads can defeat fast storage, NFS caching, and otherwise capable local-AI infrastructure.
A production-inference analysis of PR #28320, which temporarily reclaims llama.cpp compute scratch so a vision projector can run on GPU without reducing context.
A llama.cpp Vulkan guard rejected valid hybrid-model KV-cache views, silently disabling a faster Flash Attention path for quantized caches.
Two new llama.cpp reports show how batch-dependent CUDA kernels can break greedy speculative-decoding equivalence on Blackwell GPUs.
A forensic analysis of llama.cpp issue #28049, where accepted tokens past EOG can invalidate reusable state and trigger large re-prefill costs.
A source-path experiment on the first LongCat-Flash-Lite-Sparse llama.cpp fork: LSA token selection is real, but selected tokens are converted into a mask over full K/V rather than compact sparse attention execution.
Independent RTX 4090 testing shows Qwen3.8 Flash-Next decode falling from 21.00 to 6.93 tok/s under concurrent STREAM traffic, while matched CPU-only contention reduced throughput by only 5.4%.
RAMGPT's tensor-aware Qwen3-4B recipe cut mean full-vocabulary KL 4.72% vs Unsloth UD-Q4_K_XL while producing a 0.10% smaller GGUF.
A code-level analysis of llama.cpp's DFlash2 merge: dynamic local convolution, lattice path selection, runtime integration, benchmarks, and review tradeoffs.
Implementation strategy for Qwen3.8-Flash-Next in llama.cpp: Gated DeltaNet reuse, Hyper-Connections, PLE, QSA, state handling, CUDA, and GGUF.
PR #27742 shows Qwen4 needs no new GGML ops; the hard engineering is indexed state, PLE residency, quantizer staging, and backend memory invariants.
A systems audit of Qwen4Exp in llama.cpp: state ownership, indexer-cache lifecycle, multi-slot correctness, quantized KV, graph reservation, and PLE.
Independent RTX 4090 CUDA testing of adaptive DFlash2 in llama.cpp: fixed n=7 reached 80.11 tok/s on structured JSON, versus 70.61 tok/s for adaptive 3–7.
A beginner-friendly guide to local voice cloning with Audio8 TTS Preview 0.1B, from reference audio and transcripts to generation and FFmpeg speed control.
Explore three published GGUF comparison groups, inspect KL and decision-flip results, and download the QuantBench forensic summaries and comparison registry.
A detailed look at llama.cpp v0.2.0: CUDA decode tuning, DSpark for LFM2, KV-cache work, lower-memory quantization, server hardening, and the new stable release track.
An independent llama.cpp PCTree test on RTX 4090 with Qwen3-8B Q8: DSpark doubled decode speed, while wider trees raised acceptance but still lost throughput.
Why llama.cpp is evolving from a GGUF model runner into a broader local inference stack spanning servers, scheduling, KV caches, and speculative decoding.
A hands-on guide to running Qwen3.8-27B on one RTX 4090 with patched vLLM, 64K context, MTP speculative decoding, tool calling, and OpenWebUI.
A lively look at how ik_llama.cpp split from llama.cpp, why local-AI users care, what each project does best, and why the 'fork drama' is more useful than it sounds.
Learn how to run an LLM locally on Windows, Linux, or macOS, choose a model and quantization, understand hardware needs, and avoid common mistakes.