Local AI Systems & Inference Contributor
bettercallcaleb writes about local AI inference, open-source runtimes, quantization, model behavior, and practical systems experimentation. His work emphasizes implementation details, reproducible methodology, and the gap between headline claims and what software actually does on real hardware.
Published work
38 published
A source-and-runtime audit of BAAI AREX-2: model size, Qwen3.5 multimodal architecture, 262K context, MTP metadata, current llama.cpp support, and what must be verified before local quantization.
By bettercallcaleb ·
A controlled RTX 4090 + RTX 3060 experiment traced a severe Strata Q2_0 multi-GPU prefill slowdown to the Linux expert-copy path. Allowing full CUDA host registration raised 32K prefill from 665 to 2,110 tok/s.
By bettercallcaleb ·
If you searched whether llama.cpp supports NVFP4, the answer depends on the checkpoint, converter, model architecture, backend, and quant-scale path. Use this checklist to identify the real blocker.
By bettercallcaleb ·
Independent EXL3 4.0 bpw measurements for Xiaomi MiMo-V2.6-Distill-Qwen-9B across RTX 4090, RTX 3060 12GB, mixed-GPU layer split, long context, TabbyAPI, vision, and tool calling.
By bettercallcaleb ·
Source analysis of a Sep 28 llama.cpp Vulkan bug where flash attention or large softmax can overwrite cached scratch data, causing a later matmul to reuse corrupted input.
By bettercallcaleb ·
Fix llama.cpp KV-cache allocation failures by reducing context or concurrency, checking cache types and device placement, and reading the backend allocator error.
By bettercallcaleb ·
Diagnose llama.cpp's unknown model architecture 'glm5next' error, verify the runtime you are actually using, and check whether upstream support has landed.
By bettercallcaleb ·
A Sep 25 llama.cpp report finds MTP throughput on an RX 7800 XT dropping from 55.79 to 25.80 t/s across a 146-commit range; Linux RADV does not reproduce it.
By bettercallcaleb ·
A local RTX 4090 agent benchmark shows Gemma 4 winning small tasks on speed while Qwen3.8 keeps agent turns flat by batching parallel tool calls.
By bettercallcaleb ·
A real AI-agent workflow audit shows how task routing, compact state checks, shared validation scripts, and smaller handoffs can reduce context and tool-call overhead.
By bettercallcaleb ·
A fresh vLLM fix traces a DeepSeek-V4 CUDA-graph startup assertion to one stale max_model_len copy left behind after KV-cache auto-fitting reduces context.
By bettercallcaleb ·
We tested public MiMo-V2.6-Distill-Qwen-9B GGUFs on one RTX 4090, comparing loadability, perplexity, speed, exact-task accuracy, and BF16 fidelity.
By bettercallcaleb ·
An independent Qwen3.8 Flash-Next benchmark on RTX 4090 + RTX 3060 showing how auto-fit, manual MoE placement, quantization, and MTP changed decode speed.
By bettercallcaleb ·
A Sep 21 vLLM bug report shows speculative decoding can overrun a shorter draft model's RoPE cache, causing CUDA illegal-memory-access errors and full engine restarts.
By bettercallcaleb ·
A Sep 20 llama.cpp report bisects an MTP speculative-decoding regression to CUDA MoE weighted-reduction fusion: draft acceptance drops from 0.82 to 0.48 and MTP becomes slower than no draft.
By bettercallcaleb ·
A Sep 19 llama.cpp fix traces OpenVINO large-context failures to the backend reporting SIZE_MAX instead of the GPU's single-allocation limit, causing GGML to request one oversized KV-cache buffer.
By bettercallcaleb ·
A Sep 18 vLLM fix traces a silent Mistral-Large-3 accuracy regression to config remapping: YaRN frequency interpolation was preserved, but its no-mscale flag was not reaching DeepSeek-style attention.
By bettercallcaleb ·
A Sep 15 llama.cpp Vulkan fix targets silent wrong results when mul_mat reads a slice of a larger cache, a failure more dangerous than a clean crash.
By bettercallcaleb ·
A source-backed llama.cpp troubleshooting map for unknown architectures, wrong tensor shapes, corrupted GGUFs, CUDA OOM, multimodal mistakes, and runtime regressions.
By bettercallcaleb ·
A fresh llama.cpp draft fix explains an off-by-one scheduler-capacity crash triggered by larger speculative-decoding graphs and --spec-draft-n-max.
By bettercallcaleb ·
PR #28898 proposes moving FP8 and NVFP4 quantization scales into GGML so more scaled safetensors checkpoints can convert directly instead of relying on model-specific handling.
By bettercallcaleb ·
Unsloth did not begin as a generic AI startup. It began with a narrow technical advantage, reproducible notebooks, and direct distribution to the local-LLM community.
By bettercallcaleb ·
A fresh llama.cpp Metal PR isolates a large-tensor addressing failure at the 2 GiB slice offset and replaces Tensor API slicing with explicit 64-bit address calculation.
By bettercallcaleb ·
A paired benchmark of refusal behavior, calibrated reasoning, open-ended response drift, and epistemic integrity in a 27B Base vs Heretic model comparison.
By bettercallcaleb ·
A new llama.cpp CUDA PR targets duplicate activation quantization and intermediate FP32 traffic in dense Q4_K MLP prefill, with contributor-reported gains from 4.1% to 14.7%.
By bettercallcaleb ·
A new llama.cpp prototype wires Qwen3.8-Flash-Next's built-in NextN/MTP head into speculative decoding. Here is what it changes, the reported speedup, and why mainline still cannot use it.
By bettercallcaleb ·
A RAMGPT ExactBench research note on Qwen3.8-27B Q4: 94 short deterministic reasoning items, zero wrong answers, and several benchmark-design failures uncovered along the way.
By bettercallcaleb ·
A fixed 16-row quantization batch can cross a Windows allocator boundary, making GGUF conversion time depend dramatically on tensor shape.
By bettercallcaleb ·
I asked ChatGPT how hard it would be to make a token and ended with one billion RAMG on Base Sepolia, backed by public source and an on-chain receipt.
By bettercallcaleb ·
Independent RTX 4090 testing shows Qwen3.8 Flash-Next decode falling from 21.00 to 6.93 tok/s under concurrent STREAM traffic, while matched CPU-only contention reduced throughput by only 5.4%.
By bettercallcaleb ·
RAMGPT's tensor-aware Qwen3-4B recipe cut mean full-vocabulary KL 4.72% vs Unsloth UD-Q4_K_XL while producing a 0.10% smaller GGUF.
By bettercallcaleb ·
Independent RTX 4090 CUDA testing of adaptive DFlash2 in llama.cpp: fixed n=7 reached 80.11 tok/s on structured JSON, versus 70.61 tok/s for adaptive 3–7.
By bettercallcaleb ·
RAMGPT-QuantBench is a reproducible research project for comparing practical GGUF quantizations under a tightly controlled llama.cpp runtime.
By bettercallcaleb ·
A detailed look at llama.cpp v0.2.0: CUDA decode tuning, DSpark for LFM2, KV-cache work, lower-memory quantization, server hardening, and the new stable release track.
By bettercallcaleb ·
A reproducible guide to training LightGBM on Yahoo Finance data, ranking stocks before each week, buying the first session, and exiting the last.
By bettercallcaleb ·
What I learned hardening an open-source coding agent for enterprise use, including controls for MCP, shell execution, secrets, downloads, and runtime trust.
By bettercallcaleb ·
Qwen3.8-27B is here. See which GGUF fits 24GB GPUs, the best llama.cpp starting point, context limits, MTP, and what makes this 27B model exciting.
By bettercallcaleb ·
A practical first-person workflow for using ChatGPT to understand technical subjects, test your reasoning, create practice questions, and verify important answers.
By bettercallcaleb ·