Technical library / 132 articles
Independent experiments, practical procedures, and clear explanations. Newest first.
132 articles · Newest first
No articles match those filters. Try fewer words or choose “All topics”.
AI Foundations #28 explains static batching, continuous batching, request scheduling, chunked prefill, head-of-line blocking, fairness, and why server throughput and user latency can pull in opposite directions.
Diagnose GGML_ASSERT(ret.axis != GGML_BACKEND_SPLIT_AXIS_UNKNOWN) by separating tensor-split graph incompatibility from ordinary VRAM problems, then testing layer split, KV types, model paths and revisions.
If a Qwen3.8-Flash-Next MTP draft GGUF fails on output_hc_norm.weight, separate target-model loading from detached draft-head compatibility, inspect the sidecar tensor layout, and verify the exact llama.cpp MTP implementation.
An October 3 llama.cpp change rewrites Qwen4exp indexer scoring around the lightning indexer, reducing live score-buffer memory and extending backend handling across CUDA, Metal, and Vulkan.
A source-and-runtime audit of BAAI AREX-2: model size, Qwen3.5 multimodal architecture, 262K context, MTP metadata, current llama.cpp support, and what must be verified before local quantization.
Holo4 is a new computer-use VLM family built to click, type, run code, and call tools. Here is what the 27B and 35B-A3B models are actually for.
AI Foundations #27 explains time to first token, time per output token, prefill, decode, batching, throughput, context length, and why one tokens-per-second number can hide the real user experience.
Diagnose llama.cpp File Not Found errors by checking the actual GGUF path, process working directory, split-model shards, router configuration, and file readability before blaming model compatibility.
Current llama.cpp supports MTP speculative decoding through draft-mtp. Learn how to verify model support, enable it, isolate failed MTP contexts, and measure whether it actually improves your workload.
Fix the Qwen3.5-family blk.32.attn_norm.weight missing-tensor error when config metadata declares an MTP/NextN layer that the checkpoint does not contain.
Fix the llama.cpp error unknown model architecture 'mllama' by recognizing that current llama.cpp does not implement the Mllama vision architecture and choosing a supported path.
AI Foundations #26 explains production monitoring, data and concept drift, feedback loops, human review, and why a model that passed evaluation can still fail after deployment.
A controlled RTX 4090 + RTX 3060 experiment traced a severe Strata Q2_0 multi-GPU prefill slowdown to the Linux expert-copy path. Allowing full CUDA host registration raised 32K prefill from 665 to 2,110 tok/s.
A Sep 30 vLLM report traces a large H100 decode regression to FA4 ignoring SplitKV num_splits on SM90; the one-line kernel fix is merged upstream but not yet pinned by vLLM.
AI Foundations #25 explains alignment, preference training, guardrails, instruction hierarchy, runtime policy, and why capability and safe system behavior are different goals.
If you searched whether llama.cpp supports NVFP4, the answer depends on the checkpoint, converter, model architecture, backend, and quant-scale path. Use this checklist to identify the real blocker.
Find the largest llama.cpp context that fits your GPU by separating weights, KV cache, compute buffers, concurrency, auto-fit, and backend limits.
Independent EXL3 4.0 bpw measurements for Xiaomi MiMo-V2.6-Distill-Qwen-9B across RTX 4090, RTX 3060 12GB, mixed-GPU layer split, long context, TabbyAPI, vision, and tool calling.
A Sep 29 vLLM bug shows that adapting OpenAI-style tool call IDs to Mistral by keeping only the last nine characters can reject valid requests or create duplicate IDs.
Fix the llama.cpp error 'LLAMA_SPLIT_MODE_TENSOR not implemented for architecture qwen4exp' by using a supported split mode and understanding the architecture gate.
Source analysis of a Sep 28 llama.cpp Vulkan bug where flash attention or large softmax can overwrite cached scratch data, causing a later matmul to reuse corrupted input.
Explains what dbirks/Qwen3.8-27B-W4A16-AutoRound actually contains, why vLLM is its native target, and what current llama.cpp conversion does with pack-quantized weights.
AI Foundations #24 explains tool use, function calling, agents, action loops, observations, permissions, failure modes, and why an LLM does not execute a tool by itself.
Fix llama.cpp check_tensor_dims failures by separating missing tensors from wrong shapes, then checking model metadata, converter provenance, runtime support, and MTP assumptions.
Fix llama.cpp compute pp buffer allocation failures by shrinking micro-batches, preserving device-memory headroom, checking backend limits, fit warnings, and speculative decoding.
An independent Ubuntu RTX 4090 reproduction of NInfer's Qwen3.8-27B A8 claims reaches 4,999.5 tok/s pp2048, 500.9 tok/s edit decode, and exact retrieval at 260K context.
AI Foundations #23 explains retrieval-augmented generation, embeddings, chunking, retrieval, context injection, citations, failure modes, and why RAG is different from training.
A Sep 27 vLLM fix traces Qwen3-Omni multimodal failures to an M-RoPE offset double-count that can crash at prompt end or silently shift positions.
AI Foundations #22 explains test sets, metrics, benchmark design, sampling variance, contamination, and why a fair model comparison must control more than one score.
Fix llama.cpp KV-cache allocation failures by reducing context or concurrency, checking cache types and device placement, and reading the backend allocator error.
Diagnose llama.cpp's unknown model architecture 'glm5next' error, verify the runtime you are actually using, and check whether upstream support has landed.
vLLM 0.30.0 can fail before startup on Pixtral-based models because its release code imports a symbol renamed by Transformers 5.17; main already contains the compatibility fix.
A Sep 25 llama.cpp report finds MTP throughput on an RX 7800 XT dropping from 55.79 to 25.80 t/s across a 146-commit range; Linux RADV does not reproduce it.
AI Foundations #21 explains reasoning tokens, inference-time compute, reward signals, RLHF, PPO, and how reinforcement-learning post-training changes model behavior.
llama.cpp issue #29313 and PR #29370 trace an EAGLE-3 acceptance collapse to stale GGML allocation plans that ignore changing tensor input/output lifetimes.
AI Foundations #20 builds a practical mental model of multimodal AI: how text, image, audio, and video representations are converted, aligned, and fed into transformer-style systems.
AI Foundations #19 explains Mixture of Experts models: routers, top-k expert selection, active versus total parameters, load balancing, and why MoE changes inference economics.
SGLang issue #40901 shows that incremental streaming can resend the full accumulated text on graceful abort; PR #40902 fixes the text and token delta bookkeeping.
A local RTX 4090 agent benchmark shows Gemma 4 winning small tasks on speed while Qwen3.8 keeps agent turns flat by batching parallel tool calls.
A real AI-agent workflow audit shows how task routing, compact state checks, shared validation scripts, and smaller handoffs can reduce context and tool-call overhead.
AI Foundations #18 builds quantization from number representation and rounding error, then connects 4-bit weights to model size, speed, calibration, and quality tradeoffs.
A fresh vLLM fix traces a DeepSeek-V4 CUDA-graph startup assertion to one stale max_model_len copy left behind after KV-cache auto-fitting reduces context.
AI Foundations #17 explains context windows and KV cache: what the model can see, what keys and values are cached, why decoding gets faster, and why long chats consume more memory.
We tested public MiMo-V2.6-Distill-Qwen-9B GGUFs on one RTX 4090, comparing loadability, perplexity, speed, exact-task accuracy, and BF16 fidelity.
A first-pass RAMGPT field test of Base, Heretic, and OrcaRouter Qwen3.8-27B variants across AI-agent security, secure code review, and modern glibc exploitation.
An independent Qwen3.8 Flash-Next benchmark on RTX 4090 + RTX 3060 showing how auto-fit, manual MoE placement, quantization, and MTP changed decode speed.
A Sep 21 vLLM bug report shows speculative decoding can overrun a shorter draft model's RoPE cache, causing CUDA illegal-memory-access errors and full engine restarts.
A Sep 20 llama.cpp report bisects an MTP speculative-decoding regression to CUDA MoE weighted-reduction fusion: draft acceptance drops from 0.82 to 0.48 and MTP becomes slower than no draft.
AI Foundations #16 explains how logits become generated text through greedy decoding, temperature, top-k, and top-p sampling, with simple probability intuition and worked examples.
AI Foundations #15 explains inference as the trained model's forward-only phase: tokenize a prompt, compute logits, choose a token, append it, and repeat without backpropagation or weight updates.
A Sep 19 llama.cpp fix traces OpenVINO large-context failures to the backend reporting SIZE_MAX instead of the GPU's single-allocation limit, causing GGML to request one oversized KV-cache buffer.
AI Foundations #14 explains how fine-tuning continues gradient-based training from pretrained weights, using narrower data to change model behavior without relearning language from scratch.
A Sep 18 vLLM fix traces a silent Mistral-Large-3 accuracy regression to config remapping: YaRN frequency interpolation was preserved, but its no-mscale flag was not reaching DeepSeek-style attention.
A Sep 15 llama.cpp Vulkan fix targets silent wrong results when mul_mat reads a slice of a larger cache, a failure more dangerous than a clean crash.
A practical research playbook for reconstructing CVE-centric remediation event logs from change, incident, request, and vulnerability records before forming hypotheses.
AI Foundations #13 connects the Transformer architecture to the training objective that turns random parameters into a useful language model.
A source-backed llama.cpp troubleshooting map for unknown architectures, wrong tensor shapes, corrupted GGUFs, CUDA OOM, multimodal mistakes, and runtime regressions.
A fresh llama.cpp draft fix explains an off-by-one scheduler-capacity crash triggered by larger speculative-decoding graphs and --spec-draft-n-max.
AI Foundations #12 assembles attention, residual connections, normalization, and MLP layers into the repeating Transformer block behind modern language models.
AI Foundations #11 follows tokenization and embeddings into attention: queries, keys, values, similarity scores, softmax weights, and context-dependent representations.
PR #28898 proposes moving FP8 and NVFP4 quantization scales into GGML so more scaled safetensors checkpoints can convert directly instead of relying on model-specific handling.
Unsloth did not begin as a generic AI startup. It began with a narrow technical advantage, reproducible notebooks, and direct distribution to the local-LLM community.
A source-level look at llama.cpp PR #28809, which makes server router discovery prefer a GGUF whose filename matches its model directory instead of relying on directory iteration order.
AI Foundations #10 connects embeddings to tokenization: how text is split into tokens, mapped to vocabulary IDs, and then looked up as learned vectors.
AI Foundations #9 explains embeddings as learned vectors that give neural networks a useful numerical space for tokens and other objects.
PR 28751 traces multi-image llama.cpp latency to repeated scheduler reservation when causal attention changes around vision chunks.
AI Foundations #8 explains backpropagation as repeated chain-rule bookkeeping that carries loss information backward through a neural network to every trainable parameter.
A fresh llama.cpp Metal PR isolates a large-tensor addressing failure at the 2 GiB slice offset and replaces Tensor API slicing with explicit 64-bit address calculation.
AI Foundations #7 explains gradient descent as the rule that turns loss into a direction for changing weights, using one small numerical example before backpropagation.
A paired benchmark of refusal behavior, calibrated reasoning, open-ended response drift, and epistemic integrity in a 27B Base vs Heretic model comparison.
A new llama.cpp CUDA PR targets duplicate activation quantization and intermediate FP32 traffic in dense Q4_K MLP prefill, with contributor-reported gains from 4.1% to 14.7%.
A source-level look at the new vLLM GGUF plugin patch for DeepSeek-V4, the exact architecture-resolution failure it fixes, and why new model names can break before their tensors do.
AI Foundations #6 connects the forward pass to training by showing why a model needs targets and a loss function before its weights can improve.
A new llama.cpp prototype wires Qwen3.8-Flash-Next's built-in NextN/MTP head into speculative decoding. Here is what it changes, the reported speedup, and why mainline still cannot use it.
AI Foundations #5 follows one input through a tiny neural network to show what weights, biases, activations, and layers actually do during a forward pass.
A cybersecurity research report on the gap between fast-moving exploitation and slow enterprise remediation, and why compensating controls should be measured by survival rather than deployment alone.
A new llama.cpp CUDA patch shows that Blackwell NVFP4 prefill was limited by tile delivery, barriers, and register pressure—not simply tensor-core arithmetic.
llama.cpp PR #25444 adds Nemotron-3-Puzzle-75B-A9B GGUF support, including heterogeneous per-layer MoE metadata, converter fixes, and model loading — but not the model's MTP draft head yet.
A RAMGPT ExactBench research note on Qwen3.8-27B Q4: 94 short deterministic reasoning items, zero wrong answers, and several benchmark-design failures uncovered along the way.
AI Foundations #4 explains tensors through shapes, axes, images, batches, model weights, and memory, without assuming you already know linear algebra or PyTorch.
A source-level look at llama.cpp PR #28024: how fusing RMSNorm, residual math, RoPE, views, and row writes cuts Vulkan graph overhead in Gemma 4.
A source-driven history of the increasingly uncomfortable relationship between Ollama and llama.cpp: upstream engineering, attribution disputes, forks, compatibility failures, community backlash, and the cost of hiding the engine.
AI Foundations #3 explains what parameters, weights, and biases actually are, why large models have billions of them, and how precision changes model size.
AI Foundations #1 builds a clear map from artificial intelligence to machine learning, neural networks, deep learning, generative AI, and LLMs.
AI Foundations #2 explains what machine learning actually changes inside a model, using simple math, examples, and a student's point of view.
A fresh llama.cpp issue shows how MSVC feature detection can silently compile out AVX-VNNI paths even when the CPU supports them.
A fixed 16-row quantization batch can cross a Windows allocator boundary, making GGUF conversion time depend dramatically on tensor shape.
A llama.cpp Qwen4 experimental report exposes how tiny inference-time reads can defeat fast storage, NFS caching, and otherwise capable local-AI infrastructure.
I asked ChatGPT how hard it would be to make a token and ended with one billion RAMG on Base Sepolia, backed by public source and an on-chain receipt.
A production-inference analysis of PR #28320, which temporarily reclaims llama.cpp compute scratch so a vision projector can run on GPU without reducing context.
A llama.cpp Vulkan guard rejected valid hybrid-model KV-cache views, silently disabling a faster Flash Attention path for quantized caches.
Two new llama.cpp reports show how batch-dependent CUDA kernels can break greedy speculative-decoding equivalence on Blackwell GPUs.
Independent RTX 4090 testing finds UD-IQ3_XXS only ~2% slower than UD-Q3_K_XL at matched context, while reducing the 200K GPU memory budget from 16 to 15 GiB cuts prompt processing by 19%.
A forensic analysis of llama.cpp issue #28049, where accepted tokens past EOG can invalidate reusable state and trigger large re-prefill costs.
A source-path experiment on the first LongCat-Flash-Lite-Sparse llama.cpp fork: LSA token selection is real, but selected tokens are converted into a mask over full K/V rather than compact sparse attention execution.
Independent RTX 4090 testing shows Qwen3.8 Flash-Next decode falling from 21.00 to 6.93 tok/s under concurrent STREAM traffic, while matched CPU-only contention reduced throughput by only 5.4%.
An RTX 4090 reproduction of Qwen3.8-27B at ~2.5 bpw finds 1.93% lower WikiText-2 perplexity for GSQ-RCO than Unsloth UD-IQ2_S, while tensor dumps reveal a far more heterogeneous precision allocation and slightly lower throughput.
An RTX 4090 reproduction of Qwen3.8-Flash-Next PLE paging shows a 38.4GB table reaching only 221.7MiB resident after an 8K high-diversity prompt sweep.
RAMGPT's tensor-aware Qwen3-4B recipe cut mean full-vocabulary KL 4.72% vs Unsloth UD-Q4_K_XL while producing a 0.10% smaller GGUF.
A code-level analysis of llama.cpp's DFlash2 merge: dynamic local convolution, lattice path selection, runtime integration, benchmarks, and review tradeoffs.
Implementation strategy for Qwen3.8-Flash-Next in llama.cpp: Gated DeltaNet reuse, Hyper-Connections, PLE, QSA, state handling, CUDA, and GGUF.
PR #27742 shows Qwen4 needs no new GGML ops; the hard engineering is indexed state, PLE residency, quantizer staging, and backend memory invariants.
A systems audit of Qwen4Exp in llama.cpp: state ownership, indexer-cache lifecycle, multi-slot correctness, quantized KV, graph reservation, and PLE.
Independent RTX 4090 CUDA testing of adaptive DFlash2 in llama.cpp: fixed n=7 reached 80.11 tok/s on structured JSON, versus 70.61 tok/s for adaptive 3–7.
A beginner-friendly guide to local voice cloning with Audio8 TTS Preview 0.1B, from reference audio and transcripts to generation and FFmpeg speed control.
RAMGPT-QuantBench is a reproducible research project for comparing practical GGUF quantizations under a tightly controlled llama.cpp runtime.
A detailed look at llama.cpp v0.2.0: CUDA decode tuning, DSpark for LFM2, KV-cache work, lower-memory quantization, server hardening, and the new stable release track.
An independent llama.cpp PCTree test on RTX 4090 with Qwen3-8B Q8: DSpark doubled decode speed, while wider trees raised acceptance but still lost throughput.
Why 1-bit and ternary LLMs may reshape local AI, and why tiny weights do not mean one-quarter the inference cost of a Q4 model.
A Whitby high school student explains how she uses ChatGPT for hints, error analysis, proofs, calculus, vectors, and practice without outsourcing the thinking.
Why llama.cpp is evolving from a GGUF model runner into a broader local inference stack spanning servers, scheduling, KV caches, and speculative decoding.
A hands-on guide to running Qwen3.8-27B on one RTX 4090 with patched vLLM, 64K context, MTP speculative decoding, tool calling, and OpenWebUI.
A reproducible guide to training LightGBM on Yahoo Finance data, ranking stocks before each week, buying the first session, and exiting the last.
What I learned hardening an open-source coding agent for enterprise use, including controls for MCP, shell execution, secrets, downloads, and runtime trust.
A simple step-by-step llama.cpp guide for beginners: compile from source, run a GGUF model, use GPU acceleration, and tune the main inference settings.
A lively look at how ik_llama.cpp split from llama.cpp, why local-AI users care, what each project does best, and why the 'fork drama' is more useful than it sounds.
Qwen3.8-27B is here. See which GGUF fits 24GB GPUs, the best llama.cpp starting point, context limits, MTP, and what makes this 27B model exciting.
Learn what seed, inference steps, CFG guidance, samplers and schedulers do in AI image generation, why settings change results, and how to tune them.
Learn what GGUF quantization labels such as Q4_K_M, Q5_K_M, Q6_K and Q8_0 mean, how they affect model size, quality, RAM, VRAM, and speed.
Learn how AI image generators turn text into images, including noise, denoising, latent space, text encoders, guidance, samplers, seeds, and diffusion steps.
Learn how AI video generators create motion from text or images, including latent video, temporal consistency, diffusion transformers, frames, seeds, and camera control.
Learn how OpenAI Whisper turns audio into text using log-Mel spectrograms, encoder-decoder Transformers, language tokens, timestamps, and autoregressive decoding.
A practical first-person workflow for using ChatGPT to understand technical subjects, test your reasoning, create practice questions, and verify important answers.
Learn how to reduce local LLM VRAM usage with quantization, shorter context, KV-cache compression, CPU offloading, smaller models, and runtime tuning.
Learn the difference between speech-to-text and text-to-speech, how ASR and TTS models work, key accuracy metrics, streaming, voice cloning, and local AI use.
Learn what LLM context length means, what counts toward the context window, how tokens, chat history, KV cache, VRAM, truncation, and long-context models work.
Learn how LLM quantization reduces model memory, what FP16, INT8 and INT4 mean, and how GGUF, AWQ, GPTQ, scales, groups, and quality trade-offs work.
Learn why a local LLM runs slowly and how model size, GPU offloading, context length, memory bandwidth, CPU threads, and runtime settings affect speed.
GGUF and AWQ solve different parts of local LLM deployment. Learn how they differ in format, quantization, hardware support, speed, and practical use.
Learn how to estimate LLM VRAM requirements from model size, quantization, KV cache, context length, runtime overhead, and GPU offloading.
Learn how to run an LLM locally on Windows, Linux, or macOS, choose a model and quantization, understand hardware needs, and avoid common mistakes.
Learn what an LLM KV cache stores, why it speeds up token generation, how to estimate its VRAM use, and how context length and quantization affect memory.