Local AI

Qwen3.8 Flash-Next Lost 67% of Its Decode Speed Under Memory Contention

Independent RTX 4090 testing shows Qwen3.8 Flash-Next decode falling from 21.00 to 6.93 tok/s under concurrent STREAM traffic, while matched CPU-only contention reduced throughput by only 5.4%.

Approximately 12 min read

A fresh LocalLLaMA report described a striking failure mode for Qwen3.8 Flash-Next on a memory-rich workstation: decode throughput was healthy while the machine was otherwise idle, then collapsed when another memory-heavy workload ran at the same time.

The obvious question was whether this was really a memory-subsystem effect or simply CPU scheduling pressure.

I reproduced the slowdown on a very different system and added a CPU-only control.

The result was unusually clean.

Condition Decode throughput Change vs idle
Idle 21.00 tok/s —
Four CPU stress workers 19.93 tok/s -5.4%
Four STREAM memory workers 6.93 tok/s -67.0%
Recovery after STREAM 21.08 tok/s +0.4%

The CPU-only and STREAM tests drove system CPU utilization to nearly the same level: 55.6% versus 57.8%. But only the STREAM workload starved the model badly enough to cut decode speed by two thirds.

During STREAM contention, GPU utilization fell from 54.4% to 22.7% and GPU power fell from 104.2 W to 78.3 W. VRAM usage stayed flat at roughly 6993 MiB. Major faults and swap-in activity remained essentially zero.

On this configuration, CPU pressure alone cannot explain the collapse. Concurrent memory traffic is the dominant factor.

That does not prove that every Flash-Next deployment is memory-bandwidth-bound. It does show that a model can look perfectly healthy in an isolated local benchmark and then lose most of its throughput when a second workload starts competing for the host memory subsystem.

Test system

The benchmark was run on one machine and one GPU:

Component Configuration
GPU NVIDIA GeForce RTX 4090 24 GB
CPU AMD Ryzen 5 7600, 6 cores / 12 threads
System RAM 128 GB
Backend CUDA
llama.cpp build 10709, commit a7cc83bba
Model Qwen3.8-Flash-Next UD-Q4_K_XL
GPU layers 999
CPU MoE layers 48
Context 8192
llama.cpp threads 6
Generated tokens 256
Repetitions per stage 3
Seed 12345
Temperature 0.0
Prompt cache disabled
EOS handling ignored for the fixed generation run

The benchmark used one persistent llama-server process per experiment. The model was loaded once, warmed up once, then measured through three stages: idle, contention, and recovery.

The CPU-only control used the same model, prompt, generation settings, worker count, and stage structure.

The prompt was synthetic

The prompt was deliberately synthetic. It contained repeated Python service and concurrency snippets generated by the benchmark script and tokenized to 4899 prompt tokens.

This matters.

The original community report also raised a separate question about synthetic versus real agentic coding context. This experiment does not answer that question. It isolates the contention effect first.

Because the prompt is synthetic, these numbers should not be read as a claim about end-to-end coding-agent throughput. They are a controlled systems benchmark designed to make the before/after comparison stable.

Primary STREAM result

The second clean STREAM run produced:

===== idle =====
run=1 prompt=4899 pp=152.51 t/s tg=21.05 t/s
run=2 prompt=4899 pp=152.36 t/s tg=21.01 t/s
run=3 prompt=4899 pp=152.51 t/s tg=20.95 t/s

===== contention =====
run=1 prompt=4899 pp=43.83 t/s tg=6.98 t/s
run=2 prompt=4899 pp=44.04 t/s tg=6.83 t/s
run=3 prompt=4899 pp=43.88 t/s tg=6.98 t/s

===== recovery =====
run=1 prompt=4899 pp=152.38 t/s tg=21.11 t/s
run=2 prompt=4899 pp=152.13 t/s tg=21.03 t/s
run=3 prompt=4899 pp=152.45 t/s tg=21.12 t/s

===== RESULT =====
idle            21.00 t/s
contention       6.93 t/s
recovery        21.08 t/s
decode_drop       67.0%
recovery         100.4%

Prefill was hit even harder in absolute terms. Mean prefill throughput fell from about 152.46 tok/s to 43.92 tok/s, a reduction of roughly 71%.

A previous STREAM run showed almost the same behavior:

idle            21.02 t/s
contention       7.24 t/s
recovery        21.08 t/s
decode_drop       65.5%
recovery         100.3%

So the effect reproduced across two independent executions.

How much competing memory traffic?

stress-ng --stream 4 reported the following per-worker rates in the primary run:

instance 0: 4901.11 MB/s read, 3267.41 MB/s write
instance 1: 5265.09 MB/s read, 3510.06 MB/s write
instance 2: 5285.79 MB/s read, 3523.86 MB/s write
instance 3: 4794.60 MB/s read, 3196.40 MB/s write

Summed across the four workers, that is approximately 20.25 GB/s of reported read traffic plus 13.50 GB/s of reported write traffic.

These are stress-ng self-reported rates. stress-ng explicitly warns that its STREAM stressor is only loosely based on STREAM and that the numbers should not be submitted as official STREAM benchmark results. I therefore use them only as an indication of the amount of concurrent memory traffic, not as a calibrated measurement of peak DRAM bandwidth.

The CPU-only control

The biggest confounder in a --stream 4 test is obvious: four STREAM workers consume CPU time as well as memory bandwidth.

So I repeated the experiment with four CPU workers using --cpu-method sqrt.

===== idle =====
run=1 prompt=4899 pp=150.86 t/s tg=21.08 t/s
run=2 prompt=4899 pp=150.94 t/s tg=21.04 t/s
run=3 prompt=4899 pp=150.99 t/s tg=21.06 t/s

===== contention =====
run=1 prompt=4899 pp=150.81 t/s tg=19.90 t/s
run=2 prompt=4899 pp=150.08 t/s tg=19.85 t/s
run=3 prompt=4899 pp=149.76 t/s tg=20.04 t/s

===== recovery =====
run=1 prompt=4899 pp=150.80 t/s tg=21.10 t/s
run=2 prompt=4899 pp=150.42 t/s tg=21.10 t/s
run=3 prompt=4899 pp=150.78 t/s tg=21.10 t/s

===== RESULT =====
idle            21.06 t/s
contention      19.93 t/s
recovery        21.10 t/s
decode_drop        5.4%
recovery         100.2%

The monitor measured:

Metric CPU-only idle CPU-only contention STREAM idle STREAM contention
CPU utilization 23.0% 55.6% 23.3% 57.8%
GPU utilization 53.4% 55.7% 54.4% 22.7%
GPU power 103.2 W 103.9 W 104.2 W 78.3 W
VRAM 6993 MiB 6993 MiB 6993 MiB 6993 MiB
Mean major faults/s 0.00 0.02 0.01 0.01
Swap-in/s 0.00 0.00 0.00 0.00

This is the key control.

CPU utilization under CPU-only stress and STREAM stress was nearly identical, but the model behaved completely differently. Under CPU-only load, the GPU stayed fed and decode lost only 5.4%. Under STREAM load, GPU activity collapsed and decode lost 67%.

The most plausible interpretation is that host-memory traffic is starving the execution path feeding the GPU and CPU-resident MoE work.

Memory accounting: do not confuse VRAM, RAM, mmap and swap

Memory accounting for a large GGUF model is layered.

The measured 6993 MiB VRAM is only device memory. It does not tell us how much of the model is being serviced from host RAM.

The model also uses CPU-resident expert tensors because the run specifies:

--n-cpu-moe 48

In addition, llama.cpp uses mmap for GGUF loading by default. File-backed mapped pages, process RSS, page cache, MemAvailable, committed virtual address space, and swap allocation are different quantities. A large mapped model therefore cannot be reduced to a single “RAM used” number from one system counter.

The monitor showed roughly 115 GB MemAvailable during these runs and approximately 2 GB of swap already allocated by the operating system, but the experiment did not record per-process RSS or hardware memory-controller counters. Those quantities are therefore not claimed here.

What was measured is more limited and more useful for this question: during the slowdown, major faults stayed near zero, swap-in stayed at zero, VRAM stayed constant, and throughput recovered immediately after the competing workload stopped.

That makes a paging explanation unlikely for the measured collapse.

Reproduction scripts

The following scripts are the benchmark scripts with user-specific filesystem paths removed. Use a placeholder or relative path for the model.

1. STREAM contention runner

Save as run_flashnext_contention.sh:

#!/usr/bin/env bash
set -euo pipefail

MODEL="${1:-}"
PROMPT_FILE="${2:-}"

LLAMA_SERVER="${LLAMA_SERVER:-./llama.cpp/build/bin/llama-server}"
GPU="${GPU:-0}"
PORT="${PORT:-8099}"
CTX="${CTX:-8192}"
NGL="${NGL:-999}"
CPU_MOE="${CPU_MOE:-48}"
THREADS="${THREADS:-6}"
N_PREDICT="${N_PREDICT:-256}"
REPS="${REPS:-3}"
STRESS_WORKERS="${STRESS_WORKERS:-4}"
STRESS_WARMUP="${STRESS_WARMUP:-8}"
RECOVERY_WAIT="${RECOVERY_WAIT:-10}"

if [[ -z "$MODEL" ]]; then
    echo "Usage: $0 MODEL.gguf [prompt.txt]"
    exit 1
fi

for x in curl python3 stress-ng; do
    command -v "$x" >/dev/null || { echo "Missing: $x"; exit 1; }
done

if [[ ! -x "$LLAMA_SERVER" ]]; then
    echo "llama-server not found: $LLAMA_SERVER"
    exit 1
fi

if [[ ! -f "$MODEL" ]]; then
    echo "Model not found: $MODEL"
    exit 1
fi

STAMP="$(date +%Y%m%d-%H%M%S)"
OUT="$PWD/results/flashnext-contention-$STAMP"
mkdir -p "$OUT"

STAGE_FILE="/tmp/flashnext_stage"
SERVER_LOG="$OUT/server.log"
STRESS_LOG="$OUT/stress-ng.log"
CSV="$OUT/results.csv"
AUTO_PROMPT="$OUT/prompt.txt"
PAYLOAD="$OUT/payload.json"
SERVER_PID=""
STRESS_PID=""

cleanup() {
    echo "stopped" > "$STAGE_FILE"
    if [[ -n "${STRESS_PID:-}" ]] && kill -0 "$STRESS_PID" 2>/dev/null; then
        kill -INT "$STRESS_PID" 2>/dev/null || true
        wait "$STRESS_PID" 2>/dev/null || true
    fi
    if [[ -n "${SERVER_PID:-}" ]] && kill -0 "$SERVER_PID" 2>/dev/null; then
        kill "$SERVER_PID" 2>/dev/null || true
        wait "$SERVER_PID" 2>/dev/null || true
    fi
}
trap cleanup EXIT INT TERM

if [[ -z "$PROMPT_FILE" ]]; then
python3 - "$AUTO_PROMPT" <<'PY'
import sys
out = sys.argv[1]
parts = ["You are reviewing a large production Python codebase. Analyze correctness, concurrency, memory usage, error handling, API boundaries, and performance.\n"]
for i in range(35):
    parts.append(f"""
def process_request_{i}(items, cache, executor):
    futures = []
    for item in items:
        if item.key not in cache:
            futures.append(executor.submit(load_record, item.key))
        else:
            update_record(cache[item.key])
    results = [f.result() for f in futures]
    return results

class Service_{i}:
    def __init__(self, repository):
        self.repository = repository
        self.cache = {{}}

    def execute(self, request):
        records = self.repository.fetch(request.account_id)
        return process_request_{i}(records, self.cache, request.executor)
""")
with open(out, "w") as f:
    f.write("\n".join(parts))
PY
    PROMPT_FILE="$AUTO_PROMPT"
fi

python3 - "$PROMPT_FILE" "$PAYLOAD" "$N_PREDICT" <<'PY'
import json
import sys
prompt_file, payload_file, n_predict = sys.argv[1], sys.argv[2], int(sys.argv[3])
with open(prompt_file) as f:
    prompt = f.read()
payload = {
    "prompt": prompt,
    "n_predict": n_predict,
    "ignore_eos": True,
    "cache_prompt": False,
    "seed": 12345,
    "temperature": 0.0,
    "stream": False
}
with open(payload_file, "w") as f:
    json.dump(payload, f)
PY

echo "timestamp,stage,run,prompt_tokens,predicted_tokens,prompt_ms,predicted_ms,prompt_tps,decode_tps" > "$CSV"

"$LLAMA_SERVER" --version 2>&1 | tee "$OUT/llama-version.txt"

CUDA_VISIBLE_DEVICES="$GPU" "$LLAMA_SERVER" \
    -m "$MODEL" \
    -ngl "$NGL" \
    --n-cpu-moe "$CPU_MOE" \
    -c "$CTX" \
    -t "$THREADS" \
    --host 127.0.0.1 \
    --port "$PORT" \
    > "$SERVER_LOG" 2>&1 &
SERVER_PID=$!

READY=0
for _ in $(seq 1 300); do
    if curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
        READY=1
        break
    fi
    if ! kill -0 "$SERVER_PID" 2>/dev/null; then
        tail -100 "$SERVER_LOG" || true
        exit 1
    fi
    sleep 1
done

if [[ "$READY" -ne 1 ]]; then
    tail -100 "$SERVER_LOG" || true
    exit 1
fi

echo "warmup" > "$STAGE_FILE"
curl -sf -H "Content-Type: application/json" --data-binary "@$PAYLOAD" \
    "http://127.0.0.1:$PORT/completion" > "$OUT/warmup.json"

run_stage() {
    local stage="$1"
    echo "$stage" > "$STAGE_FILE"
    for run in $(seq 1 "$REPS"); do
        local response="$OUT/${stage}-${run}.json"
        curl -sf -H "Content-Type: application/json" --data-binary "@$PAYLOAD" \
            "http://127.0.0.1:$PORT/completion" > "$response"
        python3 - "$response" "$CSV" "$stage" "$run" <<'PY'
import csv
import datetime
import json
import sys
response_file, csv_file, stage, run = sys.argv[1:5]
with open(response_file) as f:
    data = json.load(f)
t = data["timings"]
row = [
    datetime.datetime.now().isoformat(), stage, run,
    t.get("prompt_n"), t.get("predicted_n"),
    t.get("prompt_ms"), t.get("predicted_ms"),
    t.get("prompt_per_second"), t.get("predicted_per_second")
]
with open(csv_file, "a", newline="") as f:
    csv.writer(f).writerow(row)
print(f"run={run} prompt={t.get('prompt_n')} pp={t.get('prompt_per_second', 0):.2f} t/s tg={t.get('predicted_per_second', 0):.2f} t/s")
PY
    done
}

run_stage idle

stress-ng --stream "$STRESS_WORKERS" --metrics-brief > "$STRESS_LOG" 2>&1 &
STRESS_PID=$!
sleep "$STRESS_WARMUP"
run_stage contention
kill -INT "$STRESS_PID" 2>/dev/null || true
wait "$STRESS_PID" 2>/dev/null || true
STRESS_PID=""

echo "recovery_wait" > "$STAGE_FILE"
sleep "$RECOVERY_WAIT"
run_stage recovery
echo "done" > "$STAGE_FILE"

python3 - "$CSV" <<'PY'
import csv
import statistics
import sys
rows = list(csv.DictReader(open(sys.argv[1])))
groups = {}
for r in rows:
    groups.setdefault(r["stage"], []).append(float(r["decode_tps"]))
print("===== RESULT =====")
for stage in ["idle", "contention", "recovery"]:
    v = groups.get(stage, [])
    if v:
        print(f"{stage:12s} {statistics.mean(v):8.2f} t/s   {v}")
if groups.get("idle") and groups.get("contention"):
    idle = statistics.mean(groups["idle"])
    cont = statistics.mean(groups["contention"])
    print(f"decode_drop   {(idle-cont)/idle*100:8.1f}%")
if groups.get("idle") and groups.get("recovery"):
    idle = statistics.mean(groups["idle"])
    rec = statistics.mean(groups["recovery"])
    print(f"recovery      {rec/idle*100:8.1f}%")
PY

cat "$STRESS_LOG" || true
trap - EXIT
cleanup

Run it with a relative path or placeholder:

GPU=0 CPU_MOE=48 ./run_flashnext_contention.sh \
  ./models/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

For a sharded GGUF, point llama.cpp at the first shard.

2. CPU-only control

The CPU-only control used the same runner. Replace only the STREAM launch block:

stress-ng --stream "$STRESS_WORKERS" --metrics-brief > "$STRESS_LOG" 2>&1 &

with:

stress-ng \
    --cpu "$STRESS_WORKERS" \
    --cpu-method sqrt \
    --metrics-brief \
    > "$STRESS_LOG" 2>&1 &

Everything else stays identical, including four stress workers.

3. Independent monitoring terminal

The monitoring process was deliberately kept independent from the benchmark runner. Save this as monitor_flashnext.sh and run it in a second terminal before starting the benchmark.

#!/usr/bin/env bash
set -euo pipefail

GPU="${GPU:-0}"
STAGE_FILE="${STAGE_FILE:-/tmp/flashnext_stage}"
INTERVAL="${INTERVAL:-1}"
STAMP="$(date +%Y%m%d-%H%M%S)"
OUT="${1:-$PWD/results/flashnext-monitor-$STAMP}"
mkdir -p "$OUT"
CSV="$OUT/system.csv"

echo "timestamp,stage,cpu_pct,load1,mem_available_mb,swap_used_mb,pgfault_delta,pgmajfault_delta,pswpin_delta,pswpout_delta,gpu_pct,gpu_mem_pct,vram_mb,power_w" > "$CSV"

read_cpu() {
    read -r _ user nice system idle iowait irq softirq steal _ _ < /proc/stat
    local total=$((user + nice + system + idle + iowait + irq + softirq + steal))
    local idleall=$((idle + iowait))
    echo "$total $idleall"
}

read_vm() {
    awk -v key="$1" '$1 == key {print $2}' /proc/vmstat
}

read -r prev_total prev_idle <<< "$(read_cpu)"
prev_pgfault="$(read_vm pgfault)"
prev_pgmajfault="$(read_vm pgmajfault)"
prev_pswpin="$(read_vm pswpin)"
prev_pswpout="$(read_vm pswpout)"
ARMED=0

while true; do
    sleep "$INTERVAL"
    raw_stage="waiting"
    [[ -f "$STAGE_FILE" ]] && raw_stage="$(cat "$STAGE_FILE" 2>/dev/null || echo waiting)"
    [[ "$raw_stage" == "warmup" ]] && ARMED=1
    if [[ "$ARMED" -eq 1 ]]; then stage="$raw_stage"; else stage="waiting"; fi

    read -r total idle <<< "$(read_cpu)"
    delta_total=$((total - prev_total))
    delta_idle=$((idle - prev_idle))
    if (( delta_total > 0 )); then
        cpu_pct="$(awk -v t="$delta_total" -v i="$delta_idle" 'BEGIN {printf "%.1f",100*(t-i)/t}')"
    else
        cpu_pct="0.0"
    fi
    prev_total="$total"
    prev_idle="$idle"

    load1="$(awk '{print $1}' /proc/loadavg)"
    mem_available_kb="$(awk '/MemAvailable:/ {print $2}' /proc/meminfo)"
    swap_total_kb="$(awk '/SwapTotal:/ {print $2}' /proc/meminfo)"
    swap_free_kb="$(awk '/SwapFree:/ {print $2}' /proc/meminfo)"
    mem_available_mb=$((mem_available_kb / 1024))
    swap_used_mb=$(((swap_total_kb - swap_free_kb) / 1024))

    pgfault="$(read_vm pgfault)"
    pgmajfault="$(read_vm pgmajfault)"
    pswpin="$(read_vm pswpin)"
    pswpout="$(read_vm pswpout)"
    pgfault_delta=$((pgfault - prev_pgfault))
    pgmajfault_delta=$((pgmajfault - prev_pgmajfault))
    pswpin_delta=$((pswpin - prev_pswpin))
    pswpout_delta=$((pswpout - prev_pswpout))
    prev_pgfault="$pgfault"
    prev_pgmajfault="$pgmajfault"
    prev_pswpin="$pswpin"
    prev_pswpout="$pswpout"

    gpu_line="$(nvidia-smi -i "$GPU" --query-gpu=utilization.gpu,utilization.memory,memory.used,power.draw --format=csv,noheader,nounits)"
    IFS=',' read -r gpu_pct gpu_mem_pct vram_mb power_w <<< "$gpu_line"
    gpu_pct="$(xargs <<< "$gpu_pct")"
    gpu_mem_pct="$(xargs <<< "$gpu_mem_pct")"
    vram_mb="$(xargs <<< "$vram_mb")"
    power_w="$(xargs <<< "$power_w")"
    ts="$(date --iso-8601=seconds)"

    echo "$ts,$stage,$cpu_pct,$load1,$mem_available_mb,$swap_used_mb,$pgfault_delta,$pgmajfault_delta,$pswpin_delta,$pswpout_delta,$gpu_pct,$gpu_mem_pct,$vram_mb,$power_w" >> "$CSV"

    printf "\r%-14s CPU %5s%% load %5s RAM %6sMB majflt %5s swapin %5s GPU %3s%% VRAM %5sMB %6sW" \
        "$stage" "$cpu_pct" "$load1" "$mem_available_mb" "$pgmajfault_delta" "$pswpin_delta" "$gpu_pct" "$vram_mb" "$power_w"

    if [[ "$ARMED" -eq 1 && ( "$raw_stage" == "done" || "$raw_stage" == "stopped" ) ]]; then
        break
    fi
done

Terminal 2:

GPU=0 ./monitor_flashnext.sh

Terminal 1:

GPU=0 CPU_MOE=48 STRESS_WORKERS=4 \
  ./run_flashnext_contention.sh \
  ./models/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

Then repeat Terminal 1 with the CPU-only stress block.

Next experiment: measure the traffic directly

The matched CPU-load control points toward host-memory contention in this CPU-offloaded configuration. The test used one machine, one quantization, a synthetic 4,899-token prompt, and 48 CPU-resident MoE layers. It did not use hardware DRAM-controller counters, CPU affinity, or NUMA pinning; stress-ng reported the CPU frequency governor as powersave.

A follow-up should vary competing memory traffic, measure memory-controller bandwidth, and compare decode throughput across that range. Real coding traces and alternative MoE placement would then test how well this controlled result transfers to a working application.

References