Qwen3.8 Flash-Next Lost 67% of Its Decode Speed Under Memory Contention

A fresh LocalLLaMA report described a striking failure mode for Qwen3.8 Flash-Next on a memory-rich workstation: decode throughput was healthy while the machine was otherwise idle, then collapsed when another memory-heavy workload ran at the same time.

The obvious question was whether this was really a memory-subsystem effect or simply CPU scheduling pressure.

I reproduced the slowdown on a very different system and added a CPU-only control.

The result was unusually clean.

The CPU-only and STREAM tests drove system CPU utilization to nearly the same level: 55.6% versus 57.8%. But only the STREAM workload starved the model badly enough to cut decode speed by two thirds.

During STREAM contention, GPU utilization fell from 54.4% to 22.7% and GPU power fell from 104.2 W to 78.3 W. VRAM usage stayed flat at roughly 6993 MiB. Major faults and swap-in activity remained essentially zero.

On this configuration, CPU pressure alone cannot explain the collapse. Concurrent memory traffic is the dominant factor.

That does not prove that every Flash-Next deployment is memory-bandwidth-bound. It does show that a model can look perfectly healthy in an isolated local benchmark and then lose most of its throughput when a second workload starts competing for the host memory subsystem.

Test system

The benchmark was run on one machine and one GPU:

The benchmark used one persistent llama-server process per experiment. The model was loaded once, warmed up once, then measured through three stages: idle, contention, and recovery.

The CPU-only control used the same model, prompt, generation settings, worker count, and stage structure.

The prompt was synthetic

The prompt was deliberately synthetic. It contained repeated Python service and concurrency snippets generated by the benchmark script and tokenized to 4899 prompt tokens.

This matters.

The original community report also raised a separate question about synthetic versus real agentic coding context. This experiment does not answer that question. It isolates the contention effect first.

Because the prompt is synthetic, these numbers should not be read as a claim about end-to-end coding-agent throughput. They are a controlled systems benchmark designed to make the before/after comparison stable.

Primary STREAM result

The second clean STREAM run produced:

===== idle =====
run=1 prompt=4899 pp=152.51 t/s tg=21.05 t/s
run=2 prompt=4899 pp=152.36 t/s tg=21.01 t/s
run=3 prompt=4899 pp=152.51 t/s tg=20.95 t/s

===== contention =====
run=1 prompt=4899 pp=43.83 t/s tg=6.98 t/s
run=2 prompt=4899 pp=44.04 t/s tg=6.83 t/s
run=3 prompt=4899 pp=43.88 t/s tg=6.98 t/s

===== recovery =====
run=1 prompt=4899 pp=152.38 t/s tg=21.11 t/s
run=2 prompt=4899 pp=152.13 t/s tg=21.03 t/s
run=3 prompt=4899 pp=152.45 t/s tg=21.12 t/s

===== RESULT =====
idle            21.00 t/s
contention       6.93 t/s
recovery        21.08 t/s
decode_drop       67.0%
recovery         100.4%

Prefill was hit even harder in absolute terms. Mean prefill throughput fell from about 152.46 tok/s to 43.92 tok/s, a reduction of roughly 71%.

A previous STREAM run showed almost the same behavior:

idle            21.02 t/s
contention       7.24 t/s
recovery        21.08 t/s
decode_drop       65.5%
recovery         100.3%

So the effect reproduced across two independent executions.

How much competing memory traffic?

stress-ng --stream 4 reported the following per-worker rates in the primary run:

instance 0: 4901.11 MB/s read, 3267.41 MB/s write
instance 1: 5265.09 MB/s read, 3510.06 MB/s write
instance 2: 5285.79 MB/s read, 3523.86 MB/s write
instance 3: 4794.60 MB/s read, 3196.40 MB/s write

Summed across the four workers, that is approximately 20.25 GB/s of reported read traffic plus 13.50 GB/s of reported write traffic.

These are stress-ng self-reported rates. stress-ng explicitly warns that its STREAM stressor is only loosely based on STREAM and that the numbers should not be submitted as official STREAM benchmark results. I therefore use them only as an indication of the amount of concurrent memory traffic, not as a calibrated measurement of peak DRAM bandwidth.

The CPU-only control

The biggest confounder in a --stream 4 test is obvious: four STREAM workers consume CPU time as well as memory bandwidth.

So I repeated the experiment with four CPU workers using --cpu-method sqrt.

===== idle =====
run=1 prompt=4899 pp=150.86 t/s tg=21.08 t/s
run=2 prompt=4899 pp=150.94 t/s tg=21.04 t/s
run=3 prompt=4899 pp=150.99 t/s tg=21.06 t/s

===== contention =====
run=1 prompt=4899 pp=150.81 t/s tg=19.90 t/s
run=2 prompt=4899 pp=150.08 t/s tg=19.85 t/s
run=3 prompt=4899 pp=149.76 t/s tg=20.04 t/s

===== recovery =====
run=1 prompt=4899 pp=150.80 t/s tg=21.10 t/s
run=2 prompt=4899 pp=150.42 t/s tg=21.10 t/s
run=3 prompt=4899 pp=150.78 t/s tg=21.10 t/s

===== RESULT =====
idle            21.06 t/s
contention      19.93 t/s
recovery        21.10 t/s
decode_drop        5.4%
recovery         100.2%

The monitor measured:

This is the key control.

CPU utilization under CPU-only stress and STREAM stress was nearly identical, but the model behaved completely differently. Under CPU-only load, the GPU stayed fed and decode lost only 5.4%. Under STREAM load, GPU activity collapsed and decode lost 67%.

The most plausible interpretation is that host-memory traffic is starving the execution path feeding the GPU and CPU-resident MoE work.

Memory accounting: do not confuse VRAM, RAM, mmap and swap

Memory accounting for a large GGUF model is layered.

The measured 6993 MiB VRAM is only device memory. It does not tell us how much of the model is being serviced from host RAM.

The model also uses CPU-resident expert tensors because the run specifies:

--n-cpu-moe 48

In addition, llama.cpp uses mmap for GGUF loading by default. File-backed mapped pages, process RSS, page cache, MemAvailable, committed virtual address space, and swap allocation are different quantities. A large mapped model therefore cannot be reduced to a single “RAM used” number from one system counter.

The monitor showed roughly 115 GB MemAvailable during these runs and approximately 2 GB of swap already allocated by the operating system, but the experiment did not record per-process RSS or hardware memory-controller counters. Those quantities are therefore not claimed here.

What was measured is more limited and more useful for this question: during the slowdown, major faults stayed near zero, swap-in stayed at zero, VRAM stayed constant, and throughput recovered immediately after the competing workload stopped.

That makes a paging explanation unlikely for the measured collapse.

Why the GPU utilization drop matters

A GPU-bound workload normally gets slower because the GPU itself has more work to do or because its clocks, power, or occupancy change unfavorably.

That is not what happened here.

During STREAM contention, the 4090 did less work:

GPU utilization: 54.4% -> 22.7%
GPU power:       104.2W -> 78.3W
decode:          21.00 -> 6.93 tok/s

The GPU was being starved from upstream.

The CPU-only control makes that observation much stronger. At nearly the same total CPU utilization, GPU utilization stayed around 56% and throughput stayed near 20 tok/s.

Reproduction scripts

The following scripts are the benchmark scripts with user-specific filesystem paths removed. Use a placeholder or relative path for the model.

1. STREAM contention runner

Save as run_flashnext_contention.sh:

#!/usr/bin/env bash
set -euo pipefail

MODEL="${1:-}"
PROMPT_FILE="${2:-}"

LLAMA_SERVER="${LLAMA_SERVER:-./llama.cpp/build/bin/llama-server}"
GPU="${GPU:-0}"
PORT="${PORT:-8099}"
CTX="${CTX:-8192}"
NGL="${NGL:-999}"
CPU_MOE="${CPU_MOE:-48}"
THREADS="${THREADS:-6}"
N_PREDICT="${N_PREDICT:-256}"
REPS="${REPS:-3}"
STRESS_WORKERS="${STRESS_WORKERS:-4}"
STRESS_WARMUP="${STRESS_WARMUP:-8}"
RECOVERY_WAIT="${RECOVERY_WAIT:-10}"

if [[ -z "$MODEL" ]]; then
    echo "Usage: $0 MODEL.gguf [prompt.txt]"
    exit 1
fi

for x in curl python3 stress-ng; do
    command -v "$x" >/dev/null || { echo "Missing: $x"; exit 1; }
done

if [[ ! -x "$LLAMA_SERVER" ]]; then
    echo "llama-server not found: $LLAMA_SERVER"
    exit 1
fi

if [[ ! -f "$MODEL" ]]; then
    echo "Model not found: $MODEL"
    exit 1
fi

STAMP="$(date +%Y%m%d-%H%M%S)"
OUT="$PWD/results/flashnext-contention-$STAMP"
mkdir -p "$OUT"

STAGE_FILE="/tmp/flashnext_stage"
SERVER_LOG="$OUT/server.log"
STRESS_LOG="$OUT/stress-ng.log"
CSV="$OUT/results.csv"
AUTO_PROMPT="$OUT/prompt.txt"
PAYLOAD="$OUT/payload.json"
SERVER_PID=""
STRESS_PID=""

cleanup() {
    echo "stopped" > "$STAGE_FILE"
    if [[ -n "${STRESS_PID:-}" ]] && kill -0 "$STRESS_PID" 2>/dev/null; then
        kill -INT "$STRESS_PID" 2>/dev/null || true
        wait "$STRESS_PID" 2>/dev/null || true
    fi
    if [[ -n "${SERVER_PID:-}" ]] && kill -0 "$SERVER_PID" 2>/dev/null; then
        kill "$SERVER_PID" 2>/dev/null || true
        wait "$SERVER_PID" 2>/dev/null || true
    fi
}
trap cleanup EXIT INT TERM

if [[ -z "$PROMPT_FILE" ]]; then
python3 - "$AUTO_PROMPT" <<'PY'
import sys
out = sys.argv[1]
parts = ["You are reviewing a large production Python codebase. Analyze correctness, concurrency, memory usage, error handling, API boundaries, and performance.\n"]
for i in range(35):
    parts.append(f"""
def process_request_{i}(items, cache, executor):
    futures = []
    for item in items:
        if item.key not in cache:
            futures.append(executor.submit(load_record, item.key))
        else:
            update_record(cache[item.key])
    results = [f.result() for f in futures]
    return results

class Service_{i}:
    def __init__(self, repository):
        self.repository = repository
        self.cache = {{}}

    def execute(self, request):
        records = self.repository.fetch(request.account_id)
        return process_request_{i}(records, self.cache, request.executor)
""")
with open(out, "w") as f:
    f.write("\n".join(parts))
PY
    PROMPT_FILE="$AUTO_PROMPT"
fi

python3 - "$PROMPT_FILE" "$PAYLOAD" "$N_PREDICT" <<'PY'
import json
import sys
prompt_file, payload_file, n_predict = sys.argv[1], sys.argv[2], int(sys.argv[3])
with open(prompt_file) as f:
    prompt = f.read()
payload = {
    "prompt": prompt,
    "n_predict": n_predict,
    "ignore_eos": True,
    "cache_prompt": False,
    "seed": 12345,
    "temperature": 0.0,
    "stream": False
}
with open(payload_file, "w") as f:
    json.dump(payload, f)
PY

echo "timestamp,stage,run,prompt_tokens,predicted_tokens,prompt_ms,predicted_ms,prompt_tps,decode_tps" > "$CSV"

"$LLAMA_SERVER" --version 2>&1 | tee "$OUT/llama-version.txt"

CUDA_VISIBLE_DEVICES="$GPU" "$LLAMA_SERVER" \
    -m "$MODEL" \
    -ngl "$NGL" \
    --n-cpu-moe "$CPU_MOE" \
    -c "$CTX" \
    -t "$THREADS" \
    --host 127.0.0.1 \
    --port "$PORT" \
    > "$SERVER_LOG" 2>&1 &
SERVER_PID=$!

READY=0
for _ in $(seq 1 300); do
    if curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
        READY=1
        break
    fi
    if ! kill -0 "$SERVER_PID" 2>/dev/null; then
        tail -100 "$SERVER_LOG" || true
        exit 1
    fi
    sleep 1
done

if [[ "$READY" -ne 1 ]]; then
    tail -100 "$SERVER_LOG" || true
    exit 1
fi

echo "warmup" > "$STAGE_FILE"
curl -sf -H "Content-Type: application/json" --data-binary "@$PAYLOAD" \
    "http://127.0.0.1:$PORT/completion" > "$OUT/warmup.json"

run_stage() {
    local stage="$1"
    echo "$stage" > "$STAGE_FILE"
    for run in $(seq 1 "$REPS"); do
        local response="$OUT/${stage}-${run}.json"
        curl -sf -H "Content-Type: application/json" --data-binary "@$PAYLOAD" \
            "http://127.0.0.1:$PORT/completion" > "$response"
        python3 - "$response" "$CSV" "$stage" "$run" <<'PY'
import csv
import datetime
import json
import sys
response_file, csv_file, stage, run = sys.argv[1:5]
with open(response_file) as f:
    data = json.load(f)
t = data["timings"]
row = [
    datetime.datetime.now().isoformat(), stage, run,
    t.get("prompt_n"), t.get("predicted_n"),
    t.get("prompt_ms"), t.get("predicted_ms"),
    t.get("prompt_per_second"), t.get("predicted_per_second")
]
with open(csv_file, "a", newline="") as f:
    csv.writer(f).writerow(row)
print(f"run={run} prompt={t.get('prompt_n')} pp={t.get('prompt_per_second', 0):.2f} t/s tg={t.get('predicted_per_second', 0):.2f} t/s")
PY
    done
}

run_stage idle

stress-ng --stream "$STRESS_WORKERS" --metrics-brief > "$STRESS_LOG" 2>&1 &
STRESS_PID=$!
sleep "$STRESS_WARMUP"
run_stage contention
kill -INT "$STRESS_PID" 2>/dev/null || true
wait "$STRESS_PID" 2>/dev/null || true
STRESS_PID=""

echo "recovery_wait" > "$STAGE_FILE"
sleep "$RECOVERY_WAIT"
run_stage recovery
echo "done" > "$STAGE_FILE"

python3 - "$CSV" <<'PY'
import csv
import statistics
import sys
rows = list(csv.DictReader(open(sys.argv[1])))
groups = {}
for r in rows:
    groups.setdefault(r["stage"], []).append(float(r["decode_tps"]))
print("===== RESULT =====")
for stage in ["idle", "contention", "recovery"]:
    v = groups.get(stage, [])
    if v:
        print(f"{stage:12s} {statistics.mean(v):8.2f} t/s   {v}")
if groups.get("idle") and groups.get("contention"):
    idle = statistics.mean(groups["idle"])
    cont = statistics.mean(groups["contention"])
    print(f"decode_drop   {(idle-cont)/idle*100:8.1f}%")
if groups.get("idle") and groups.get("recovery"):
    idle = statistics.mean(groups["idle"])
    rec = statistics.mean(groups["recovery"])
    print(f"recovery      {rec/idle*100:8.1f}%")
PY

cat "$STRESS_LOG" || true
trap - EXIT
cleanup

Run it with a relative path or placeholder:

GPU=0 CPU_MOE=48 ./run_flashnext_contention.sh \
  ./models/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

For a sharded GGUF, point llama.cpp at the first shard.

2. CPU-only control

The CPU-only control used the same runner. Replace only the STREAM launch block:

stress-ng --stream "$STRESS_WORKERS" --metrics-brief > "$STRESS_LOG" 2>&1 &

with:

stress-ng \
    --cpu "$STRESS_WORKERS" \
    --cpu-method sqrt \
    --metrics-brief \
    > "$STRESS_LOG" 2>&1 &

Everything else stays identical, including four stress workers.

3. Independent monitoring terminal

The monitoring process was deliberately kept independent from the benchmark runner. Save this as monitor_flashnext.sh and run it in a second terminal before starting the benchmark.

#!/usr/bin/env bash
set -euo pipefail

GPU="${GPU:-0}"
STAGE_FILE="${STAGE_FILE:-/tmp/flashnext_stage}"
INTERVAL="${INTERVAL:-1}"
STAMP="$(date +%Y%m%d-%H%M%S)"
OUT="${1:-$PWD/results/flashnext-monitor-$STAMP}"
mkdir -p "$OUT"
CSV="$OUT/system.csv"

echo "timestamp,stage,cpu_pct,load1,mem_available_mb,swap_used_mb,pgfault_delta,pgmajfault_delta,pswpin_delta,pswpout_delta,gpu_pct,gpu_mem_pct,vram_mb,power_w" > "$CSV"

read_cpu() {
    read -r _ user nice system idle iowait irq softirq steal _ _ < /proc/stat
    local total=$((user + nice + system + idle + iowait + irq + softirq + steal))
    local idleall=$((idle + iowait))
    echo "$total $idleall"
}

read_vm() {
    awk -v key="$1" '$1 == key {print $2}' /proc/vmstat
}

read -r prev_total prev_idle <<< "$(read_cpu)"
prev_pgfault="$(read_vm pgfault)"
prev_pgmajfault="$(read_vm pgmajfault)"
prev_pswpin="$(read_vm pswpin)"
prev_pswpout="$(read_vm pswpout)"
ARMED=0

while true; do
    sleep "$INTERVAL"
    raw_stage="waiting"
    [[ -f "$STAGE_FILE" ]] && raw_stage="$(cat "$STAGE_FILE" 2>/dev/null || echo waiting)"
    [[ "$raw_stage" == "warmup" ]] && ARMED=1
    if [[ "$ARMED" -eq 1 ]]; then stage="$raw_stage"; else stage="waiting"; fi

    read -r total idle <<< "$(read_cpu)"
    delta_total=$((total - prev_total))
    delta_idle=$((idle - prev_idle))
    if (( delta_total > 0 )); then
        cpu_pct="$(awk -v t="$delta_total" -v i="$delta_idle" 'BEGIN {printf "%.1f",100*(t-i)/t}')"
    else
        cpu_pct="0.0"
    fi
    prev_total="$total"
    prev_idle="$idle"

    load1="$(awk '{print $1}' /proc/loadavg)"
    mem_available_kb="$(awk '/MemAvailable:/ {print $2}' /proc/meminfo)"
    swap_total_kb="$(awk '/SwapTotal:/ {print $2}' /proc/meminfo)"
    swap_free_kb="$(awk '/SwapFree:/ {print $2}' /proc/meminfo)"
    mem_available_mb=$((mem_available_kb / 1024))
    swap_used_mb=$(((swap_total_kb - swap_free_kb) / 1024))

    pgfault="$(read_vm pgfault)"
    pgmajfault="$(read_vm pgmajfault)"
    pswpin="$(read_vm pswpin)"
    pswpout="$(read_vm pswpout)"
    pgfault_delta=$((pgfault - prev_pgfault))
    pgmajfault_delta=$((pgmajfault - prev_pgmajfault))
    pswpin_delta=$((pswpin - prev_pswpin))
    pswpout_delta=$((pswpout - prev_pswpout))
    prev_pgfault="$pgfault"
    prev_pgmajfault="$pgmajfault"
    prev_pswpin="$pswpin"
    prev_pswpout="$pswpout"

    gpu_line="$(nvidia-smi -i "$GPU" --query-gpu=utilization.gpu,utilization.memory,memory.used,power.draw --format=csv,noheader,nounits)"
    IFS=',' read -r gpu_pct gpu_mem_pct vram_mb power_w <<< "$gpu_line"
    gpu_pct="$(xargs <<< "$gpu_pct")"
    gpu_mem_pct="$(xargs <<< "$gpu_mem_pct")"
    vram_mb="$(xargs <<< "$vram_mb")"
    power_w="$(xargs <<< "$power_w")"
    ts="$(date --iso-8601=seconds)"

    echo "$ts,$stage,$cpu_pct,$load1,$mem_available_mb,$swap_used_mb,$pgfault_delta,$pgmajfault_delta,$pswpin_delta,$pswpout_delta,$gpu_pct,$gpu_mem_pct,$vram_mb,$power_w" >> "$CSV"

    printf "\r%-14s CPU %5s%% load %5s RAM %6sMB majflt %5s swapin %5s GPU %3s%% VRAM %5sMB %6sW" \
        "$stage" "$cpu_pct" "$load1" "$mem_available_mb" "$pgmajfault_delta" "$pswpin_delta" "$gpu_pct" "$vram_mb" "$power_w"

    if [[ "$ARMED" -eq 1 && ( "$raw_stage" == "done" || "$raw_stage" == "stopped" ) ]]; then
        break
    fi
done

Terminal 2:

GPU=0 ./monitor_flashnext.sh

Terminal 1:

GPU=0 CPU_MOE=48 STRESS_WORKERS=4 \
  ./run_flashnext_contention.sh \
  ./models/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf

Then repeat Terminal 1 with the CPU-only stress block.

What this result does and does not show

This experiment supports a narrow claim:

On this RTX 4090 + Ryzen 5 7600 + 128 GB system, with 48 MoE layers assigned to CPU memory, concurrent STREAM-style memory traffic reduced Qwen3.8 Flash-Next decode throughput by roughly two thirds, while a CPU-only workload at nearly the same aggregate CPU utilization reduced decode by only 5.4%.

It does not establish that Qwen3.8 Flash-Next is universally memory-bound, that every MoE configuration behaves this way, or that GPU performance is irrelevant.

Important limitations:

• one machine;
• one RTX 4090;
• one quantization;
• one synthetic 4899-token prompt;
• one --n-cpu-moe 48 placement strategy;
• no hardware DRAM-controller counters;
• no CPU affinity or NUMA pinning;
• CPU frequency governors were reported as powersave by stress-ng;
• no real agentic coding trace in this first experiment.

The next useful experiment is not another generic TPS run. It is to vary memory traffic deliberately and map decode throughput against measured memory-controller bandwidth, then repeat with real coding contexts and alternative MoE placement.

Bottom line

The interesting result is not simply that Qwen3.8 Flash-Next can be slowed down by another program.

Almost anything can.

The interesting result is the control:

4 CPU workers:    21.06 -> 19.93 tok/s   (-5.4%)
4 STREAM workers: 21.00 ->  6.93 tok/s  (-67.0%)

Both conditions pushed total CPU utilization to roughly 56–58%.

Only the memory-heavy workload starved the GPU and collapsed throughput.

For local MoE inference, isolated benchmark numbers can therefore hide a systems-level bottleneck that appears only when the machine is doing something else at the same time.

References

• Original LocalLLaMA experience report
• llama.cpp
• llama.cpp commit used in this experiment
• stress-ng