AREX-2 Local Readiness: Qwen3.8 Base, Runtime and Quantization
A local-deployment audit of BAAI AREX-2: its Qwen3.8 base, Qwen3.5-compatible model class, memory budget, multimodal conversion and reference-runtime checks.
Approximately 5 min read
BAAI’s AREX-2 is interesting for local inference because it is a 27B Qwen3.8-based multimodal checkpoint with a large context window, MTP metadata, and an artifact size that immediately forces a deployment decision.
The useful question is not whether AREX-2 is new. It is what has to work before this model becomes a credible local quantization target.
This analysis compares published model metadata with documented upstream llama.cpp support. The result is a readiness map rather than an inference benchmark.
What the published artifact says
The current model card identifies a 27B dense Qwen3.8-compatible multimodal model with a 262,144-token context. The original artifact snapshot inspected for this audit contained about 54.7 GB of model files.
The configuration recorded in that snapshot declared:
architecture: Qwen3_5ForConditionalGeneration
dtype: bfloat16
hidden_size: 5120
num_hidden_layers: 64
max_position_embeddings: 262144
mtp_num_hidden_layers: 1
language_model_only: false
The vision section was also present. The Qwen3_5ForConditionalGeneration class name identifies the compatibility path; it should not be used to overwrite the model card’s Qwen3.8 base-model attribution.
So this is not a plain text-only Qwen checkpoint with a familiar conversion path. It combines Qwen3.5 text architecture, multimodal processing, hybrid linear/full attention, a 262K configured context ceiling, MTP metadata, and a large BF16 source artifact.
Each item adds a separate compatibility surface.
The raw BF16 artifact is not a 4090 model
A 54.7 GB repository is already larger than the 24 GB VRAM on an RTX 4090.
Even a hypothetical 36 GB aggregate VRAM budget would be below the raw source artifact size before runtime overhead; multi-GPU placement introduces additional constraints.
So local deployment is mainly a question of whether a suitable quantized artifact can be produced and whether the selected runtime understands every required model component.
A rough 4-bit weight estimate for 27B parameters is:
27 billion parameters
x 4 bits
/ 8
≈ 13.5 GB
That is only a weight-level estimate. It does not include quantization metadata, vision components, runtime workspaces, KV cache, MTP state, allocator fragmentation, or context-dependent buffers.
So “about 13.5 GB at 4-bit” is not the same as “guaranteed to fit in 13.5 GB VRAM.”
Current upstream llama.cpp does recognize Qwen3.5
This is where checking current source matters.
The current upstream llama.cpp Qwen converter registers:
Qwen3_5ForConditionalGeneration
Qwen3_5ForCausalLM
and maps the architecture to the Qwen3.5 GGUF path.
The current llama.cpp README also uses a Qwen3.5 GGUF in its quick-start example.
That means a blanket statement such as “llama.cpp does not support Qwen3.5” would be wrong today.
Why runtime version checks matter
A locally installed runtime may lag behind upstream architecture support. An absent conversion mapping in an older checkout does not establish that the current project lacks support.
Before assessing AREX-2 conversion or loading, record the runtime revision, compare it with current upstream registrations, and rerun the same smoke tests on an updated build.
The multimodal path needs separate validation
AREX-2 declares language_model_only as false and includes a vision configuration.
A text-only smoke test would therefore prove only part of the deployment path.
A useful validation plan should separate:
text generation
-> chat template
-> structured tool use
-> image input
-> long context
-> MTP/speculative path
Record the result of each feature test separately.
MTP metadata is another place to be careful
The config declares one MTP hidden layer.
Metadata alone is not enough.
A checkpoint can advertise an MTP side model in its configuration while the corresponding MTP tensors are absent from the source artifact. The tensor index is the evidence to inspect.
That possibility does not establish a defect in AREX-2.
It does mean the correct workflow is:
read config
-> inspect tensor index
-> verify MTP/NextN tensors actually exist
-> convert
-> load without MTP
-> load with MTP
-> compare behavior and memory
Do not infer tensor presence from one configuration field.
Establish the reference with the author’s inference example
The author-written inference section now uses Transformers with AutoModelForMultimodalLM and AutoProcessor, and calls for Qwen3.8 support. Use that documented path as the first reference. Hugging Face’s automatically generated “Use this model” recipes are separate from the author’s tested instructions.
If a local GGUF or EXL3 conversion fails, a successful supported-runtime load helps separate a source-checkpoint problem from a quantizer/runtime compatibility problem.
Without a control path, a failed conversion can be misdiagnosed as a bad model release.
What I would test next
The next useful experiment is a staged compatibility gate.
Gate 1: artifact integrity
Verify config, processor metadata, tokenizer files, safetensors index, vision files, and actual MTP/NextN tensor presence.
Gate 2: reference runtime
Load the original checkpoint through a currently documented runtime and record the exact runtime version, GPU placement, peak memory, text smoke, tool-call smoke, and vision smoke.
Gate 3: conversion
Try one quantization path at a time.
For GGUF:
current llama.cpp converter
-> F16/BF16 GGUF if practical
-> one moderate quant
-> load test
For EXL3:
current ExLlamaV3
-> architecture recognition
-> calibration/conversion
-> load
-> text/tool/vision validation
A conversion that finishes but cannot preserve required model features is not a complete success.
Gate 4: context scaling
Only after load correctness is established should context length be increased.
The configured 262K ceiling does not mean every consumer-GPU configuration can allocate or sustain 262K.
Measure at practical steps such as 4K, 16K, 32K, and 64K, then go higher only if memory and correctness remain sane.
Why this model is worth watching
AREX-2 is large enough to expose real runtime boundaries but still small enough that a 4-bit-class artifact could plausibly fit on enthusiast hardware.
It also combines several features that tend to fail independently:
new architecture
+ multimodal path
+ very long context
+ MTP
+ quantization
That makes it a better compatibility target than a simple release post.
The publishable result today is therefore not a speed claim.
Current upstream llama.cpp understands the Qwen3.5 architecture, the published AREX-2 artifact is far too large for raw BF16 consumer-GPU loading, and a credible local result requires separate validation of quantization, vision, MTP, and context behavior.
A follow-up deployment report can then record which of those features work at each tested context size.