llama.cpp Host-RAM MoE Pipeline Parallelism: What PR #29963 Changes

A draft llama.cpp change is trying to solve an awkward multi-GPU MoE case: the model is large enough that some expert weights stay in system memory, but the rest of the graph is spread across several accelerators.

The short version is:

PR #29963 tries to keep pipeline parallelism useful even when MoE expert weights must be streamed from host RAM to the GPU responsible for each layer.

That is not the same problem as ordinary tensor splitting. It combines three constraints at once:

1. the model spans multiple devices;
2. some MoE weights cannot stay resident on those devices;
3. the scheduler still wants several pieces of work in flight without forcing global synchronization.

The PR is a draft. Treat the design as work under review, not a released llama.cpp capability.

Why host-resident experts complicate pipeline parallelism

Mixture-of-Experts models route each token to only a subset of experts.

If all expert weights are already on the GPU, the runtime can execute the selected experts without first moving their matrices across the host-device boundary.

If some expert weights remain in host RAM, the runtime has to answer another question:

Which device needs this expert, and when should the weight transfer happen?

A naive implementation can lose the benefit of pipeline parallelism by inserting synchronization points between devices.

The draft changes try to keep transfers ordered on each backend stream so that one device can continue useful work while another device is handling a different layer or upload.

The fit logic changes first

The PR adds a special placement path when several conditions are true, including:

• more than one device;
• an MoE model;
• layer split mode;
• operation offload enabled;
• K/Q/V offload enabled;
• backends that support the required asynchronous execution and events.

Instead of letting one device absorb most of the host-overflowing expert layers, the fit logic initially gives each device a roughly equal slice.

It then converts as many dense-only layers into full layers as memory permits.

That matters because pipeline throughput is limited by the slowest stage. If one GPU owns most of the host-resident expert traffic, the other devices can spend time waiting for that stage.

The code explicitly checks whether host weights are sufficiently spread across devices before enabling this path.

Host weights become transient scheduler inputs

One important scheduler change is that host-resident weights are treated differently from normal graph inputs.

The draft introduces a helper that identifies a tensor as a host weight when:

• it has a backing buffer;
• the buffer is marked for weight usage;
• the buffer lives in host memory.

For pipeline-parallel execution, those weights can be copied to the backend that consumes them without keeping a separate persistent copy for every in-flight graph.

That allows the allocator to reuse device memory after the split has consumed the weight.

Conceptually:

host expert weight
        |
        v
GPU for layer N
        |
   execute split
        |
temporary device storage can be reused

This is a different memory trade-off from keeping all experts permanently resident.

The figure shows the proposed scheduler path; this remains unmerged and has no claimed production speedup.

User inputs are uploaded earlier

The patch also adds per-backend input events.

With several graph copies in flight, user inputs can be uploaded before the corresponding split begins. The scheduler records an event for the upload and can wait on the appropriate copy only when needed.

The goal is to avoid a later split stalling because an input transfer was delayed until the last moment.

This is one of the central requirements of useful pipeline parallelism: overlap transfers and compute rather than serializing them accidentally.

Large batches may upload all experts

For selected-expert MoE operations, the scheduler normally needs routing IDs to know which expert slices to copy.

Reading those IDs can itself create a synchronization point.

The draft therefore adds a threshold where a sufficiently large batch uploads all experts for the relevant operation instead of waiting to inspect the exact routing set.

That sounds wasteful, but it can make sense when most experts are likely to be used anyway and the alternative is to stall an in-flight pipeline.

The code uses different thresholds depending on whether a previous graph is already in flight.

This is a latency-throughput trade-off, not a universal rule.

Graph reservation also changes

Pipeline parallelism is sensitive to graph shape changes because a new topology can trigger compute-buffer reallocation and synchronize devices.

The PR adds logic to reserve a worst-case graph with a stable output shape and to repeat that reservation when a different graph would otherwise shrink the buffers.

This overlaps with a broader llama.cpp theme visible in recent GLM/Qwen work: graph correctness is not just about tensor math. Memory reservation and topology stability can determine whether a multi-device path performs well.

What this could enable

If the approach survives review, the interesting target is not “run any giant MoE from RAM for free.”

The more realistic target is:

use several GPUs efficiently even when VRAM is insufficient to keep every expert resident.

That could make heterogeneous local systems more useful for MoE models whose dense/shared components fit comfortably but whose full expert set does not.

The cost is additional host-to-device traffic. PCIe bandwidth, pinned-memory behavior, batch size, expert routing distribution and device balance can all matter.

What the PR does not prove yet

There are no RAMGPT measurements attached to this article.

PR #29963 is still a draft, so several questions remain open:

• how much throughput is recovered versus non-pipelined host offload;
• how the path behaves on mixed GPUs;
• whether small-batch decode gains anything;
• how sensitive performance is to PCIe topology;
• whether uploading all experts for large batches is consistently worthwhile;
• whether new graph-reservation behavior creates regressions elsewhere.

Those are benchmarking questions, not conclusions that can be read directly from a diff.

Practical takeaway

For llama.cpp users, the important signal is that multi-GPU MoE offload is moving beyond a simple “which tensors fit where?” problem.

The scheduler increasingly has to coordinate:

• layer placement;
• host-resident weight streaming;
• asynchronous copies;
• graph copies in flight;
• expert routing;
• compute-buffer reservation.

PR #29963 is an explicit attempt to make those pieces work together.

Until it merges, keep it in the “promising implementation under review” category rather than treating it as production behavior.