Local AI

llama.cpp Gemma 4 Vision: One Padding Flag, a Large Embedding Gap

An October 31 llama.cpp patch removed letterbox padding from Gemma 4 vision preprocessing. Audit the source, the reported embedding-cosine results, and a controlled parity test.

Approximately 8 min read

A one-line change in a vision model’s image preprocessor can matter more than a large kernel optimization. On October 31, 2026, llama.cpp merged PR #30922, changing Gemma 4’s multimodal projector configuration to use PAD_NONE instead of the default PAD_CEIL. The author reported that, under a tightly controlled image-token budget, the mean per-token cosine similarity between llama.cpp image embeddings and the Transformers reference rose from 0.60–0.95 to 0.998–1.000 for nine non-square test images. Three square or already-aligned controls were unchanged.

Those numbers are upstream-reported measurements, not an independent RAMGPT reproduction. The merged commit 7bd1808 modifies one line in the Gemma 4 projector loader. The unusually large reported effect deserves a source-level explanation: a preprocessing mismatch can corrupt the comparison before the GPU backend, quantization scheme or language model receives an image embedding.

The actual change, not the headline

The patch is in tools/mtmd/clip.cpp, inside the Gemma 4 projector configuration. The existing path already selected bicubic image resizing and set the merge/pooling configuration. The new assignment is:

hparams.image_resize_pad = PAD_NONE;

It is important to identify the scope correctly. This is not a new model, a change to Gemma 4’s trained weights, a universal image-resolution increase or an inference speedup. It changes the way the multimodal image path handles padding during resize. The patch does not itself alter the language model’s decoding logic.

According to the PR description, the old default PAD_CEIL could add black letterbox bars when a dynamically chosen target size had a different aspect ratio from the source image. The Transformers Gemma 4 processor instead resizes the image to its selected target dimensions without adding those letterbox bars at that step. That mismatch changes the pixels fed into the vision encoder.

The difference is especially visible for non-square inputs. If the source is wide and the selected target dimensions do not preserve the same aspect ratio exactly, a pad-to-fit implementation and a direct-resize implementation can place image content at different coordinates. Even if both produce the same number of image tokens, the corresponding tokens no longer represent the same visual locations.

This is why a text-only smoke test can pass while vision parity fails.

Read the reference processor before blaming the model

The version-pinned Transformers v5.17.0 Gemma 4 processor is a better reference than a generic claim that images are resized. Its aspect-ratio-aware size-selection routine uses the source height and width, patch size, maximum patches and pooling kernel size to calculate a target height and width. The processor then resizes directly to that target size. Later patchification and padding for batching are separate operations.

That distinction prevents an easy misunderstanding: PAD_NONE does not mean that Gemma 4 never pads any image tensor. The patch concerns letterboxing during the resize stage, not every possible form of padding in the vision pipeline. Patch-grid alignment, position IDs and batch shapes can still involve padding or other transformations.

The relevant comparison is therefore not “padding versus no padding” in the abstract. It is whether the exact sequence of resize, patchify and projector operations matches the reference implementation for a fixed image and token budget.

A reproducibility check should hold all of these constant: original image bytes, orientation, color conversion, target image-token budget, resize interpolation, projector precision, model revision, output token ordering and the point at which embeddings are captured. Otherwise a cosine comparison can mix several independent sources of drift.

What the upstream experiment actually measured

The PR author describes a comparison using google/gemma-4-E2B-it with a float32 multimodal projector, llama.cpp on CPU and Transformers 5.17.0 on CPU in float32. Both sides were configured for a 280-token image budget. On the llama.cpp side, the author used the MTMD embedding debug output from llama-mtmd-cli, with the minimum and maximum image tokens both fixed at 280. On the reference side, the author called Gemma4ImageProcessor and model.get_image_features(…).

The reported statistic was per-image mean token cosine similarity, calculated from corresponding image embeddings. For the nine non-square images where the selected target size matched across implementations, the old path produced values from 0.60 to 0.95; the patched path produced 0.998 to 1.000. Three control images that were square or already aligned did not change.

There are four useful aspects to that design.

First, the author compared intermediate image embeddings, not final free-form text answers. That localizes the discrepancy closer to the image encoder and reduces sampling noise from language generation.

Second, both sides used float32 in the reported comparison. This reduces quantization as a competing explanation. It does not guarantee bitwise-identical arithmetic across two software stacks.

Third, the fixed 280-token budget avoids silently comparing different image token counts. It is not proof that all supported image budgets or resolutions behave identically.

Fourth, unchanged controls are a meaningful negative control. If the fix were merely changing unrelated model weights or globally scaling embeddings, the unchanged square-image cases would be harder to explain.

The important limitation is sample size and scope: nine affected images and three controls, one model variant and one budget. No per-image source dataset, broad task-accuracy benchmark, GPU-backend parity matrix or confidence interval is established by the PR description. We should not convert a high embedding cosine into a claim of improved OCR accuracy, visual reasoning accuracy or universal cross-backend correctness.

Why an embedding cosine can mislead

For two nonzero vectors a and b, cosine similarity is their dot product divided by the product of their lengths. A value close to 1 indicates near alignment in direction. Here the author averaged corresponding token-level cosine values within each image.

That statistic is valuable for a localized preprocessing investigation, but it has limitations:

A stronger follow-up would report the full distribution: minimum token cosine, 5th percentile, median, mean, maximum absolute difference and per-position heatmaps. The headline mean should be accompanied by an explicit alignment check.

For downstream quality, pair the embedding comparison with a small, fixed set of image questions: object counting, text reading, spatial relations and visually similar distractors. Keep the prompt and generation settings identical. Treat changes in answers as observations, not proof that this single resize flag caused every change.

A controlled parity test readers can run

Use a known pre-fix revision, the exact merged fix and the version-pinned Transformers reference. Build each llama.cpp revision separately; do not overwrite one executable and later assume its label identifies the code. Record the complete commit hash, build flags, compiler, model and mmproj artifact hashes.

The upstream PR supplies a useful command pattern for the llama.cpp side. The following is a template, not a command executed for this article:

MTMD_DEBUG_EMBEDDINGS=before.bin ./llama-mtmd-cli \
  -m /path/to/gemma-4-E2B-it.gguf \
  --mmproj /path/to/mmproj-f32.gguf \
  --image /path/to/wide-test.png \
  --image-min-tokens 280 --image-max-tokens 280 \
  -p "Describe the image." -n 1

Run it with the pre-fix binary and then with the patched binary, writing to different output files. Generate the reference embedding using Transformers v5.17.0 and the same image budget. The debug format and extraction code should be validated against the exact revision before attempting to parse the binary output. The example does not establish that a particular model artifact or flag combination is available on every platform.

Choose a small test matrix: portrait, landscape, square, odd dimensions near patch boundaries, high-contrast text, narrow diagrams and images whose selected dimensions change with the token budget. Record the actual target dimensions on both sides. If those dimensions differ, do not calculate a headline token-aligned cosine and call it a valid parity test.

An ablation is particularly informative: hold the patched binary fixed and compare the resize path with and without letterboxing in an instrumented local build. If the black bars explain the discrepancy, the affected image embeddings should track the resize policy while the already-aligned controls remain stable. This is a proposed experiment, not one RAMGPT has run.

Release and operational implications

The fix landed in the October 31 b12021 pre-release. Because llama.cpp publishes frequent builds, pin the exact commit or release artifact when reporting a result. “Latest llama.cpp” is not a reproducible version identifier.

Teams running Gemma 4 vision behind a local inference service should distinguish three checks:

  1. Artifact provenance: confirm that the executable actually includes commit 7bd1808 or a descendant and that the mmproj corresponds to the intended model.
  2. Preprocessing parity: compare selected target sizes and image embeddings for representative image shapes, including non-square cases.
  3. Application behavior: rerun the specific image tasks the application depends on; do not infer correctness from a single embedding similarity statistic.

If an older build produces worse visual answers, this PR offers a plausible, source-backed hypothesis to test before changing GPU drivers or re-quantizing a model. It does not prove that every vision failure on an older build has this cause.

The broader lesson: preprocessing is part of the model

A multimodal model is not just a weight file and a decoder. Its effective computation includes the image processor, spatial token layout, projector, prompt serialization and inference runtime. Two implementations can load nominally equivalent weights yet feed the language model different visual representations.

The October 31 Gemma 4 patch is a concise demonstration. The merged code diff is one line; the reported embedding change is large; the controls point to a concrete preprocessing mechanism. The next useful research result is an independent, artifact-pinned parity reproduction across image shapes and token budgets, followed by downstream task tests.

Until that reproduction exists, the evidence supports a narrower but still important conclusion: llama.cpp corrected a source-visible Gemma 4 resize mismatch, and the upstream author reports a substantial embedding-parity improvement under specified conditions. That is actionable engineering evidence without inventing a universal benchmark win.

Sources and further reading