Hugging Face Shared Xet Blobs: Cache Savings and Safe Deletion
How cross-repository Xet file deduplication works, why cache sizes can mislead, and how to delete models without breaking shared references.
Approximately 8 min read
A machine that experiments with several model repositories can download the same large tensor file more than once even when the bytes are identical. The interesting part of Hugging Face’s recent cache changes is not merely that duplicate bytes can be avoided. It is how the cache preserves existing repository paths while sharing physical storage across repositories—and how deletion must change when a file has multiple owners.
Hugging Face’s October 26, 2026 CLI and library recap highlights a shared Xet-file store. The underlying functionality was already included in the huggingface_hub v1.32.0 release, so this is an analysis of a newly highlighted capability, not a claim that the feature first shipped on October 26.
The vendor’s example is unusually concrete: three FLUX.1 repositories include an identical roughly 9.5 GB text encoder. The published comparison is approximately 28.6 GB if that file is stored three times versus 9.5 GB when the physical payload is shared. That is roughly 19.1 GB of avoided duplication in the example, not a universal saving for all checkpoints.
Three kinds of reuse that should not be conflated
Repository revision reuse. Traditional Hub caching keeps snapshots of different commits in a repository and points them at common local blobs when their content matches. This avoids a second copy when one repository changes its README but not its weights.
Cross-repository file reuse. The newer shared blob store recognizes identical Xet-backed files in different repositories. The repositories keep their own snapshot and blob references, but the physical payload is stored once in a cache-root store. This is the change relevant to users who download multiple checkpoints that share a tokenizer, encoder, or weight shard.
Xet chunk reuse. Xet also has chunk-oriented transfer and caching machinery. Reusing a byte range during transfer is not the same as the cross-repository, whole-file shared-blob layout discussed here. A successful chunk cache hit does not by itself prove that two final local files occupy one physical payload.
The practical unit for the new shared store is an identical Xet file. A model with slightly different serialization, dtype, shard boundaries, or quantization may not qualify even when its tensors are conceptually related. The Diffusers documentation explicitly warns that changes to those properties can defeat reuse.
The path layout explains the behavior
According to the Hub local-cache specification, an Xet-backed payload is placed under a cache-root path shaped like this:
<CACHE_DIR>/
blobs/
.huggingface-shared-blobs
ab/
<xet-hash>
<xet-hash>.refs
<xet-hash>.lock
models--organization--repository/
blobs/
<etag> -> ../../blobs/ab/<xet-hash>
snapshots/
<commit>/
model-file -> ../../blobs/<etag>
The hexadecimal prefix is illustrative, not a real file identifier. The repo-local blob path still uses the familiar ETag name; its target becomes a relative symbolic link into the shared store. Snapshot consumers continue to follow the same paths. That indirection is the compatibility mechanism: readers need not understand the new physical layout to open the model file.
The marker file identifies a shared store owned by the Hub cache implementation. The software does not assume that any arbitrary top-level directory named blobs is safe to manage. Each shared payload also has a .refs manifest of repo-local references and a lock used during cleanup. These details matter more than the headline disk-space number.
Why cache deletion is now a reference-management problem
Imagine two model repositories, A and B, whose cached snapshots ultimately point to the same Xet payload P.
A/snapshots/... -> A/blobs/etag-A -> shared/P
B/snapshots/... -> B/blobs/etag-B -> shared/P
Deleting A should remove A’s references, not P, because B still needs it. Deleting B later may make P reclaimable. But the implementation cannot blindly trust an old manifest: a process may have changed the filesystem, a download may have been interrupted, or metadata may be incomplete.
The cache implementation guide describes a conservative policy. The manifest narrows which references to inspect, but each entry is checked against the filesystem. If necessary metadata is missing or unreadable, cleanup keeps the shared file rather than risking deletion of a valid payload. This is a deliberate preference for some reclaimable disk usage over a broken cached model.
Two commands serve different purposes:
hf cache rm hf://models/unsloth/gemma-3-27b-it-GGUF/gemma-3-27b-it-Q4_K_M.gguf --dry-run
hf cache prune --dry-run
The first previews removal of one cached file reference from the named repository; the second previews garbage collection, including shared files no cached repository still references, detached revisions and incomplete downloads. Remove --dry-run only after reviewing the proposed changes. The file URI is the official CLI’s example, not a claim that the named model was downloaded or tested for this article.
The CLI guide also documents hf cache ls, hf cache ls --revisions and hf cache verify. A useful first diagnostic is to inspect cached revisions before deleting anything. A deletion tool from an older or different client is not necessarily aware of the shared store.
Logical size versus physical size
Once two repositories reference one physical payload, the word size becomes ambiguous.
A per-repository listing can legitimately count the full logical size of the model’s referenced files. If A and B each point at a 9.5 GB shared encoder, both repositories may report roughly 9.5 GB attributable to that file. Adding those per-repository values double-counts the physical bytes.
The official cache guide distinguishes those logical repository sizes from cache-wide size_on_disk, which counts shared payloads once. It also notes that the physical total can include unreferenced shared files pending cleanup. A snapshot’s apparent size, the sum of repository sizes, the cache-wide physical size and the number of bytes fetched over the network answer four different questions.
An important boundary: disk deduplication does not imply that two simultaneously loaded models share GPU VRAM. Model-loading frameworks may allocate separate tensors in device memory even when their source files resolve to the same on-disk bytes. Nor does one shared file imply that all different quantizations share a common payload.
When sharing silently does not happen
The feature requires the Xet download path and a filesystem that supports the symbolic-link layout. The official environment-variable reference identifies HF_HUB_DISABLE_SHARED_BLOBS=1 as the explicit opt-out and HF_HUB_DISABLE_XET=1 as a condition that disables this shared-store path.
The cache documentation lists other reasons for falling back to repository-local storage: unsupported symlinks, permissions, or a pre-existing unmarked cache-root blobs directory. These failures can be silent by design. A download succeeding therefore does not prove that shared-file deduplication occurred.
For Windows users, symlink support may depend on Developer Mode or administrator permissions. On a network filesystem, permission and symlink behavior deserve a separate check. In a container, verify that two jobs actually mount the same cache directory; identical HF_HOME strings inside separate isolated filesystems do not create a shared physical cache.
A small, falsifiable verification plan
The following is a proposed test, not a test reported as executed here. It can be performed with two publicly accessible repositories known to contain an identical Xet-backed file. Use a disposable cache directory with sufficient free space and pin both revisions.
- Record the
huggingface_hubandhf_xetversions, filesystem type,HF_HOME/HF_HUB_CACHEvalues and symlink capability. - Download the first repository’s target file. Record its resolved real path, logical size, physical cache size and network bytes.
- Download the identical file from the second repository using the same cache root. Check whether both repo-local blob links resolve to one shared Xet-hash payload. Measure incremental physical bytes and network traffic.
- Run
hf cache rm ... --dry-runfor the first reference. Delete it only in the disposable test cache. Confirm that the second reference still opens and that the shared payload remains. - Remove the second reference and run
hf cache prune --dry-run. Check whether the payload is now listed as reclaimable. Repeat withHF_HUB_DISABLE_SHARED_BLOBS=1as a control.
A decisive positive result is not merely that both model paths exist. It is that both resolve to the same shared physical payload, the second fetch transfers no redundant file bytes, and deleting one reference leaves the other usable. If the control does not produce different behavior, inspect whether the test files were actually Xet-backed and whether the client and filesystem meet the prerequisites.
For a more robust experiment, record cold and warm runs separately. A warm HTTP or Xet chunk cache can reduce network transfer without demonstrating shared final-file storage. A valid benchmark needs to isolate those effects.
Compatibility and supply-chain boundaries
The cache guide says older readers—including programs that simply follow symlinks—can still open files from the expected repository layout. The danger is with writers and deletion tools. Some older cache cleaners may remove a shared payload that another repository still references; the affected file then has to be downloaded again. Other scanners may treat the new cache-root blobs directory as unfamiliar. Prefer a current Hub client for cleanup operations and test any third-party cache cleaner against a disposable shared store.
Content-addressed storage is not an authorization policy. Identical Xet hashes help identify identical stored files, but they do not prove that a model is trustworthy, that its repository is correctly licensed, or that executing code from a model repository is safe. Continue to pin revisions, verify the intended publisher and artifact, and avoid enabling remote code execution solely because the bytes were found in a cache.
The operational takeaway
This is a meaningful improvement for machines that maintain overlapping model families: the new store can save disk and transfer bandwidth without changing the paths most loaders consume. Its more interesting engineering property is the conservative deletion protocol: shared ownership must be tracked, and uncertainty must not cause data loss.
For practitioners, the immediate checklist is simple: use a sufficiently recent Hub client, keep a consistent cache root, verify symlink/Xet prerequisites, compare physical rather than summed logical sizes, and preview deletions. Treat Hugging Face’s 9.5 GB example as a documented illustration, not a benchmark of your own machine.
Primary references
- October 26 Hub CLI/library feature recap
- Hub local-cache layout and shared-blob rules
- Versioned Python cache-management documentation
- CLI cache-management commands
- Environment-variable behavior
- Hugging Face Hub release notes