# Optimizing Model Loading with a CPU Bottleneck

**URL:** <https://discuss.huggingface.co/t/optimizing-model-loading-with-a-cpu-bottleneck/166495>\
**Category:** 🤗Transformers\
**Created:** [August 12, 2025, 7:51am UTC](https://discuss.huggingface.co/t/optimizing-model-loading-with-a-cpu-bottleneck/166495 "2025-08-12T07:51:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhishek-P](https://avatars.discourse-cdn.com/v4/letter/a/51bf81/32.png) [@Abhishek-P](https://discuss.huggingface.co/u/Abhishek-P)\
**Post date:** [August 12, 2025, 7:51am UTC](https://discuss.huggingface.co/t/optimizing-model-loading-with-a-cpu-bottleneck/166495/1 "2025-08-12T07:51:05Z")

</div>

I am trying to load the pretrained [mt5-xl](https://huggingface.co/google/mt5-xl) in a GCP VM with 4 vCPUs, 15 GB Memory, and [NVIDIA Tesla L4 GPU](https://www.nvidia.com/en-us/data-center/l4/) with 24 GB GPU Memory

Codewise simply

`model = AutoModel.from_pretrained(“google/mt5-xl”)`

My model loading is failing with `RuntimeError: unable to mmap 14970735570 bytes from file` both in CPU only VMs and with GPU VMs, indicating this is a model loading error rather than a GPU OOM memory.

The model loading seems to be hitting the CPU bottleneck before model is actually put on the GPU (GPU memory usage monitored with `nvidia-smi`

I have come across Accelerate [Utilities](https://huggingface.co/docs/accelerate/v1.10.0/en/package_reference/utilities#accelerate.infer_auto_device_map)and [Big Modeling](https://huggingface.co/docs/accelerate/usage_guides/big_modeling)so I am working through debugging the `device_map` and `max_memory` to make this work.

But if there are solutions, or workarounds available, it would be much appreciated

`low_cpu_mem_usage` in `from_pretrained` did not make a difference.

With `offload_state_dict=True` about 25000 MB got loaded to GPU before getting the same error.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 12, 2025, 10:24am UTC](https://discuss.huggingface.co/t/optimizing-model-loading-with-a-cpu-bottleneck/166495/2 "2025-08-12T10:24:37Z")

</div>

> Accelerate Utilities

I think that’s the correct workaround. If you want to treat VRAM, RAM, and disk as a single entity, you should use the Accelerate library.😅

Environments where RAM is less than VRAM are not commonly expected, so this issue is not often reported, but excessive RAM consumption during model loading can occasionally become an issue.

- [infer\_auto\_device\_map inefficiently allocates GPU memory for models with imbalanced module sizes](https://github.com/huggingface/accelerate/issues/3041)
- [device\_map=‘auto’ causes memory to not be freed with torch.cuda.empty\_cache()](https://github.com/huggingface/accelerate/issues/973)
- [How to avoid the peak RAM memory usage of a model when I want to load to GPU](https://github.com/huggingface/transformers/issues/28476)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 12, 2025, 10:34am UTC](https://discuss.huggingface.co/t/optimizing-model-loading-with-a-cpu-bottleneck/166495/3 "2025-08-12T10:34:17Z")

</div>

> google/mt5-xl

Sorry. That’s the general rule, but in this case, it seems to be a problem with the model. _A 15GB file is stored without being split_…😅

 ![mt5xl15gb](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/b/3bd932657dda49745158f8bf13f7e30c9405135e.png)

Recently, files are often saved in split form, which is more convenient when loading large models. The quickest solution would be to save it again yourself. You can either upload it or store it somewhere _in advance_ and load it onto the GPU.

```py
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-xl")
model.save_pretrained("mt5-xl-sft-2gb", safe_serialization=True, max_shard_size="2GB") # or "5GB", "10GB", etc.
#model.push_to_hub("mt5-xl-sft-2gb", safe_serialization=True, max_shard_size="2GB") # if uploading to Hugging Face Hub directly

```

---

<div class="post-metadata">

**Author:** ![Abhishek-P](https://avatars.discourse-cdn.com/v4/letter/a/51bf81/32.png) [@Abhishek-P](https://discuss.huggingface.co/u/Abhishek-P)\
**Post date:** [August 17, 2025, 2:24am UTC](https://discuss.huggingface.co/t/optimizing-model-loading-with-a-cpu-bottleneck/166495/4 "2025-08-17T02:24:39Z")

</div>

Thanks. Realized that was the problem.  
At that point I didn’t have a infra that could take care of sharding it either.

I now was able to get access to some L40 which circumvents this problem, and should help with solving this problem.

I will probably work through sharding this as you suggested and probably keep it as my copy version of the model.

I was hoping I could check with maintainers to actually add the safetensors (which is still an open PR on it) and then the sharded version too.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 17, 2025, 3:23am UTC](https://discuss.huggingface.co/t/optimizing-model-loading-with-a-cpu-bottleneck/166495/5 "2025-08-17T03:23:05Z")

</div>

> I was hoping I could check with maintainers to actually add the safetensors (which is still an open PR on it) and then the sharded version too.

It seems that [a PR has already been opened regarding that](https://huggingface.co/google/mt5-xl/discussions/2). All that’s needed is for the maintainer to merge it, but I guess it’s been forgotten…😅