# General question about large model loading

**URL:** <https://discuss.huggingface.co/t/general-question-about-large-model-loading/43073>\
**Category:** 🤗Accelerate\
**Created:** [June 13, 2023, 12:24pm UTC](https://discuss.huggingface.co/t/general-question-about-large-model-loading/43073 "2023-06-13T12:24:16Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ykacer](https://avatars.discourse-cdn.com/v4/letter/y/eb8c5e/32.png) [@ykacer](https://discuss.huggingface.co/u/ykacer)\
**Post date:** [June 13, 2023, 12:24pm UTC](https://discuss.huggingface.co/t/general-question-about-large-model-loading/43073/1 "2023-06-13T12:24:16Z")

</div>

Hi,

My question is about model loading : how is it possible that one model can be loaded on gpu while another one of same number of params (even a very little bit lighter on disk) cannot be loaded?  
For a concrete case, i’m trying to load a “Text Generation” model on Free Google Colab (12.7Go RAM and GPU 15GoRAM) using:

```auto
import torch
from transformers import AutoTokenizer
import transformers

model = "tiiuae/falcon-7b-instruct"
# model = "bigscience/bloomz-7b1-mt"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    trust_remote_code=True,
    device_map="auto",
)

```

While it works for this model (`9.95 + 4.48 = 14.43Go` checkpoints on disk) it does not work for `model = "bigscience/bloomz-7b1-mt"` (`14.10Go` single checkpoint on disk), exceeding the 12.7Go RAM : what is the root cause? is it because the first model is a two-sharded model?

So i moved to cpu offload strategy for the second model using :

```auto
import torch
from transformers import BitsAndBytesConfig, AutoTokenizer
import transformers

model = "tiiuae/falcon-7b-instruct"
# model = "bigscience/bloomz-7b1-mt"

quantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
    model_kwargs = {"quantization_config": quantization_config}
)

```

Here, the two models loading result in memory exceeding, which disturbs me more because i expected the ~14/15Go model weights to be dispatched between 12.7Go RAM and 15Go GPU RAM… Is it because `device=`map`` does not dispatch model components properly between cpu and gpu?

Thanks a lot for any link or discussion to help me understand what happens under the hood.

Regards

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [June 13, 2023, 12:45pm UTC](https://discuss.huggingface.co/t/general-question-about-large-model-loading/43073/2 "2023-06-13T12:45:37Z")

</div>

> [@ykacer](#):
>
> is it because the first model is a two-sharded model?

Yes, as we need the model sharded to load only a chunk of it in.

---

<div class="post-metadata">

**Author:** ![hhhmichel](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@hhhmichel](https://discuss.huggingface.co/u/hhhmichel)\
**Post date:** [November 28, 2024, 11:09am UTC](https://discuss.huggingface.co/t/general-question-about-large-model-loading/43073/3 "2024-11-28T11:09:44Z")

</div>

hi ykacer, did you find a way to load the model ? I am facing the same issue as you but with collab pro