# Why the model loading of llama2 is so slow?

**URL:** <https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927>\
**Category:** 🤗Transformers\
**Created:** [July 24, 2023, 1:48pm UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927 "2023-07-24T13:48:00Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![anxiangyinyun](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@anxiangyinyun](https://discuss.huggingface.co/u/anxiangyinyun)\
**Post date:** [July 24, 2023, 1:48pm UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927/1 "2023-07-24T13:48:00Z")

</div>

It took me about 1 hour to load the model of llama2-7b-hf. It’s such weird. What can I do to resolve this issue?  
The code is attached as follows:

```auto
from transformers import AutoModelForCausalLM
model_dir = "meta-llama/Llama-2-7b-chat-hf"
model = AutoModelForCausalLM.from_pretrained(
        model_dir,
        local_files_only=True,
        torch_dtype=torch.float16,
        device_map='auto'
    )

```

---

<div class="post-metadata">

**Author:** ![anxiangyinyun](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@anxiangyinyun](https://discuss.huggingface.co/u/anxiangyinyun)\
**Post date:** [July 26, 2023, 9:00am UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927/2 "2023-07-26T09:00:57Z")

</div>

Issue solved. It’s the disk problem, I copy the model to a “close” disk and the loading time reduce to 7~8 minutes.

---

<div class="post-metadata">

**Author:** ![vartu](https://avatars.discourse-cdn.com/v4/letter/v/3be4f8/32.png) [@vartu](https://discuss.huggingface.co/u/vartu)\
**Post date:** [August 11, 2023, 8:32pm UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927/3 "2023-08-11T20:32:27Z")

</div>

Can you explain what “close” disk refers to. Actually I am also facing the similar kind of issue. I am using ml.g5.12xlarge to infer llama2 model. I downloaded the model locally using snapshot\_download method. But model loading is taking more than 30 minutes.

---

<div class="post-metadata">

**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\
**Post date:** [August 13, 2023, 8:41am UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927/4 "2023-08-13T08:41:51Z")

</div>

hi @philschmid, I hope you are doing well. Sorry for fine tuning llama2, I create csv file with the Alpaca structure which has text column including ### instruction ### input ### response, for fine tuning the model I am confused which method with PEFT and QLora should I use, I am confused with many codes, would you please refer me to any code that is right for fine tuning with alpaca structure, and saving and inference for testing the model? In some code I saw they did tokenizer truncate and padding and refer label to -100 and in other no preprocessing is done. I appreciate your help. Many thanks.

---

<div class="post-metadata">

**Author:** ![teodor98](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/teodor98/32/23223_2.png) [@teodor98](https://discuss.huggingface.co/u/teodor98)\
**Post date:** [January 23, 2024, 7:18am UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927/5 "2024-01-23T07:18:48Z")

</div>

Hey, did you find a solution? I was loading the model for max 3min and out of nowhere it takes more than 30 mins.

Thanks

---

<div class="post-metadata">

**Author:** ![lepotatoguy](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@lepotatoguy](https://discuss.huggingface.co/u/lepotatoguy)\
**Post date:** [January 29, 2024, 3:41pm UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927/6 "2024-01-29T15:41:55Z")

</div>

I solved this by turning local\_files\_only to true. (Note: I had previously downloaded LLaMA 2)  
From loading in around 3.2 hours, it loaded within around 29 seconds after changing this.

```auto
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map=device_map,
    local_files_only = True
)

```

---

<div class="post-metadata">

**Author:** ![saikumar305](https://avatars.discourse-cdn.com/v4/letter/s/cdc98d/32.png) [@saikumar305](https://discuss.huggingface.co/u/saikumar305)\
**Post date:** [April 24, 2024, 7:42am UTC](https://discuss.huggingface.co/t/why-the-model-loading-of-llama2-is-so-slow/47927/7 "2024-04-24T07:42:37Z")

</div>

@lepotatoguy , can you please provide bnb\_config and device\_map values in details?