# CPU Memory Usage with ‘low\_cpu\_mem\_usage=True’ and ‘torch\_dtype=“auto”’ flags

**URL:** <https://discuss.huggingface.co/t/cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/52893>\
**Category:** 🤗Accelerate\
**Created:** [August 31, 2023, 12:10am UTC](https://discuss.huggingface.co/t/cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/52893 "2023-08-31T00:10:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![hanguo](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hanguo](https://discuss.huggingface.co/u/hanguo)\
**Post date:** [August 31, 2023, 12:10am UTC](https://discuss.huggingface.co/t/cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/52893/1 "2023-08-31T00:10:31Z")

</div>

I have noticed that when I load the 70B model (specifically LLaMA-2) into the CPU using the ‘low\_cpu\_mem\_usage=True’ and ‘torch\_dtype=“auto”’ flags, it has almost no effect on the CPU memory usage. However, if I remove either of these flags, it consumes a significant amount of memory. I am curious about the reason behind this behavior. Is there any memory-mapping happening in the background? And if so, when does it trigger? I would really appreciate it if you could help me understand this better.

Thank you for your time!

---

<div class="post-metadata">

**Author:** ![marcsun13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcsun13/32/19059_2.png) [@marcsun13](https://discuss.huggingface.co/u/marcsun13)\
**Post date:** [August 31, 2023, 10:40pm UTC](https://discuss.huggingface.co/t/cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/52893/2 "2023-08-31T22:40:30Z")

</div>

Hi @hanguo, I tested and it has something to do with safetensors format + fp16 (torch\_dtype=“auto will set torch\_dtype=torch.float16) + low\_cpu\_mem\_usage = True. The memory consumption also skyrocket with we use the PyTorch bin format (pytorch bin + fp16 + low\_cpu\_mem\_usage = True). Maybe try in another channel as it is related to the integration of llama2 and safetensors ?

---

<div class="post-metadata">

**Author:** ![hanguo](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hanguo](https://discuss.huggingface.co/u/hanguo)\
**Post date:** [August 31, 2023, 11:51pm UTC](https://discuss.huggingface.co/t/cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/52893/3 "2023-08-31T23:51:38Z")

</div>

Thanks for the response! What’s a good alternative channel for question like this?

---

<div class="post-metadata">

**Author:** ![marcsun13](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcsun13/32/19059_2.png) [@marcsun13](https://discuss.huggingface.co/u/marcsun13)\
**Post date:** [September 1, 2023, 1:58pm UTC](https://discuss.huggingface.co/t/cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/52893/4 "2023-09-01T13:58:20Z")

</div>

Maybe in the transformers channel as they might have more insights about safetensors and how llama was implemented in transformers !

---

<div class="post-metadata">

**Author:** ![hanguo](https://avatars.discourse-cdn.com/v4/letter/h/3bc359/32.png) [@hanguo](https://discuss.huggingface.co/u/hanguo)\
**Post date:** [September 1, 2023, 2:44pm UTC](https://discuss.huggingface.co/t/cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/52893/5 "2023-09-01T14:44:40Z")

</div>

Got it, thanks again!

EDIT: the [new post](https://discuss.huggingface.co/t/llama-2-cpu-memory-usage-with-low-cpu-mem-usage-true-and-torch-dtype-auto-flags/53154)