# Big Model Inference: CPU/Disk Offloading for Transformers Using from\_pretrained

**URL:** <https://discuss.huggingface.co/t/big-model-inference-cpu-disk-offloading-for-transformers-using-from-pretrained/75165>\
**Category:** 🤗Accelerate\
**Created:** [February 28, 2024, 12:24pm UTC](https://discuss.huggingface.co/t/big-model-inference-cpu-disk-offloading-for-transformers-using-from-pretrained/75165 "2024-02-28T12:24:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hellov](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@hellov](https://discuss.huggingface.co/u/hellov)\
**Post date:** [February 28, 2024, 12:24pm UTC](https://discuss.huggingface.co/t/big-model-inference-cpu-disk-offloading-for-transformers-using-from-pretrained/75165/1 "2024-02-28T12:24:21Z")

</div>

Hello, I’m exploring methods to manage CUDA Out of Memory (OOM) errors during the inference of 70 billion parameter models without resorting to quantization. Specifically, I’m interested in leveraging CPU/disk offloading. Does the Accelerate library offer solutions for this?

I’ve examined the current documentation, which appears to focus on custom NLP models rather than facilitating the use of pretrained large models like LLaMA 2 70B. Here’s the guide I found: [Hugging Face’s Accelerate Big Model Inference Guide](https://huggingface.co/docs/accelerate/concept_guides/big_model_inference).

Additionally, I’ve looked into FlexGen, but it seems the project is no longer active: [FlexGen GitHub Repository](https://github.com/FMInference/FlexGen).

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [February 28, 2024, 3:56pm UTC](https://discuss.huggingface.co/t/big-model-inference-cpu-disk-offloading-for-transformers-using-from-pretrained/75165/2 "2024-02-28T15:56:32Z")

</div>

> [@hellov](#):
>
> Does the Accelerate library offer solutions for this?

Yes. Just set `device_map="auto"` in your call and it’ll do this automatically

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [March 6, 2024, 12:39pm UTC](https://discuss.huggingface.co/t/big-model-inference-cpu-disk-offloading-for-transformers-using-from-pretrained/75165/3 "2024-03-06T12:39:30Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.