# Infer\_auto\_device\_map returns empty

**URL:** <https://discuss.huggingface.co/t/infer-auto-device-map-returns-empty/24286>\
**Category:** 🤗Accelerate\
**Created:** [October 11, 2022, 6:20pm UTC](https://discuss.huggingface.co/t/infer-auto-device-map-returns-empty/24286 "2022-10-11T18:20:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rachith](https://avatars.discourse-cdn.com/v4/letter/r/e0b2c6/32.png) [@rachith](https://discuss.huggingface.co/u/rachith)\
**Post date:** [October 11, 2022, 6:20pm UTC](https://discuss.huggingface.co/t/infer-auto-device-map-returns-empty/24286/1 "2022-10-11T18:20:37Z")

</div>

Hi,  
Following the instructions in [this post](https://huggingface.co/blog/accelerate-large-models) to load the same opt 13b. I have access to 8 Nvidea A100 80GB machines.

`model = AutoConfig.from_pretrained("facebook/opt-13b")` runs successfully and the model card is available to use. Example of the model

```auto
OPTForCausalLM(
  (model): OPTModel(
    (decoder): OPTDecoder(
      (embed_tokens): Embedding(50272, 5120, padding_idx=1)
      (embed_positions): OPTLearnedPositionalEmbedding(2050, 5120)
      (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      (layers): ModuleList(
...
...

```

However, `device_map = infer_auto_device_map(model)` returns `{'': 0}` despite 8 GPUs being available.

The code is exactly the same as the blog post

```auto
from accelerate import infer_auto_device_map, init_empty_weights
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("facebook/opt-13b")
with init_empty_weights():
    model = AutoModelForCausalLM.from_config(config)

device_map = infer_auto_device_map(model)

```

Not sure why this is the case.  
Any suggestions/help appreciated. Thanks!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 11, 2022, 7:06pm UTC](https://discuss.huggingface.co/t/infer-auto-device-map-returns-empty/24286/2 "2022-10-11T19:06:17Z")

</div>

Your model fits on the whole GPU, so `infer_auto_device_map` just returns that (`{'': 0"}` means put everything on GPU 0). You can pass along a `max_memory` argument if you want to limit how much memory to use on each GPU.

---

<div class="post-metadata">

**Author:** ![YzyLmc](https://avatars.discourse-cdn.com/v4/letter/y/eb9ed0/32.png) [@YzyLmc](https://discuss.huggingface.co/u/YzyLmc)\
**Post date:** [March 15, 2023, 5:43pm UTC](https://discuss.huggingface.co/t/infer-auto-device-map-returns-empty/24286/3 "2023-03-15T17:43:07Z")

</div>

I found that Auto classes sometimes cause this `{'': 0}` issue. In my case, I’m using `T5-large` for inference, and I found `infer_auto_device_map()` would return the device map properly if I load the model with `T5ForConditionalGeneration` class but `{'': 0}` if loaded with `AutoModelForSeq2SeqLM`. However, Auto classes are not always like that. For example, `infer_auto_device_map()` works fine with `google/flan_ul2` loaded with `AutoModelForSeq2SeqLM`