# CUDA error: device-side assert triggered on device\_map="auto"

**URL:** <https://discuss.huggingface.co/t/cuda-error-device-side-assert-triggered-on-device-map-auto/71489>\
**Category:** 🤗Transformers\
**Created:** [February 1, 2024, 3:07pm UTC](https://discuss.huggingface.co/t/cuda-error-device-side-assert-triggered-on-device-map-auto/71489 "2024-02-01T15:07:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![CKeibel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ckeibel/32/55416_2.png) [@CKeibel](https://discuss.huggingface.co/u/CKeibel)\
**Post date:** [February 1, 2024, 3:07pm UTC](https://discuss.huggingface.co/t/cuda-error-device-side-assert-triggered-on-device-map-auto/71489/1 "2024-02-01T15:07:22Z")

</div>

Hello community!

I’m currently trying to fine-tune the `HuggingFaceH4/zephyr-7b-beta` by splitting the model to multiple GPUs.  
I have 4 RTX 4090 available, but I get an error during inference/training when I load it with device\_map=“auto”.

```auto
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "HuggingFaceH4/zephyr-7b-beta"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map=device_map)

x = tokenizer("Hello World", return_tensors="pt").to("cuda")
with torch.no_grad():
    model(**x)

```

```auto
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

```

But strangely enough, I can move single layers (one per GPU) to a different GPU without getting this error. As soon as I move a second one, I get the error.

**Works fine** :

```auto
device_map = {
    'model.embed_tokens': 0,
    'model.layers.0': 0,
    'model.layers.1': 0,
    'model.layers.2': 0,
    'model.layers.3': 1, # 1
    'model.layers.4': 3, # 3
    'model.layers.5': 2, # 2
    'model.layers.6': 0,
    ...
    'model.layers.31': 0,
    'model.norm': 1, # 1
    'lm_head': 1 # 1
}

```

**Not working** :

```auto
device_map = {
    'model.embed_tokens': 0,
    'model.layers.0': 0,
    'model.layers.1': 0,
    'model.layers.2': 0,
    'model.layers.3': 3, # 3 <------------------ changed to 3
    'model.layers.4': 3, # 3
    'model.layers.5': 2, # 2
    'model.layers.6': 0,
    ...
    'model.layers.31': 0,
    'model.norm': 1, # 1
    'lm_head': 1 # 1
}

```

I would be very grateful for any help or ideas. 🙂

Kind regards,  
Christopher

---

<div class="post-metadata">

**Author:** ![CKeibel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ckeibel/32/55416_2.png) [@CKeibel](https://discuss.huggingface.co/u/CKeibel)\
**Post date:** [February 7, 2024, 7:39am UTC](https://discuss.huggingface.co/t/cuda-error-device-side-assert-triggered-on-device-map-auto/71489/2 "2024-02-07T07:39:51Z")

</div>

I finally had the opportunity to test the code on another cluster (4xA100), the same error occurs there.

---

<div class="post-metadata">

**Author:** ![Sahibsingh12](https://avatars.discourse-cdn.com/v4/letter/s/ba9def/32.png) [@Sahibsingh12](https://discuss.huggingface.co/u/Sahibsingh12)\
**Post date:** [February 11, 2024, 6:33pm UTC](https://discuss.huggingface.co/t/cuda-error-device-side-assert-triggered-on-device-map-auto/71489/3 "2024-02-11T18:33:45Z")

</div>

@CKeibel I am also getting the same error and that for summarization pipeline

---

<div class="post-metadata">

**Author:** ![CKeibel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ckeibel/32/55416_2.png) [@CKeibel](https://discuss.huggingface.co/u/CKeibel)\
**Post date:** [February 12, 2024, 10:19am UTC](https://discuss.huggingface.co/t/cuda-error-device-side-assert-triggered-on-device-map-auto/71489/4 "2024-02-12T10:19:05Z")

</div>

Unfortunately, I don’t have any update yet either ☹  
Haven’t gotten it to run yet

---

<div class="post-metadata">

**Author:** ![shadowshadow](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@shadowshadow](https://discuss.huggingface.co/u/shadowshadow)\
**Post date:** [December 8, 2024, 8:57am UTC](https://discuss.huggingface.co/t/cuda-error-device-side-assert-triggered-on-device-map-auto/71489/5 "2024-12-08T08:57:46Z")

</div>

Hi, I also come up with this problem when setting `device_map="auto"`, changing it with `device_map="cuda"` will fix this error but will only use 1 gpu for inference. Have you fixed it now?