# Multi gpu not working

**URL:** <https://discuss.huggingface.co/t/multi-gpu-not-working/31059>\
**Category:** 🤗Transformers\
**Created:** [February 3, 2023, 3:53pm UTC](https://discuss.huggingface.co/t/multi-gpu-not-working/31059 "2023-02-03T15:53:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon47283947](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@anon47283947](https://discuss.huggingface.co/u/anon47283947)\
**Post date:** [February 3, 2023, 3:53pm UTC](https://discuss.huggingface.co/t/multi-gpu-not-working/31059/1 "2023-02-03T15:53:14Z")

</div>

Hi,  
I am using huggingface run\_clm.py to train gptj-6b model with 8 gpu’s. But it is not using all gpus and throwing cuda out of memory error. I have tried changing batch\_size with multiple of gpus. is there anything I have to mention for using all gpus?  
python3 run\_clm.py   
–model\_name\_or\_path EleutherAI/gpt-j-6B   
–dataset\_name glue   
–dataset\_config\_name cola   
–per\_device\_train\_batch\_size 8  
–per\_device\_eval\_batch\_size 8  
–do\_train   
–do\_eval   
–output\_dir /tmp/gptneo20b\_100  
–num\_train\_epochs=100   
–weight\_decay=0.01   
–learning\_rate=1e-5  
this is what am currently running.

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\
**Post date:** [February 3, 2023, 7:50pm UTC](https://discuss.huggingface.co/t/multi-gpu-not-working/31059/2 "2023-02-03T19:50:43Z")

</div>

> [@anon47283947](#):
>
> python3 run\_clm.py

You need to launch with `torchrun --n_procs_per_node=NGPUS run_clm.py ...` in order to enable multi-GPU. Another option is to use Accelerate’s CLI launcher directly:

```bash
accelerate launch --multi_gpu --num_processes=NGPUS run_clm.py...

```

---

<div class="post-metadata">

**Author:** ![anon47283947](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@anon47283947](https://discuss.huggingface.co/u/anon47283947)\
**Post date:** [February 3, 2023, 8:11pm UTC](https://discuss.huggingface.co/t/multi-gpu-not-working/31059/3 "2023-02-03T20:11:08Z")

</div>

> [@muellerzr](#):
>
> torch

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/0203f4ad7df5dedecd02021957b7a9e1abbb3d93.png)  
Here, it shows all 8 gpu’s with same memory error. And why pytorch reserving memory in all 8gpu’s? and not 1.  
(Using accelerate launcher)