# Torch.distributed.launch question

**URL:** <https://discuss.huggingface.co/t/torch-distributed-launch-question/24653>\
**Category:** Beginners\
**Created:** [October 19, 2022, 8:11am UTC](https://discuss.huggingface.co/t/torch-distributed-launch-question/24653 "2022-10-19T08:11:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [October 19, 2022, 8:11am UTC](https://discuss.huggingface.co/t/torch-distributed-launch-question/24653/1 "2022-10-19T08:11:51Z")

</div>

Hello,  
I’m trying a 4 GPU training, using this [code](https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translation.py):

`python -m torch.distributed.launch --nproc_per_node = 4 run_translation.py --other_args`

Using that code, it seems that with `per_device_train_batch_size=3` I have the `cuda Out of memory` error, using `per_device_train_batch_size=2` instead I don’t get the error.

But if I train on a single GPU, with `per_device_train_batch_size=3`, I have no memory error.

I have such a small batch size because each GPU has 15GB of RAM and I have inputs of len 1024.

I also tried these lines of code but nothing has changed:

```auto
torch.cuda.set_device(rank)
torch.cuda.empty_cache()

```

Does distributed training use more GPU memory?  
I’m only using the code above, is there anything else I should use?

Thanks

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 19, 2022, 1:44pm UTC](https://discuss.huggingface.co/t/torch-distributed-launch-question/24653/2 "2022-10-19T13:44:17Z")

</div>

> Does distributed training use more GPU memory?

Yes, it uses slightly more memory on GPU 0 which contains buffers to synchronize gradients, so if you were very close to your memory, you might need to slightly lower the batch size (though it’s multiplied by 4 when running on 4 GPUs, so you are still training at bigger batch size).

---

<div class="post-metadata">

**Author:** ![GenV](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/genv/32/12132_2.png) [@GenV](https://discuss.huggingface.co/u/GenV)\
**Post date:** [October 19, 2022, 2:06pm UTC](https://discuss.huggingface.co/t/torch-distributed-launch-question/24653/3 "2022-10-19T14:06:55Z")

</div>

Yes, 2\*4. Thanks for the answer!