# Multi gpu training

**URL:** <https://discuss.huggingface.co/t/multi-gpu-training/4021>\
**Category:** 🤗Transformers\
**Created:** [February 25, 2021, 9:15pm UTC](https://discuss.huggingface.co/t/multi-gpu-training/4021 "2021-02-25T21:15:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kaoutar55](https://avatars.discourse-cdn.com/v4/letter/k/7ba0ec/32.png) [@kaoutar55](https://discuss.huggingface.co/u/kaoutar55)\
**Post date:** [February 25, 2021, 9:15pm UTC](https://discuss.huggingface.co/t/multi-gpu-training/4021/1 "2021-02-25T21:15:36Z")

</div>

It seems that the hugging face implementation still uses nn.DataParallel for one node multi-gpu training.  
In the pytorch documentation page, it clearly states that " It is recommended to use DistributedDataParallel instead of DataParallel to do multi-GPU training, even if there is only a single node. Could you please clarify if my understanding is correct? and if your training support DistributedDataParallel for one node with multiple GPUs.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [February 25, 2021, 11:51pm UTC](https://discuss.huggingface.co/t/multi-gpu-training/4021/2 "2021-02-25T23:51:52Z")

</div>

Both are supported by the Hugging Face `Trainer`. You just have to use the pytorch launcher to use `DistributedDataParallel`, see an example [here](https://github.com/huggingface/transformers/tree/master/examples#distributed-training-and-mixed-precision).

---

<div class="post-metadata">

**Author:** ![Lee1990](https://avatars.discourse-cdn.com/v4/letter/l/4da419/32.png) [@Lee1990](https://discuss.huggingface.co/u/Lee1990)\
**Post date:** [April 24, 2022, 9:04am UTC](https://discuss.huggingface.co/t/multi-gpu-training/4021/3 "2022-04-24T09:04:25Z")

</div>

how to find the example?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [April 24, 2022, 2:03pm UTC](https://discuss.huggingface.co/t/multi-gpu-training/4021/4 "2022-04-24T14:03:25Z")

</div>

See [here](https://pytorch.org/docs/stable/distributed.html#launch-utility). E.g.

```auto
python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE YOUR_TRAINING_SCRIPT.py (--arg1 --arg2 --arg3 and all other arguments of your training script)

```