# Running a Trainer in DistributedDataParallel mode

**URL:** <https://discuss.huggingface.co/t/running-a-trainer-in-distributeddataparallel-mode/1718>\
**Category:** 🤗Transformers\
**Created:** [October 23, 2020, 7:16pm UTC](https://discuss.huggingface.co/t/running-a-trainer-in-distributeddataparallel-mode/1718 "2020-10-23T19:16:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![deppen8](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/deppen8/32/516_2.png) [@deppen8](https://discuss.huggingface.co/u/deppen8)\
**Post date:** [October 23, 2020, 7:16pm UTC](https://discuss.huggingface.co/t/running-a-trainer-in-distributeddataparallel-mode/1718/1 "2020-10-23T19:16:25Z")

</div>

I am trying to train a model on four GPUs (AWS ml.p3.8xlarge). As far as I can tell, to get my model to train in `DistributedDataParallel`, I only need to specify some integer value for `local_rank`. But my understanding is that this will only distribute the training across a single GPU (whichever I specify with `local_rank`).

What is the proper way to launch `DistributedDataParallel` training across all four GPUs using a `Trainer`? Do I have to launch something via the command line (as hinted at here [https://github.com/huggingface/transformers/issues/1651](https://github.com/huggingface/transformers/issues/1651))?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 24, 2020, 8:24am UTC](https://discuss.huggingface.co/t/running-a-trainer-in-distributeddataparallel-mode/1718/2 "2020-10-24T08:24:39Z")

</div>

Hi @deppen8  
Yes, you’ll need to use `torch.distributed.launch` for distributed training.  
See [this command](https://github.com/huggingface/transformers/tree/master/examples/text-classification#distributed-training) for an example.