# Does the HF Trainer class support multi-node training?

**URL:** <https://discuss.huggingface.co/t/does-the-hf-trainer-class-support-multi-node-training/5527>\
**Category:** Beginners\
**Created:** [April 15, 2021, 4:00pm UTC](https://discuss.huggingface.co/t/does-the-hf-trainer-class-support-multi-node-training/5527 "2021-04-15T16:00:24Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\
**Post date:** [April 15, 2021, 4:00pm UTC](https://discuss.huggingface.co/t/does-the-hf-trainer-class-support-multi-node-training/5527/1 "2021-04-15T16:00:24Z")

</div>

Does the HF [Trainer](https://huggingface.co/transformers/main_classes/trainer.html) class support multi-node training? Or only single-host multi-GPU training?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 15, 2021, 4:10pm UTC](https://discuss.huggingface.co/t/does-the-hf-trainer-class-support-multi-node-training/5527/2 "2021-04-15T16:10:19Z")

</div>

It supports both single-node and multi-node distributed training with the PyTorch launcher (`torch.distributed.launch`)

---

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\
**Post date:** [April 15, 2021, 4:13pm UTC](https://discuss.huggingface.co/t/does-the-hf-trainer-class-support-multi-node-training/5527/3 "2021-04-15T16:13:44Z")

</div>

thanks! who is doing the cross-instance allreduce? PyTorch DDP? Or Horovod? or some custom HF allreduce? any sample 🙂 ?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 15, 2021, 4:27pm UTC](https://discuss.huggingface.co/t/does-the-hf-trainer-class-support-multi-node-training/5527/4 "2021-04-15T16:27:12Z")

</div>

It’s standard PyTorch DDP behind the scenes.

---

<div class="post-metadata">

**Author:** ![mnichls](https://avatars.discourse-cdn.com/v4/letter/m/90ced4/32.png) [@mnichls](https://discuss.huggingface.co/u/mnichls)\
**Post date:** [January 9, 2023, 9:03pm UTC](https://discuss.huggingface.co/t/does-the-hf-trainer-class-support-multi-node-training/5527/5 "2023-01-09T21:03:35Z")

</div>

@sgugger I tried the distributed training per this [HF blog](https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face) and it failed with multi-node training. It gave an mpirun error when i tried to use multiple instances (similar to when i tried the [AWS blog](https://aws.amazon.com/blogs/machine-learning/distributed-fine-tuning-of-a-bert-large-model-for-a-question-answering-task-using-hugging-face-transformers-on-amazon-sagemaker/)). In both cases i tried with ml.p3.16xlarge and ml.p3dn.24xlarge instances.

Edit: I tried as written and with all variations of current PyTorch/HF URI containers (can’t link due to new account, limited to 2)

Any ideas on how to fix?

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9c45936b4066dbd79560bd4d7c873d78538b5573.png)