# Transformer model parallel does not work with Pytorch DDP for multi-node training

**URL:** <https://discuss.huggingface.co/t/transformer-model-parallel-does-not-work-with-pytorch-ddp-for-multi-node-training/22375>\
**Category:** 🤗Transformers\
**Created:** [September 1, 2022, 1:16am UTC](https://discuss.huggingface.co/t/transformer-model-parallel-does-not-work-with-pytorch-ddp-for-multi-node-training/22375 "2022-09-01T01:16:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dyu2](https://avatars.discourse-cdn.com/v4/letter/d/779978/32.png) [@dyu2](https://discuss.huggingface.co/u/dyu2)\
**Post date:** [September 1, 2022, 1:16am UTC](https://discuss.huggingface.co/t/transformer-model-parallel-does-not-work-with-pytorch-ddp-for-multi-node-training/22375/1 "2022-09-01T01:16:54Z")

</div>

The [naive model parallelism](https://github.com/huggingface/transformers/blob/b487096b02307cd6e0f132b676cdcc7255fe8e74/src/transformers/models/t5/modeling_t5.py#L1301) in Transformer models such as T5 does not work with pytorch Distributed Data Parallel for multi-node training. It will freeze at the step of torch.nn.parallel.DistributedDataParallel(Model). I follow [this instruction](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#combining-ddp-with-model-parallelism) to write the code. Anyone met similar situations?