# Model Parallelism, how to parallelize transformer?

**URL:** <https://discuss.huggingface.co/t/model-parallelism-how-to-parallelize-transformer/6260>\
**Category:** Beginners\
**Created:** [May 19, 2021, 5:01pm UTC](https://discuss.huggingface.co/t/model-parallelism-how-to-parallelize-transformer/6260 "2021-05-19T17:01:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![valgi0](https://avatars.discourse-cdn.com/v4/letter/v/e5b9ba/32.png) [@valgi0](https://discuss.huggingface.co/u/valgi0)\
**Post date:** [May 19, 2021, 5:01pm UTC](https://discuss.huggingface.co/t/model-parallelism-how-to-parallelize-transformer/6260/1 "2021-05-19T17:01:15Z")

</div>

Hi there,  
I am pretty new, I hope to do it right:)

I have two gpus nvidia, which work fine. I can train model on each of them, I can use data parallelism. I wonder if I can parallelize the model itself. Surfing the internet I found it is possible but no one tells how. Some frameworks do it as torchgpipe, deepspeed PipelineModule, Fairscale but they wants sequential models but transformers are hard to turn sequential.

Can you point me in the right direction?

Specs:  
I want to parallelize BERT model on two gpus titan xp.

Thank you, every hints or helps will be appreciated

valgi0

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [May 20, 2021, 12:37pm UTC](https://discuss.huggingface.co/t/model-parallelism-how-to-parallelize-transformer/6260/2 "2021-05-20T12:37:46Z")

</div>

hey @valgi0 my suggestion would be to try out the new `accelerate` library: [GitHub - huggingface/accelerate: 🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision](https://github.com/huggingface/accelerate)

in particular, there is an nlp example that shows you how to configure `accelerate` for the multi-GPU case here: [accelerate/examples at main · huggingface/accelerate · GitHub](https://github.com/huggingface/accelerate/tree/main/examples)

---

<div class="post-metadata">

**Author:** ![valgi0](https://avatars.discourse-cdn.com/v4/letter/v/e5b9ba/32.png) [@valgi0](https://discuss.huggingface.co/u/valgi0)\
**Post date:** [June 18, 2021, 8:38am UTC](https://discuss.huggingface.co/t/model-parallelism-how-to-parallelize-transformer/6260/3 "2021-06-18T08:38:33Z")

</div>

thank you for the answer  
I am sorry to answer after so much time but I was pretty busy.  
However, I checked accelerate and it performs only data parallelism. Am I right?

I found out some models as T5, GPT2 have parallelize() method to split encoder and decoder on different devices. But that has serious limits, you need a balanced encoder decoder for examples.

I would like do the same but with BERT, I tried to manually distribute encoder layers on the two different gpus. It seems to work but it lacks of optimization and it does not work with Trainer and other tools any more.

I dont know, if you have any other ideas come forward 🙂  
Thank you

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [June 18, 2021, 1:44pm UTC](https://discuss.huggingface.co/t/model-parallelism-how-to-parallelize-transformer/6260/4 "2021-06-18T13:44:40Z")

</div>

ah i misunderstood your original question - from what i understand deepspeed supports model parallelism of the sort you describe: [Feature Overview - DeepSpeed](https://www.deepspeed.ai/features/#model-parallelism)

there’s also a dedicated page for the deepspeed integration in `transformers` which might help: [DeepSpeed Integration — transformers 4.7.0 documentation](https://huggingface.co/transformers/main_classes/deepspeed.html?highlight=deepspeed#deepspeed-integration)

i know stas was able to fine-tune T5 on a single gpu this way, so unless you have a very specific reason to want to parallelise BERT, this approach might be the best

hth!