# Setup for Deepspeed Multi GPU Training

**URL:** <https://discuss.huggingface.co/t/setup-for-deepspeed-multi-gpu-training/20215>\
**Category:** DeepSpeed\
**Created:** [July 10, 2022, 8:34pm UTC](https://discuss.huggingface.co/t/setup-for-deepspeed-multi-gpu-training/20215 "2022-07-10T20:34:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![maxmaier](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@maxmaier](https://discuss.huggingface.co/u/maxmaier)\
**Post date:** [July 10, 2022, 8:34pm UTC](https://discuss.huggingface.co/t/setup-for-deepspeed-multi-gpu-training/20215/1 "2022-07-10T20:34:35Z")

</div>

According to [Trainer — transformers 4.4.2 documentation](https://huggingface.co/transformers/v4.4.2/main_classes/trainer.html) (see "Deployment in Notebooks) the following code in a Notebook shall work with multiple GPUs:

# DeepSpeed requires a distributed environment even when only one process is used.

# This emulates a launcher in the notebook

import os  
os.environ[‘MASTER\_ADDR’] = ‘localhost’  
os.environ[‘MASTER\_PORT’] = ‘9994’ # modify if RuntimeError: Address already in use  
os.environ[‘RANK’] = “0”  
os.environ[‘LOCAL\_RANK’] = “0”  
os.environ[‘WORLD\_SIZE’] = “1”

# Now proceed as normal, plus pass the deepspeed config file

training\_args = TrainingArguments(…, deepspeed=“ds\_config.json”)  
trainer = Trainer(…)  
trainer.train()

However, I am struggling to get this running with 2 GPUs. There seems to be no way to manually tell deepspeed to use 2 GPUs. The documentation says deepseed should detect them automatically but it does not on my system. It only runs on 1 GPU. Depending on the Rank setting it runs either on GPU 0 or 1 but never on both.  
(I need to run this on 2 GPUs because I don’t have an RTX3090 with enough memory)

Is there a way to manually tell deepspeed to use 2 GPUs in a Jupyter Notebook like the above example?

---

<div class="post-metadata">

**Author:** ![Charm3link](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/charm3link/32/12252_2.png) [@Charm3link](https://discuss.huggingface.co/u/Charm3link)\
**Post date:** [October 26, 2022, 8:18pm UTC](https://discuss.huggingface.co/t/setup-for-deepspeed-multi-gpu-training/20215/2 "2022-10-26T20:18:34Z")

</div>

I have the same issue. It seems like the only solution is to use the deepspeed CLI launcher, or emulate a distributed environment locally - which seems like more trouble than it’s worth.  
Any other guidance here would be appreciated!

---

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](https://discuss.huggingface.co/u/Indramal)\
**Post date:** [December 7, 2022, 8:49am UTC](https://discuss.huggingface.co/t/setup-for-deepspeed-multi-gpu-training/20215/3 "2022-12-07T08:49:28Z")

</div>

> If you want to use more than 1 GPU, you must use a multi-process environment for DeepSpeed to work. That is, you have to use the launcher for that purpose and this cannot be accomplished by emulating the distributed environment presented at the beginning of this section.

Link - [DeepSpeed Integration](https://huggingface.co/docs/transformers/main_classes/deepspeed#deployment-in-notebooks)