# Multiple GPU in SFTTrainer

**URL:** <https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899>\
**Category:** Beginners\
**Created:** [June 13, 2024, 6:40am UTC](https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899 "2024-06-13T06:40:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Milku](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@Milku](https://discuss.huggingface.co/u/Milku)\
**Post date:** [June 13, 2024, 6:40am UTC](https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899/1 "2024-06-13T06:40:14Z")

</div>

How can i use SFTTrainer to leverage all GPUs automatically? If I add device\_map=“auto” I get a Cuda out of memory exception. I although I have 4x Nvidia T4 GPUs

Cuda is installed and my environment can see the available GPUs.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [June 13, 2024, 7:32am UTC](https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899/2 "2024-06-13T07:32:02Z")

</div>

Hi,

I would recommend to take a look at the example scripts here: [alignment-handbook/scripts at main · huggingface/alignment-handbook · GitHub](https://github.com/huggingface/alignment-handbook/tree/main/scripts). It includes scripts which can be run with ZeRO-3 on 8 GPUs. For that, they define an Accelerate config. Basically you need to run the script with `accelerate launch --config_file recipes/accelerate_configs/deepspeed_zero3.yaml` (in the YAML you define on what hardware the script should be run).

---

<div class="post-metadata">

**Author:** ![Milku](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@Milku](https://discuss.huggingface.co/u/Milku)\
**Post date:** [June 15, 2024, 2:02pm UTC](https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899/3 "2024-06-15T14:02:56Z")

</div>

The problem is I need to run it via python because I’m using vertex ai pipelines for MLops. Why can’t I avoid calling accelerate on my script via bash?

---

<div class="post-metadata">

**Author:** ![Milku](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@Milku](https://discuss.huggingface.co/u/Milku)\
**Post date:** [June 28, 2024, 6:00am UTC](https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899/4 "2024-06-28T06:00:36Z")

</div>

Is using Optimum a better solution if I want to run it via python without cli launch like you mentioned?

---

<div class="post-metadata">

**Author:** ![saeed899](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@saeed899](https://discuss.huggingface.co/u/saeed899)\
**Post date:** [December 27, 2024, 2:20pm UTC](https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899/5 "2024-12-27T14:20:28Z")

</div>

Same issue.  
@nielsr What when we have only python notebook access?