# Which method is use HF Trainer with multiple GPU?

**URL:** <https://discuss.huggingface.co/t/which-method-is-use-hf-trainer-with-multiple-gpu/27074>\
**Category:** 🤗Transformers\
**Created:** [December 1, 2022, 4:12pm UTC](https://discuss.huggingface.co/t/which-method-is-use-hf-trainer-with-multiple-gpu/27074 "2022-12-01T16:12:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](https://discuss.huggingface.co/u/Indramal)\
**Post date:** [December 1, 2022, 4:12pm UTC](https://discuss.huggingface.co/t/which-method-is-use-hf-trainer-with-multiple-gpu/27074/1 "2022-12-01T16:12:53Z")

</div>

According to the following question, the trainer will handle multiple GPU work. What is the method it uses?

DataParallel (DP) or TensorParallel (TP) or PipelineParallel (PP) or DPP, what?

> [@Training using multiple GPUs](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279):
>
> I would like to train some models to multiple GPUs. Let suppose that I use model from HF library, but I am using my own trainers,dataloader,collators etc. Where I should focus to implement multiple GPU training? I need to make changes only in the Trainer class? If yes, can you give me a brief description? Thank you in avance.

---

<div class="post-metadata">

**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](https://discuss.huggingface.co/u/AndreaSottana)\
**Post date:** [December 2, 2022, 3:04pm UTC](https://discuss.huggingface.co/t/which-method-is-use-hf-trainer-with-multiple-gpu/27074/2 "2022-12-02T15:04:12Z")

</div>

Based on [this line of code](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1344) it looks like it is using `nn.DataParallel`, however I haven’t fully looked at every line of the `Trainer` class so they may also be using other methods in other points.

---

<div class="post-metadata">

**Author:** ![Indramal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/indramal/32/61744_2.png) [@Indramal](https://discuss.huggingface.co/u/Indramal)\
**Post date:** [December 3, 2022, 1:56am UTC](https://discuss.huggingface.co/t/which-method-is-use-hf-trainer-with-multiple-gpu/27074/3 "2022-12-03T01:56:42Z")

</div>

Thank you for your information. Old Trainer documents have to configure that. But new document doese not mention it. Old Doc - [Trainer — transformers 4.7.0 documentation](https://huggingface.co/transformers/v4.8.0/main_classes/trainer.html)

---

<div class="post-metadata">

**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\
**Post date:** [June 19, 2023, 4:56am UTC](https://discuss.huggingface.co/t/which-method-is-use-hf-trainer-with-multiple-gpu/27074/4 "2023-06-19T04:56:01Z")

</div>

hi @AndreaSottana , sorry I am trying to fine tune got-neo because of the Cuda memory issue I need to use multiple GPU. I use the trainer in hugging face which I understand it will use multiple GPu . but my results are very strange and very different than when I use 1 GPU. Would you please help me how you use multiple GPU for fine tunning the model.?

Many thanks

---

<div class="post-metadata">

**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\
**Post date:** [June 19, 2023, 4:56am UTC](https://discuss.huggingface.co/t/which-method-is-use-hf-trainer-with-multiple-gpu/27074/5 "2023-06-19T04:56:42Z")

</div>

Hi @Indramal  
sorry I am trying to fine tune got-neo because of the Cuda memory issue I need to use multiple GPU. I use the trainer in hugging face which I understand it will use multiple GPu . but my results are very strange and very different than when I use 1 GPU. Would you please help me how you use multiple GPU for fine tunning the model.?

Many thanks