# Training using multiple GPUs

**URL:** <https://discuss.huggingface.co/t/training-using-multiple-gpus/1279>\
**Category:** Beginners\
**Created:** [September 28, 2020, 3:23pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279 "2020-09-28T15:23:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![manzar](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@manzar](https://discuss.huggingface.co/u/manzar)\
**Post date:** [September 28, 2020, 3:23pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/1 "2020-09-28T15:23:45Z")

</div>

I would like to train some models to multiple GPUs.  
Let suppose that I use model from HF library, but I am using my own trainers,dataloader,collators etc.

Where I should focus to implement multiple GPU training? I need to make changes only in the Trainer class? If yes, can you give me a brief description?

Thank you in avance.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 28, 2020, 11:53pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/2 "2020-09-28T23:53:48Z")

</div>

The Trainer class automatically handles multi-GPU training, you don’t have to do anything special.

---

<div class="post-metadata">

**Author:** ![manzar](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@manzar](https://discuss.huggingface.co/u/manzar)\
**Post date:** [September 29, 2020, 1:24pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/3 "2020-09-29T13:24:14Z")

</div>

yeah but I am using a Trainer class which is implemented from scratch. That’s why I am asking…

Here is an example of the Trainer class for EncoderDecoder models:

```
class EncoderDecoderTransformerTrainer:

    def __init__ (self, model,
                 optimizer,
                 patience,
                 scheduler=None,
                 checkpoint_dir=None,
                 clip=None,
                 device='cpu'):

        self.model = model.to(device)
        self.optimizer = optimizer
        self.scheduler = scheduler
        self.checkpoint_dir = checkpoint_dir
        self.clip = clip
        self.device = device
        self.patience = patience

    def calc_val_loss(self, val_loader):

        self.model.eval()
        with torch.no_grad():
            avg_val_loss = 0

            for index, batch in enumerate(tqdm(val_loader)):
                inputs = to_device(batch[0], device=self.device)
                inputs_att = to_device(batch[1], device=self.device)
                padded_targets = to_device(batch[2], device=self.device)
                replaced_targets = to_device(batch[3], device=self.device)
                targets_att = to_device(batch[4], device=self.device)

                outputs = self.model(input_ids=inputs,
                                     attention_mask=inputs_att,
                                     decoder_input_ids=padded_targets,
                                     decoder_attention_mask=targets_att,
                                     labels=replaced_targets)
                lm_loss = outputs[0]
                pred_scores = outputs[1]
                last_hidden = outputs[2]
                avg_val_loss += lm_loss.item()

            avg_val_loss = avg_val_loss / len(val_loader)
            return avg_val_loss

    def print_epoch(self, epoch, avg_train_epoch_loss, avg_val_epoch_loss,
                    cur_patience, strt):

        print("Epoch {}:".format(epoch+1))
        print("Train loss: {} | Train PPL: {}".format(
            avg_train_epoch_loss, math.exp(avg_train_epoch_loss)))
        print("Val loss: {} | Val PPL: {}".format(avg_val_epoch_loss,
              math.exp(avg_val_epoch_loss)))
        print("Patience left: {}".format(self.patience-cur_patience))
        print("Time: {} mins".format((time.time() - strt) / 60.0))
        print("++++++++++++++++++")

    def save_epoch(self, epoch, loss=None):

        if not os.path.exists(self.checkpoint_dir):
            os.makedirs(self.checkpoint_dir)
        # torch.save(self.model.state_dict(), os.path.join(
        # self.checkpoint_dir, '{}_{}.pth'.format(epoch, 'model_checkpoint')))

        # we use the proposed method for saving EncoderDecoder model
        self.model.save_pretrained(os.path.join(self.checkpoint_dir,'model_checkpoint'))
        torch.save(self.optimizer.state_dict(), os.path.join(
            self.checkpoint_dir,'optimizer_checkpoint'))

    def train_step(self, batch):
        self.model.train()
        self.optimizer.zero_grad()

        inputs = to_device(batch[0], device=self.device)
        inputs_att = to_device(batch[1], device=self.device)
        padded_targets = to_device(batch[2], device=self.device)
        replaced_targets = to_device(batch[3], device=self.device)
        targets_att = to_device(batch[4], device=self.device)
        print(inputs.shape)
        print(padded_targets.shape)
        # episis den eimai sigouros gia to ti prepei na dwsw san
        # decoder_input_ids (ta input ids i ta padded_targets??)

        outputs = self.model(input_ids=inputs,
                             attention_mask=inputs_att,
                             decoder_input_ids=padded_targets,
                             decoder_attention_mask=targets_att,
                             labels=replaced_targets)

        lm_loss = outputs[0]
        # print(lm_loss)
        pred_scores = outputs[1]
        last_hidden = outputs[2]
        return lm_loss, last_hidden

    def train_epochs(self, n_epochs, train_loader, val_loader):

        best_val_loss, cur_patience = 10000, 0

        print("Training model....")
        self.model.train()

        for epoch in range(n_epochs):
            if cur_patience == self.patience:
                break

            avg_train_loss = 0
            strt = time.time()

            for index, sample_batch in enumerate(tqdm(train_loader)):

                loss, _ = self.train_step(sample_batch)
                avg_train_loss += loss.item()
                loss.backward(retain_graph=False)
                if self.clip is not None:
                    torch.nn.utils.clip_grad_norm_(self.model.parameters(),
                                                   self.clip)
                self.optimizer.step()
            avg_train_loss = avg_train_loss / len(train_loader)
            avg_val_loss = self.calc_val_loss(val_loader)

            if avg_val_loss < best_val_loss:
                self.save_epoch(epoch)
                best_val_loss = avg_val_loss
                cur_patience = 0
            else:
                cur_patience += 1
            self.print_epoch(epoch, avg_train_loss, avg_val_loss,
                             cur_patience, strt)

    def fit(self, train_loader, val_loader, epochs):
        self.train_epochs(epochs, train_loader, val_loader)
```

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 29, 2020, 1:57pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/4 "2020-09-29T13:57:50Z")

</div>

That’s a generic PyTorch function in that case. We’ve implemented our `Trainer` class to help users easily train their model and leverage GPUs/TPUs and we’re happy to help if you encounter some bugs with it but if you want to build your own loop, you should try the PyTorch forum.

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [October 21, 2020, 2:21pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/5 "2020-10-21T14:21:43Z")

</div>

@sgugger I am using Trainer classes but not seeing any major speedup in training if I use a multi-GPU setup. In nvidia-smi and the W&B dashboard, I can see that both GPUs are being used. I then launched the training script on a single-GPU for comparison. The training commands are exactly the same on both machines.

I do not see any significant speedup in training. The training lasts for hours, I didn’t wait till the end, but tqdm estimates are pretty much the same on both machines. The progress should be reflected properly in tqdm, right? Any suggestions for further diagnosis?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 21, 2020, 2:32pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/6 "2020-10-21T14:32:48Z")

</div>

2 GPUs don’t bring a lot of speedup compared to one since you add all those synchronization operations. The main speedup is that you should have double the batch size automatically so less iterations (unless you used `max_steps` in your command, as always, it’s hard to help you on what’s wrong without seeing the code your un).

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [October 21, 2020, 4:35pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/7 "2020-10-21T16:35:28Z")

</div>

Riiight, excellent. I used max\_steps and didn’t notice the epoch is double for 2 GPUs. Awesome.

While we are at it. I read Thomas’ [article](https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255) about GPU training where he advocates computing the loss **in a parallel** fashion. But I don’t see support for it in the trainer. It wasn’t worth it?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 21, 2020, 4:48pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/8 "2020-10-21T16:48:45Z")

</div>

The `Trainer` lets you compute the loss how you want by subclassing and overriding `compute_loss` (see an example [here](https://huggingface.co/transformers/main_classes/trainer.html)). By default we use the basic loss since that’s the use case of most users.

---

<div class="post-metadata">

**Author:** ![aliosk](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@aliosk](https://discuss.huggingface.co/u/aliosk)\
**Post date:** [March 22, 2021, 4:05pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/10 "2021-03-22T16:05:14Z")

</div>

@sgugger I am trying to test multi-gpu training with the HF Trainer but for training a third party pytorch model. I have already overridden the compute\_loss and the Trainer.train() runs without a problem on single GPU machines. On a 4-GPU EC2 machine I get the following error:

TrainerCallback  
0%| | 0/20000 [00:00\<?, ?it/s]Traceback (most recent call last):  
File “train\_hf\_mlm\_encoder\_single\_gpu.py”, line 222, in   
main(params\_dict)  
File “train\_hf\_mlm\_encoder\_single\_gpu.py”, line 218, in main  
custom\_trainer.train()  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/transformers/trainer.py”, line 1053, in train  
tr\_loss += self.training\_step(model, inputs)  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/transformers/trainer.py”, line 1443, in training\_step  
loss = self.compute\_loss(model, inputs)  
File “/home/a204311-DataScientist/projects/trlabs\_routing\_transformer/routing\_sum/mlm\_pretrain/train\_and\_eval.py”, line 121, in compute\_loss  
return\_loss=True  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in **call**  
result = self.forward(\*input, \*\*kwargs)  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/torch/nn/parallel/data\_parallel.py”, line 154, in forward  
replicas = self.replicate(self.module, self.device\_ids[:len(inputs)])  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/torch/nn/parallel/data\_parallel.py”, line 159, in replicate  
return replicate(module, device\_ids, not torch.is\_grad\_enabled())  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/torch/nn/parallel/replicate.py”, line 104, in replicate  
buffer\_copies\_not\_rg = \_broadcast\_coalesced\_reshape(buffers\_not\_rg, devices, detach=True)  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/torch/nn/parallel/replicate.py”, line 68, in \_broadcast\_coalesced\_reshape  
return comm.broadcast\_coalesced(tensors, devices)  
File “/home/a204311-DataScientist/anaconda3/envs/routing/lib/python3.6/site-packages/torch/cuda/comm.py”, line 39, in broadcast\_coalesced  
return torch.\_C.\_broadcast\_coalesced(tensors, devices, buffer\_size)  
RuntimeError: Unconvertible NCCL type  
0%|

Any hints what may be causing this? I was under the impression that multi-GPU training should work out of the box with the Huggingface Trainer. Thank you for your help.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [March 22, 2021, 5:43pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/11 "2021-03-22T17:43:45Z")

</div>

> [@aliosk](#):
>
> I am trying to test multi-gpu training with the HF Trainer but for training a third party pytorch model.

Since you are not providing the code of that model, I can’t really help you with what is going wrong.

---

<div class="post-metadata">

**Author:** ![aliosk](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@aliosk](https://discuss.huggingface.co/u/aliosk)\
**Post date:** [March 22, 2021, 6:22pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/12 "2021-03-22T18:22:32Z")

</div>

@sgugger The model is the routing transformer language model (RoutingTransformerLM). The source code is here:

> **[lucidrains/routing-transformer](https://github.com/lucidrains/routing-transformer)**
>
> Fully featured implementation of Routing Transformer

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [March 22, 2021, 6:30pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/13 "2021-03-22T18:30:28Z")

</div>

I would guess that this model does not run on multiple GPUs if your training runs fine on one GPU. I’m afraid you will have to ask on GitHub to the author of that library.

---

<div class="post-metadata">

**Author:** ![Sahajtomar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sahajtomar/32/906_2.png) [@Sahajtomar](https://discuss.huggingface.co/u/Sahajtomar)\
**Post date:** [May 4, 2021, 4:13pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/14 "2021-05-04T16:13:18Z")

</div>

I am fine tuning T5 model on sagemaker with 4 gpu, just one gpu is being used. What do you suggest

---

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\
**Post date:** [May 6, 2021, 10:52am UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/15 "2021-05-06T10:52:28Z")

</div>

Hi @Sahajtomar thanks for reaching on the forum!

- Are you using your own for loop or the HF Trainer?
- Did you look at the T5 Distributed training example released few weeks ago? [Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker](https://huggingface.co/blog/sagemaker-distributed-training-seq2seq)

---

<div class="post-metadata">

**Author:** ![Sahajtomar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sahajtomar/32/906_2.png) [@Sahajtomar](https://discuss.huggingface.co/u/Sahajtomar)\
**Post date:** [May 7, 2021, 1:47pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/16 "2021-05-07T13:47:42Z")

</div>

Hi @OlivierCR , I am using distributed training , somehow model training does not start at all. I am using t5 large on 8 GPUs.

---

<div class="post-metadata">

**Author:** ![ashish-ram](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ashish-ram/32/19497_2.png) [@ashish-ram](https://discuss.huggingface.co/u/ashish-ram)\
**Post date:** [September 14, 2023, 3:08pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/17 "2023-09-14T15:08:25Z")

</div>

Hi @sgugger,

Is there is any special parameter that needs to be passed to the Trainer class to work with multi-GPU?  
Please have a look at [Not able to scale Trainer code to single node multi GPU - 🤗Transformers - Hugging Face Forums](https://discuss.huggingface.co/t/not-able-to-scale-trainer-code-to-single-node-multi-gpu/55025)

---

<div class="post-metadata">

**Author:** ![Koriy](https://avatars.discourse-cdn.com/v4/letter/k/7993a0/32.png) [@Koriy](https://discuss.huggingface.co/u/Koriy)\
**Post date:** [December 11, 2023, 9:59am UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/18 "2023-12-11T09:59:29Z")

</div>

Hi I’m confused about how to use Trainer class on 2 gpus. Are there any arguments I should set? And how to define model.to(device)?

---

<div class="post-metadata">

**Author:** ![Iamexperimenting](https://avatars.discourse-cdn.com/v4/letter/i/839c29/32.png) [@Iamexperimenting](https://discuss.huggingface.co/u/Iamexperimenting)\
**Post date:** [February 25, 2024, 2:20pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/19 "2024-02-25T14:20:12Z")

</div>

@sgugger hi, I’m trying to fine tune “meta-llama/Llama-2-7b” model in Kaggle notebook with (GPU T4 X 2), I’m noticing only one GPU is being used.

```auto
Sun Feb 25 14:06:28 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 76C P8 11W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 Tesla T4 Off | 00000000:00:05.0 Off | 0 |
| N/A 77C P8 12W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

```

```auto

import pandas as pd
import torch
from datasets import Dataset, load_dataset
from random import randrange
from peft import LoraConfig, get_peft_model, AutoPeftModelForCausalLM, prepare_model_for_int8_training
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments
from trl import SFTTrainer
import warnings
warnings.filterwarnings("ignore")
df = pd.read_csv("train.csv")
train = Dataset.from_pandas(df)
model_id = "meta-llama/Llama-2-7b"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

model = AutoModelForCausalLM.from_pretrained(model_id,
                                             load_in_4bit = True,
                                             torch_dtype = torch.float16,
                                             device_map="auto")
model.resize_token_embeddings(len(tokenizer))
model = prepare_model_for_int8_training(model)
peft_config = LoraConfig(
                          lora_alpha=16,
                          lora_dropout=0.1,
                          r=64,
                          bias="none",
                          task_type="CAUSAL_LM"
                        )
model = get_peft_model(model, peft_config)

args = TrainingArguments(
    output_dir='custom_domain_test',
    num_train_epochs=5,
    per_device_train_batch_size=8, 
    optim = "adamw_torch",
    logging_steps = 100,
    save_total_limit = 2,
    save_strategy = "no",
    load_best_model_at_end=False,
    learning_rate=2e-4,
    fp16=True,
    seed=42,
    warmup_ratio = 0.1,
    lr_scheduler_type = "linear",
    report_to="none",
    dataloader_num_workers = 4
)

# Create the trainer
trainer = SFTTrainer(
    model=model,
    train_dataset=train,
    # eval_dataset=test,
    dataset_text_field='text',
    peft_config=peft_config,
    max_seq_length=1042,
    tokenizer=tokenizer,
    args=args,
    packing=True,
)

# train
trainer.train()

```

can you please tell me how to utilise both the GPU and increase GPU and CPU utilisation using HuggingFace Trainer?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [February 25, 2024, 8:31pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/20 "2024-02-25T20:31:23Z")

</div>

Regarding training models using multiple GPUs, refer to the Alignment Handbook which uses DeepSpeed ZeRO-3 to run training on multiple GPUs: [alignment-handbook/scripts at main · huggingface/alignment-handbook · GitHub](https://github.com/huggingface/alignment-handbook/tree/main/scripts).

This is handled using the Accelerate library as backend (which the Trainer uses). One needs to define a configuration as done here: [alignment-handbook/recipes/accelerate\_configs/deepspeed\_zero3.yaml at main · huggingface/alignment-handbook · GitHub](https://github.com/huggingface/alignment-handbook/blob/main/recipes/accelerate_configs/deepspeed_zero3.yaml), and then pass that when running the script.

---

<div class="post-metadata">

**Author:** ![Iamexperimenting](https://avatars.discourse-cdn.com/v4/letter/i/839c29/32.png) [@Iamexperimenting](https://discuss.huggingface.co/u/Iamexperimenting)\
**Post date:** [February 25, 2024, 9:33pm UTC](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279/21 "2024-02-25T21:33:49Z")

</div>

@sgugger In my case, I can see it is using only one GPU, I’m using Kaggle 2 T4 GPU.

[Next page](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279.md?page=2)