# Multi-GPU Training using SFTTrainer

**URL:** <https://discuss.huggingface.co/t/multi-gpu-training-using-sfttrainer/141133>\
**Category:** 🤗Transformers\
**Created:** [February 16, 2025, 5:17pm UTC](https://discuss.huggingface.co/t/multi-gpu-training-using-sfttrainer/141133 "2025-02-16T17:17:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sai-santhosh](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@sai-santhosh](https://discuss.huggingface.co/u/sai-santhosh)\
**Post date:** [February 16, 2025, 5:17pm UTC](https://discuss.huggingface.co/t/multi-gpu-training-using-sfttrainer/141133/1 "2025-02-16T17:17:02Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/1/b17adf483038dad453e5a1f0d4d93f8609388240.png)

how do i use the yaml file in my code below, i want to use SFTTrainer i dont want to run it through scripts,is there a way to train my model on multi-gpu. Is there a way to pass the file to the train and automatically train on multi gpu

```auto
from datasets import load_dataset

dataset = load_dataset("Alok2304/Indian_Law_Final_Dataset",split="train[:30%]")
dataset

from transformers import BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    quantization_config = bnb_config,
    trust_remote_code = True).to(device)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B")

from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules= ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
)
model = get_peft_model(model, lora_config)

from trl import setup_chat_format
model,tokenizer = setup_chat_format(model,tokenizer)

from trl import SFTConfig, SFTTrainer

args = SFTConfig(
    output_dir = "lora_model/",
    per_device_train_batch_size = 8,
    per_device_eval_batch_size = 8,
    learning_rate = 2e-05,
    gradient_accumulation_steps = 2,
    max_steps = 300,
    logging_strategy = "steps",
    logging_steps = 25,
    save_strategy = "steps",
    save_steps = 25,
    eval_strategy = "steps",
    eval_steps = 25,
    fp16 = True,
    data_seed=42,
    max_seq_length = 2048,
    gradient_checkpointing=True,
    report_to = "none",
)

trainer = SFTTrainer(
    model = model,
    args = args,
    processing_class = tokenizer,
    train_dataset = dataset['train'],
    eval_dataset = dataset['test'],)

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 17, 2025, 6:15am UTC](https://discuss.huggingface.co/t/multi-gpu-training-using-sfttrainer/141133/2 "2025-02-17T06:15:25Z")

</div>

How about like this?

```py
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    quantization_config = bnb_config,
    device_map = "auto", # Have the accelerate library automatically distribute GPU resources
    trust_remote_code = True)#.to(device) # this ".to()" sends the model to cuda:0

```

---

<div class="post-metadata">

**Author:** ![sai-santhosh](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@sai-santhosh](https://discuss.huggingface.co/u/sai-santhosh)\
**Post date:** [February 17, 2025, 7:36am UTC](https://discuss.huggingface.co/t/multi-gpu-training-using-sfttrainer/141133/3 "2025-02-17T07:36:24Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/0/b035b416523af0a549f9e21059a14296524b7333.png)

---

<div class="post-metadata">

**Author:** ![nitishpandey04](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nitishpandey04/32/33204_2.png) [@nitishpandey04](https://discuss.huggingface.co/u/nitishpandey04)\
**Post date:** [February 17, 2025, 7:54am UTC](https://discuss.huggingface.co/t/multi-gpu-training-using-sfttrainer/141133/4 "2025-02-17T07:54:44Z")

</div>

I believe if you configure accelerate using `accelerate config` and then start the training run using `accelerate run code.py` then the configuration yaml file is picked from default location and starts distributed training.

Check here - [Quicktour](https://huggingface.co/docs/accelerate/en/quicktour)