# How to run single-node, multi-GPU training with HF Trainer? **URL:** \ **Category:** 🤗Transformers\ **Created:** [June 23, 2022, 11:40am UTC](https://discuss.huggingface.co/t/how-to-run-single-node-multi-gpu-training-with-hf-trainer/19503 "2022-06-23T11:40:40Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\ **Post date:** [June 23, 2022, 11:40am UTC](https://discuss.huggingface.co/t/how-to-run-single-node-multi-gpu-training-with-hf-trainer/19503/1 "2022-06-23T11:40:41Z")
Hi, I want to train Trainer scripts on single-node, multi-GPU setting. Do I need to launch HF with a torch launcher (torch.distributed, torchX, torchrun, Ray Train, PTL etc) or can the HF Trainer alone use multiple GPUs without being launched by a third-party distributed launcher? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [June 23, 2022, 12:44pm UTC](https://discuss.huggingface.co/t/how-to-run-single-node-multi-gpu-training-with-hf-trainer/19503/2 "2022-06-23T12:44:09Z")
See the documentation on [running scripts](https://huggingface.co/docs/transformers/run_scripts#distributed-training-and-mixed-precision). 🙂 ---
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 17, 2022, 3:38pm UTC](https://discuss.huggingface.co/t/how-to-run-single-node-multi-gpu-training-with-hf-trainer/19503/3 "2022-08-17T15:38:10Z")
I think the docs are insufficient. See my questions here: [Using Transformers with DistributedDataParallel — any examples?](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775) ---
**Author:** ![dimichhf](https://avatars.discourse-cdn.com/v4/letter/d/f05b48/32.png) [@dimichhf](https://discuss.huggingface.co/u/dimichhf)\ **Post date:** [May 28, 2023, 8:56am UTC](https://discuss.huggingface.co/t/how-to-run-single-node-multi-gpu-training-with-hf-trainer/19503/4 "2023-05-28T08:56:19Z")
My impression with HF Trainer is HF has lots of video tutorials and none talks about multi GPU training using Trainer (assuming it is so simple) but the key element is lost in the docs, which is the command to run the trainer script which is really hard to find. So the easiest API is made hard by missing to mention this script, which I finally found in one of the forums > **[transformers/examples/pytorch at main · huggingface/transformers - Distributed training and mixed precision](https://github.com/huggingface/transformers/tree/main/examples/pytorch#distributed-training-and-mixed-precision)** > > main/examples/pytorch ---
**Author:** ![jpiabrantes](https://avatars.discourse-cdn.com/v4/letter/j/a88e57/32.png) [@jpiabrantes](https://discuss.huggingface.co/u/jpiabrantes)\ **Post date:** [May 7, 2024, 10:18am UTC](https://discuss.huggingface.co/t/how-to-run-single-node-multi-gpu-training-with-hf-trainer/19503/5 "2024-05-07T10:18:54Z")
I have a script that uses HF Trainer and works fine when I run it. But if I run the command for multi-gpu training `torchrun --nproc_per_node 4 my_script.py` I get an error: ```sh [rank1]: Traceback (most recent call last): [rank1]: File "/home/jpiabrantes/rosetta/fine_tune_coder.py", line 128, in [rank1]: main() [rank1]: File "/home/jpiabrantes/rosetta/fine_tune_coder.py", line 103, in main [rank1]: training_args = TrainingArguments( [rank1]: File "", line 127, in __init__ [rank1]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/training_args.py", line 1630, in __post_init__ [rank1]: and (self.device.type == "cpu" and not is_torch_greater_or_equal_than_2_3) [rank1]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/training_args.py", line 2131, in device [rank1]: return self._setup_devices [rank1]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/utils/generic.py", line 59, in __get__ [rank1]: cached = self.fget(obj) [rank1]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/training_args.py", line 2063, in _setup_devices [rank1]: self.distributed_state = PartialState( [rank1]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/accelerate/state.py", line 278, in __init__ [rank1]: self.set_device() [rank1]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/accelerate/state.py", line 786, in set_device [rank1]: torch.cuda.set_device(self.device) [rank1]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/torch/cuda/ __init__.py", line 399, in set_device [rank1]: torch._C._cuda_setDevice(device) [rank1]: RuntimeError: CUDA error: invalid device ordinal [rank1]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. [rank1]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1. [rank1]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. [rank2]: Traceback (most recent call last): [rank2]: File "/home/jpiabrantes/rosetta/fine_tune_coder.py", line 128, in [rank2]: main() [rank2]: File "/home/jpiabrantes/rosetta/fine_tune_coder.py", line 103, in main [rank2]: training_args = TrainingArguments( [rank2]: File "", line 127, in __init__ [rank2]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/training_args.py", line 1630, in __post_init__ [rank2]: and (self.device.type == "cpu" and not is_torch_greater_or_equal_than_2_3) [rank2]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/training_args.py", line 2131, in device [rank2]: return self._setup_devices [rank2]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/utils/generic.py", line 59, in __get__ [rank2]: cached = self.fget(obj) [rank2]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/transformers/training_args.py", line 2063, in _setup_devices [rank2]: self.distributed_state = PartialState( [rank2]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/accelerate/state.py", line 278, in __init__ [rank2]: self.set_device() [rank2]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/accelerate/state.py", line 786, in set_device [rank2]: torch.cuda.set_device(self.device) [rank2]: File "/home/jpiabrantes/rosetta/.venv/lib/python3.10/site-packages/torch/cuda/ __init__.py", line 399, in set_device [rank2]: torch._C._cuda_setDevice(device) [rank2]: RuntimeError: CUDA error: invalid device ordinal [rank2]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. [rank2]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1. [rank2]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` ---
**Author:** ![shadowshadow](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@shadowshadow](https://discuss.huggingface.co/u/shadowshadow)\ **Post date:** [October 16, 2024, 1:00am UTC](https://discuss.huggingface.co/t/how-to-run-single-node-multi-gpu-training-with-hf-trainer/19503/6 "2024-10-16T01:00:55Z")
hello, I also encounter this problem, when I load the model using `AutoModelForCausalLM.from_pretrained("xxx", device_map="auto")`. It will shard my model on each device. And I set `Trainer` as follows: ```python args = TrainingArguments( output_dir="./output", per_device_train_batch_size=2, gradient_accumulation_steps=8, logging_steps=10, num_train_epochs=1 ) trainer = Trainer( model=model, args=args, tokenizer=tokenizer, train_dataset=tokenized_ds, data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True) ) trainer.train() ``` It will report the same error like yours ☹