# Distributed Training w/ Trainer **URL:** \ **Category:** 🤗Transformers\ **Created:** [July 7, 2021, 4:21pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151 "2021-07-07T16:21:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![josephgatto](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@josephgatto](https://discuss.huggingface.co/u/josephgatto)\ **Post date:** [July 7, 2021, 4:21pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/1 "2021-07-07T16:21:39Z")
Does anyone have an end-to-end example of how to do multi-gpu, multi-node distributed training using the trainer? I can’t seem to find one anywhere. ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [July 7, 2021, 5:52pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/2 "2021-07-07T17:52:14Z")
All the examples using the `Trainer` run in multi-gpu multi-node, you just have to use the PyTorch launcher to properly launch a multi-GPU multinode training. ---
**Author:** ![josephgatto](https://avatars.discourse-cdn.com/v4/letter/j/b38774/32.png) [@josephgatto](https://discuss.huggingface.co/u/josephgatto)\ **Post date:** [July 7, 2021, 6:03pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/3 "2021-07-07T18:03:40Z")
So is there no code adjustments that need to be made, only how the file is launched? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [July 7, 2021, 6:08pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/4 "2021-07-07T18:08:41Z")
Yes, the `Trainer` will deal with all the rest by itself. ---
**Author:** ![SSamDav](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ssamdav/32/29888_2.png) [@SSamDav](https://discuss.huggingface.co/u/SSamDav)\ **Post date:** [July 23, 2021, 8:33am UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/5 "2021-07-23T08:33:27Z")
Hi I’m trying to run a multi-node training using the Trainer class, for that I run my script with the `python -m torch.distributed.launch --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr="IP" \ --master_port=1234`, however, the script doesn’t wait for the master node. Also when I run in the master node the script doesn’t wait for the child node. Should I set up any env variable? The only thing that I’m doing is passing the local\_rank to the TrainingArgs. Thanks for the help! ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [July 26, 2021, 1:05pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/6 "2021-07-26T13:05:20Z")
It’s hard to know what the problem could be without seeing the script you are launching. ---
**Author:** ![Lee1990](https://avatars.discourse-cdn.com/v4/letter/l/4da419/32.png) [@Lee1990](https://discuss.huggingface.co/u/Lee1990)\ **Post date:** [April 24, 2022, 9:02am UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/7 "2022-04-24T09:02:37Z")
what do you mean by “use the PyTorch launcher to properly launch a multi-GPU multinode training” ? ---
**Author:** ![nlp](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nlp](https://discuss.huggingface.co/u/nlp)\ **Post date:** [May 31, 2022, 10:10am UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/8 "2022-05-31T10:10:45Z")
It doesnt work with Longformer. Is this expected ? ---
**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](https://discuss.huggingface.co/u/nbroad)\ **Post date:** [May 31, 2022, 1:31pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/9 "2022-05-31T13:31:53Z")
[https://pytorch.org/docs/stable/distributed.html#launch-utility](https://pytorch.org/docs/stable/distributed.html#launch-utility) ---
**Author:** ![keval-sha](https://avatars.discourse-cdn.com/v4/letter/k/c6cbf5/32.png) [@keval-sha](https://discuss.huggingface.co/u/keval-sha)\ **Post date:** [April 14, 2025, 8:27pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/10 "2025-04-14T20:27:41Z")
Are there any examples of running a distributed training job using pytorch on k8s, specifically GKE? ---
**Author:** ![theharshithh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/theharshithh/32/47921_2.png) [@theharshithh](https://discuss.huggingface.co/u/theharshithh)\ **Post date:** [May 20, 2025, 1:09pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/11 "2025-05-20T13:09:14Z")
Use raylib for distributed training. here is a guide put up by verl for mutli-node training in ray. > **[Multinode Training — verl documentation](https://verl.readthedocs.io/en/latest/start/multinode.html#set-up-multinode-ray-cluster)** ---
**Author:** ![dag332](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dag332/32/48741_2.png) [@dag332](https://discuss.huggingface.co/u/dag332)\ **Post date:** [June 3, 2025, 6:14pm UTC](https://discuss.huggingface.co/t/distributed-training-w-trainer/8151/12 "2025-06-03T18:14:56Z")
if you want to do multi-node, multi-GPU training make sure to use the `local_process_index` instead of the `process_index`. Otherwise the device count doesn’t account for the GPUs on the separate node ```auto from trl import SFTConfig, SFTTrainer from accelerate import PartialState device_string = PartialState().local_process_index model = AutoModelForCausalLM.from_pretrained( ... device_map={'':device_string} ) ```