# Using Transformers with DistributedDataParallel — any examples? **URL:** \ **Category:** Intermediate\ **Created:** [October 14, 2021, 2:33pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775 "2021-10-14T14:33:51Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![treeofknowledge](https://avatars.discourse-cdn.com/v4/letter/t/2acd7d/32.png) [@treeofknowledge](https://discuss.huggingface.co/u/treeofknowledge)\ **Post date:** [October 14, 2021, 2:33pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/1 "2021-10-14T14:33:51Z")
Hi! I’ve been consulting this page: [https://huggingface.co/transformers/parallelism.html#data-parallel](https://huggingface.co/transformers/parallelism.html#data-parallel) that says using DDP with transformers is “almost trivial”. But is there an example available? Am I supposed to follow [https://pytorch.org/tutorials/intermediate/ddp\_tutorial.html](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) just as if I was working with a regular PyTorch model and its optimizer exposed (as opposed to having it abstracted via transformers.Trainer)? Also, I have some Dataset-related questions. I’ve written a custom dataset class that extends torch.Dataset. My dataset class yields samples from stored binary chunks with pre-shuffled pre-tokenized data (to maximize reading speed within a chunk). Therefore, I had to disable Trainer’s shuffling behavior by replacing RandomSampler with SequentialSampler within Trainer.\_get\_train\_sampler. Will this hack work with DDP? Would it work if I switched to another distributed backend, like deepspeed? Is there a better way to do this? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [October 14, 2021, 2:46pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/2 "2021-10-14T14:46:56Z")
You have examples using `Accelerate` which is our library for distributed training for all tasks in the [Transformers repo](https://github.com/huggingface/transformers/tree/master/examples/pytorch). As for your hack, you will need to use the distributed version of the SequentialSampler. You might be better off replacing the sampler for the training dataloader by `_get_eval_sampler` instead of `_get_train_sampler`. ---
**Author:** ![treeofknowledge](https://avatars.discourse-cdn.com/v4/letter/t/2acd7d/32.png) [@treeofknowledge](https://discuss.huggingface.co/u/treeofknowledge)\ **Post date:** [October 15, 2021, 12:45pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/3 "2021-10-15T12:45:33Z")
Introduction for the `Accelerate` library says I have to be willing to write a forward loop (forgoing Trainer). Is there a way for me to enable DDP training while continuing using Trainer? Replacing `_get_train_sampler` with `_get_eval_sampler` looks like a much more elegant solution, thank you! ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [October 15, 2021, 12:58pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/4 "2021-10-15T12:58:44Z")
I misunderstood your question then, but I thought you wanted an example with the model and optimizer exposed. That’s why I pointed you to Accelerate. ---
**Author:** ![treeofknowledge](https://avatars.discourse-cdn.com/v4/letter/t/2acd7d/32.png) [@treeofknowledge](https://discuss.huggingface.co/u/treeofknowledge)\ **Post date:** [October 15, 2021, 1:01pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/5 "2021-10-15T13:01:27Z")
Yup! I’d like to keep using Trainer, I’m sorry if I wasn’t clear enough. ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [October 15, 2021, 1:08pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/6 "2021-10-15T13:08:45Z")
Then you just need to properly launch your training script, see [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision). ---
**Author:** ![treeofknowledge](https://avatars.discourse-cdn.com/v4/letter/t/2acd7d/32.png) [@treeofknowledge](https://discuss.huggingface.co/u/treeofknowledge)\ **Post date:** [October 15, 2021, 1:50pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/7 "2021-10-15T13:50:07Z")
I had to create a `--local_rank` launch argument and pass its value to Trainer, and then it worked. Thanks! ---
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 17, 2022, 2:29pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/8 "2022-08-17T14:29:33Z")
can you share the command you ran? and summarize what you did please? 🙂 @treeofknowledge * * * this discussion is slightly incomplete imho. For example, usually we wrap the mdl in DDP to have this type of (distributed) data parallel type of thing to work. Did you 1. wrap the model in DDP? 2. change the args to trainer or trainer args in anyway? 3. wrap the optimizer in any distributed trainer (like cherry? cherry is a pytorch lib for things like this) 4. also, what about the init group that is usually needed? Thanks in advance made a real question of this here: - [python - How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? - Stack Overflow](https://stackoverflow.com/questions/73391230/how-to-run-an-end-to-end-example-of-distributed-data-parallel-with-hugging-face) - [How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)?](https://discuss.huggingface.co/t/how-to-run-an-end-to-end-example-of-distributed-data-parallel-with-hugging-faces-trainer-api-ideally-on-a-single-node-multiple-gpus/21750) ---
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 17, 2022, 2:42pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/9 "2022-08-17T14:42:12Z")
@sgugger this doc [Efficient Training on Multiple GPUs](https://huggingface.co/docs/transformers/perf_train_gpu_many#scalability-strategy) suggests that ZeRO might also be used in this case (where only the trainer api is needed/wanted). How would one use ZeRO for in that case and keep the trainer? Do you have any links/demos/notebooks? ---
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [August 17, 2022, 7:12pm UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/10 "2022-08-17T19:12:16Z")
In case that wasn’t clear this will do everything automatically: ```auto python -m torch.distributed.launch --nproc_per_node 2 ~/src/main_debug.py ``` see details: [How to run an end to end example of distributed data parallel with hugging face's trainer api (ideally on a single node multiple gpus)? - #4 by brando](https://discuss.huggingface.co/t/how-to-run-an-end-to-end-example-of-distributed-data-parallel-with-hugging-faces-trainer-api-ideally-on-a-single-node-multiple-gpus/21750/4) ---
**Author:** ![Colorful](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Colorful](https://discuss.huggingface.co/u/Colorful)\ **Post date:** [September 5, 2022, 3:49am UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/11 "2022-09-05T03:49:16Z")
LOL. I find most questions can be solved by carefully reading the documentation. Thanks for your help~ ---
**Author:** ![fahadh4ilyas](https://avatars.discourse-cdn.com/v4/letter/f/bc79bd/32.png) [@fahadh4ilyas](https://discuss.huggingface.co/u/fahadh4ilyas)\ **Post date:** [May 8, 2023, 6:46am UTC](https://discuss.huggingface.co/t/using-transformers-with-distributeddataparallel-any-examples/10775/12 "2023-05-08T06:46:08Z")
Hi @sgugger , I’m curios about how Trainer works. After I look at the script, I found that when saving model at checkpoint, the script didn’t use `local_rank` argument to make the script only saving model on first worker. But, the example from Pytorch [here](https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multinode.py) showing that saving model at checkpoint using parameter `local_rank`. Is it okay to do what the Trainer do?