# Trainer API for Model Parallelism on Multiple GPUs **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 4, 2023, 2:08pm UTC](https://discuss.huggingface.co/t/trainer-api-for-model-parallelism-on-multiple-gpus/49493 "2023-08-04T14:08:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![zcakzhu](https://avatars.discourse-cdn.com/v4/letter/z/e79b87/32.png) [@zcakzhu](https://discuss.huggingface.co/u/zcakzhu)\ **Post date:** [August 4, 2023, 2:08pm UTC](https://discuss.huggingface.co/t/trainer-api-for-model-parallelism-on-multiple-gpus/49493/1 "2023-08-04T14:08:34Z")
According to the [main page](https://huggingface.co/docs/transformers/main_classes/trainer) of the Trainer API, > “The API supports distributed training on multiple GPUs/TPUs, mixed precision through [NVIDIA Apex](https://github.com/NVIDIA/apex) and Native AMP for PyTorch.” It seems like a user does not have to configure anything when using the Trainer class for doing distributed training. However, I am not able to find which distribution strategy this Trainer API supports. From my understanding, the Trainer class automatically uses DDP when multiple GPUs are detected. And if a user wants to also distribute model parameters onto different GPUs, then we have to pass in configurations for FSDP or deepspeed. Can someone confirm that my understanding is correct or not please? ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [August 4, 2023, 2:22pm UTC](https://discuss.huggingface.co/t/trainer-api-for-model-parallelism-on-multiple-gpus/49493/2 "2023-08-04T14:22:31Z")
> [@zcakzhu](#): > > Can someone confirm that my understanding is correct or not please? It depends on how you launch the script. If you use `torch.distributed.launch` (or have `accelerate config` setup for multi-gpu) it’ll use DistributedDataParallism. To use model parallelism just launch with `python {myscript.py}` and it should pick up model parallism. (If you find it does not, or need some more assistance, let me know!) You can verify if so by checking if `trainer.args.parallel_mode` prints `ParallelMode.NOT_DISTRIBUTED`. ---
**Author:** ![zcakzhu](https://avatars.discourse-cdn.com/v4/letter/z/e79b87/32.png) [@zcakzhu](https://discuss.huggingface.co/u/zcakzhu)\ **Post date:** [August 4, 2023, 2:38pm UTC](https://discuss.huggingface.co/t/trainer-api-for-model-parallelism-on-multiple-gpus/49493/3 "2023-08-04T14:38:23Z")
Thanks for your reply! It is super helpful 🙂 It is great to know that by just running `python {myscript.py}` the class will use model parallelism. A follow-up question from me is, how is the Trainer’s model parallelism differ from Deepspeed and FSDP? Is there any documentation that I can read into to gain more knowledge of what is happening at the backend? Thanks a lot! ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [August 4, 2023, 4:09pm UTC](https://discuss.huggingface.co/t/trainer-api-for-model-parallelism-on-multiple-gpus/49493/4 "2023-08-04T16:09:04Z")
> [@zcakzhu](#): > > how is the Trainer’s model parallelism differ from Deepspeed and FSDP? Is there any documentation that I can read into to gain more knowledge of what is happening at the backend? It just uses raw pytorch model parallism at that point, which is the equivalent of doing `.to(device_num)`. here’s a good doc discussing it: [Model Parallelism — transformers 4.7.0 documentation](https://huggingface.co/transformers/v4.9.2/parallelism.html) ---
**Author:** ![hansekbrand](https://avatars.discourse-cdn.com/v4/letter/h/9de053/32.png) [@hansekbrand](https://discuss.huggingface.co/u/hansekbrand)\ **Post date:** [August 5, 2023, 12:53pm UTC](https://discuss.huggingface.co/t/trainer-api-for-model-parallelism-on-multiple-gpus/49493/5 "2023-08-05T12:53:50Z")
is it enough to install deepspeed (`pip3 install deepspeed`) and run `accelerate config` and say `yes` when asked about `deepspeed`, to get `deepspeed` with `Trainer()`? ---
**Author:** ![yjoonjang](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yjoonjang/32/31457_2.png) [@yjoonjang](https://discuss.huggingface.co/u/yjoonjang)\ **Post date:** [September 10, 2024, 12:12pm UTC](https://discuss.huggingface.co/t/trainer-api-for-model-parallelism-on-multiple-gpus/49493/6 "2024-09-10T12:12:55Z")
Hello, @muellerzr I would like to see the inner process of how huggingface Trainer activates **model parallelism** by just giving `is_model_parallel` argument to True. I checked that this argument makes `place_model_on_device`=False in [transformers/src/transformers/trainer.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L531). However, I don’t get what happens in the next step. Are there some codes representing the division of model parameters ? It would be great if I could get your help. Thank you.