# Distributed training large models on cloud resources **URL:** \ **Category:** Beginners\ **Created:** [March 20, 2024, 5:22pm UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307 "2024-03-20T17:22:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![rmiller3](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@rmiller3](https://discuss.huggingface.co/u/rmiller3)\ **Post date:** [March 20, 2024, 5:22pm UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307/1 "2024-03-20T17:22:29Z")
I’m trying to fine tune mosaicml/mpt-7b-instruct using cloud resources ([modal.com](http://modal.com)) with mutliple GPUs but still running out of memory even on the large GPUs. I’m using the huggingface trainer to try this and confirmed the training\_args resolved n\_gpu to 2 or 4 depending on how many I configure modal to have. Can someone help me understand if separate scripts like torchrun (or maybe accelerate launch…) are required? Some information I read seems to conflict. For example, the official docs show using torchrun (and run\_clm for other forms): [https://huggingface.co/docs/transformers/en/perf\_train\_gpu\_many](https://huggingface.co/docs/transformers/en/perf_train_gpu_many) This stackoverflow seems to agree: [machine learning - How to use Huggingface Trainer with multiple GPUs? - Stack Overflow](https://stackoverflow.com/questions/75814047/how-to-use-huggingface-trainer-with-multiple-gpus) * * * However, these forum posts imply that the trainer should work 'out of the box' if n\_gpu is proper: \> https://discuss.huggingface.co/t/finetuning-gpt2-using-multiple-gpu-and-trainer/796/10 This says someone was surprised that their process was forked: [How to restrict training to one GPU if multiple are available, co - #3 by dropout05](https://discuss.huggingface.co/t/how-to-restrict-training-to-one-gpu-if-multiple-are-available-co/1244/3) * * * As a separate question maybe warranting a separate thread later, what should the torchrun training script do? Load the model, preprocess the dataset and call train? No special parameters to tell it to run on multiple machines? * * * Is there any way torchrun can be encapsulated in a single training call with parameters to split the load across all GPU? Note: I’m running this on a cloud service ([modal.com](http://modal.com)) so I’m assuming it’s not running on multiple GPUs but I can’t really see the GPU load as far as I know. ---
**Author:** ![rmiller3](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@rmiller3](https://discuss.huggingface.co/u/rmiller3)\ **Post date:** [March 20, 2024, 7:37pm UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307/2 "2024-03-20T19:37:00Z")
Perhaps the seemingly conflicting information comes from the definition of distributed (distributed across multiple nodes / machines vs distributed across GPUs). ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [March 20, 2024, 7:46pm UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307/3 "2024-03-20T19:46:19Z")
> [@rmiller3](#): > > Can someone help me understand if separate scripts like torchrun (or maybe accelerate launch…) are required? Some information I read seems to conflict. Have you configured something like DeepSpeed or FSDP? Native DDP (what it’s doing rn) still loads all of the model on each GPU. You want a sharding mechanism that will split the model up onto each GPU during training ---
**Author:** ![rmiller3](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@rmiller3](https://discuss.huggingface.co/u/rmiller3)\ **Post date:** [March 20, 2024, 8:34pm UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307/4 "2024-03-20T20:34:44Z")
Thanks, I’ll look into those now. I hadn’t. ---
**Author:** ![singhjagpreet](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@singhjagpreet](https://discuss.huggingface.co/u/singhjagpreet)\ **Post date:** [March 27, 2024, 7:28am UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307/5 "2024-03-27T07:28:41Z")
Have you tried accelerate > **[Accelerate](https://huggingface.co/docs/accelerate/en/index)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![rmiller3](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@rmiller3](https://discuss.huggingface.co/u/rmiller3)\ **Post date:** [March 27, 2024, 6:38pm UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307/6 "2024-03-27T18:38:50Z")
I got it working from a tutorial on [modal.com](http://modal.com) which uses accelerate. Working backwards off the github mentioned here. [Fine-tune an LLM in minutes (ft. Llama 2, CodeLlama, Mistral, etc.) | Modal Docs](https://modal.com/docs/examples/llm-finetuning) mosaicml/mpt-7b had some hiccups since the default configs use peft and it was missing some function that needed it, but I was able to do some finetuning there. The tutorial uses accelerate, but has a deprecated code using torch.distributed which I’m also trying to get working. I’m not sure of the pros and cons of one vs the other yet (torch.distributed vs accelerate). ---
**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](https://discuss.huggingface.co/u/muellerzr)\ **Post date:** [March 27, 2024, 7:25pm UTC](https://discuss.huggingface.co/t/distributed-training-large-models-on-cloud-resources/78307/7 "2024-03-27T19:25:15Z")
> [@rmiller3](#): > > (torch.distributed vs accelerate) They are synonymous, accelerate just wraps around it 🙂 (For on GPUs)