# Running mT5 on multiple GPUs

**URL:** <https://discuss.huggingface.co/t/running-mt5-on-multiple-gpus/20895>\
**Category:** Models\
**Created:** [July 26, 2022, 7:49pm UTC](https://discuss.huggingface.co/t/running-mt5-on-multiple-gpus/20895 "2022-07-26T19:49:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![rafae015](https://avatars.discourse-cdn.com/v4/letter/r/77aa72/32.png) [@rafae015](https://discuss.huggingface.co/u/rafae015)\
**Post date:** [July 26, 2022, 7:49pm UTC](https://discuss.huggingface.co/t/running-mt5-on-multiple-gpus/20895/1 "2022-07-26T19:49:47Z")

</div>

Hi everyone,  
I am trying to finetune the Google mT5 XXL model for MTNT French-English data. I use 2 32GB GPUs and I am using accelerate package for the model finetuning. However, I am getting a “CUDA out of memory” error for both GPUs. It appears that the script first runs on GPU 0 and then tries running on GPU 1 and therefore give the memory error for both GPUs.  
Thank you.