# Failed to increase batch size when using multi gpu

**URL:** <https://discuss.huggingface.co/t/failed-to-increase-batch-size-when-using-multi-gpu/35863>\
**Category:** 🤗Transformers\
**Created:** [April 7, 2023, 7:56am UTC](https://discuss.huggingface.co/t/failed-to-increase-batch-size-when-using-multi-gpu/35863 "2023-04-07T07:56:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomad01](https://avatars.discourse-cdn.com/v4/letter/t/dec6dc/32.png) [@tomad01](https://discuss.huggingface.co/u/tomad01)\
**Post date:** [April 7, 2023, 7:56am UTC](https://discuss.huggingface.co/t/failed-to-increase-batch-size-when-using-multi-gpu/35863/1 "2023-04-07T07:56:57Z")

</div>

Hi,

I have a machine with 8 Tesla V100.  
When I train a model with ORTTrainer (this also happens with Trainer api from transformers) api from optimum.onnxruntime on a single gpu (os.environ[“CUDA\_VISIBLE\_DEVICES”] = “0”  
) I’m able to use 32 batch size, however when I’m running a train on all gpus I get an oom error with bs=32. I was expecting to be able to increase the bs a lot with 8 gpus.  
Does the ORTTrainer api not handling well the parallelization ?  
I noticed one of the gpu has ~99% memory used while others have less, around 70%  
Do you have any suggestions for me?  
Thanks