# Trainer) training one batch with multiple GPUs

**URL:** <https://discuss.huggingface.co/t/trainer-training-one-batch-with-multiple-gpus/43722>\
**Category:** DeepSpeed\
**Created:** [June 19, 2023, 2:24am UTC](https://discuss.huggingface.co/t/trainer-training-one-batch-with-multiple-gpus/43722 "2023-06-19T02:24:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeesoo](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@jeesoo](https://discuss.huggingface.co/u/jeesoo)\
**Post date:** [June 19, 2023, 2:24am UTC](https://discuss.huggingface.co/t/trainer-training-one-batch-with-multiple-gpus/43722/1 "2023-06-19T02:24:29Z")

</div>

Hi, there.  
I’m using huggingFace Trainer code to train gpt-based large language model.  
The size is more than 8b.  
when I use input sequence length = 2048 tokens, and the per\_device\_train\_batch\_size=1, it seems it doesn’t fit on A100 (40GB) GPU. How can I load one batch to multiple gpus? It seems like that I ‘must’ load more than one batch on one gpu.