# Clarifying multi-GPU memory usage

**URL:** <https://discuss.huggingface.co/t/clarifying-multi-gpu-memory-usage/1905>\
**Category:** Beginners\
**Created:** [November 5, 2020, 6:20pm UTC](https://discuss.huggingface.co/t/clarifying-multi-gpu-memory-usage/1905 "2020-11-05T18:20:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![deppen8](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/deppen8/32/516_2.png) [@deppen8](https://discuss.huggingface.co/u/deppen8)\
**Post date:** [November 5, 2020, 6:20pm UTC](https://discuss.huggingface.co/t/clarifying-multi-gpu-memory-usage/1905/1 "2020-11-05T18:20:01Z")

</div>

Am I reading this thread ([Training using multiple GPUs](https://discuss.huggingface.co/t/training-using-multiple-gpus/1279)) correctly? I interpret that to mean:

> Training a model with batch size 16 on one GPU is equivalent to running a model with batch size 4 on 4 GPUs

Is that correct? And does it differ between `DataParallel` and `DistributedDataParallel` modes?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [November 5, 2020, 10:22pm UTC](https://discuss.huggingface.co/t/clarifying-multi-gpu-memory-usage/1905/2 "2020-11-05T22:22:55Z")

</div>

It is correct. The difference between `DataParallel` and `DistributedDataParallel` is (in your current example):

- in `DataParallel` mode you have to set the batch size to 16 for your data loaders.
- in `DistributedDataParallel` you have to set the batch size to 4 for your data loaders.