# Per\_device\_train\_batch\_size in model parallelism

**URL:** <https://discuss.huggingface.co/t/per-device-train-batch-size-in-model-parallelism/149171>\
**Category:** Beginners\
**Created:** [April 7, 2025, 12:27am UTC](https://discuss.huggingface.co/t/per-device-train-batch-size-in-model-parallelism/149171 "2025-04-07T00:27:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![quocnguyen](https://avatars.discourse-cdn.com/v4/letter/q/3d9bf3/32.png) [@quocnguyen](https://discuss.huggingface.co/u/quocnguyen)\
**Post date:** [April 7, 2025, 12:27am UTC](https://discuss.huggingface.co/t/per-device-train-batch-size-in-model-parallelism/149171/1 "2025-04-07T00:27:47Z")

</div>

If I have two GPUs and use `device_map="auto"`, by default the model evenly between them, how does setting `per_device_train_batch_size` affect the effective batch size? Specifically, is the effective batch size equal to `per_device_train_batch_size`, or is it 2 x `per_device_train_batch_size`? Is there a way to explicitly see the effective batch size

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 7, 2025, 7:47am UTC](https://discuss.huggingface.co/t/per-device-train-batch-size-in-model-parallelism/149171/2 "2025-04-07T07:47:08Z")

</div>

I haven’t been able to find any materials that specifically mention the calculation formula or checking method, but I think this is probably correct.

> or is it 2 x `per_device_train_batch_size`

So maybe this one.

```auto
# if using gradient accumulation
effective_batch_size = per_device_train_batch_size x gradient_accumulation_steps x num_gpus
# else
effective_batch_size = per_device_train_batch_size x num_gpus

```

> **[Performance and Scalability](https://huggingface.co/docs/transformers/main/en/performance)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> **[Fine-Tuning MPT-7B: A Practical Guide](https://medium.com/@heyamit10/fine-tuning-mpt-7b-a-practical-guide-34b221da7d10)**
>
> With Complete Code

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [April 17, 2025, 11:34am UTC](https://discuss.huggingface.co/t/per-device-train-batch-size-in-model-parallelism/149171/3 "2025-04-17T11:34:18Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.