# Regarding the argument \`per\_device\_train\_batch\_size\`

**URL:** <https://discuss.huggingface.co/t/regarding-the-argument-per-device-train-batch-size/95065>\
**Category:** Beginners\
**Created:** [July 2, 2024, 4:16am UTC](https://discuss.huggingface.co/t/regarding-the-argument-per-device-train-batch-size/95065 "2024-07-02T04:16:08Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![zy113](https://avatars.discourse-cdn.com/v4/letter/z/278dde/32.png) [@zy113](https://discuss.huggingface.co/u/zy113)\
**Post date:** [July 2, 2024, 4:16am UTC](https://discuss.huggingface.co/t/regarding-the-argument-per-device-train-batch-size/95065/1 "2024-07-02T04:16:08Z")

</div>

Regarding the argument `per_device_train_batch_size` in SFTTrainer, my question is, is it named like this because by fault, if I have more than 1 gpu, the SFTTrainer will do data parallelization (as opposed to sharding one single model and distribute it onto multiple gpus), so let’s say if I have n gpus and `per_device_train_batch_size=4`, then the total data used per round of backprop is 4n? But if this is the case the memory occupied on each gpu should be the same regardless of how many gpus I have, but this is not what I observed…