# Batch size in trainer eval loop

**URL:** <https://discuss.huggingface.co/t/batch-size-in-trainer-eval-loop/17069>\
**Category:** DeepSpeed\
**Created:** [April 21, 2022, 11:15pm UTC](https://discuss.huggingface.co/t/batch-size-in-trainer-eval-loop/17069 "2022-04-21T23:15:18Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![arunwzd](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@arunwzd](https://discuss.huggingface.co/u/arunwzd)\
**Post date:** [April 21, 2022, 11:15pm UTC](https://discuss.huggingface.co/t/batch-size-in-trainer-eval-loop/17069/1 "2022-04-21T23:15:18Z")

</div>

I am new to huggingface trainer. I tried to use hf trainer on t5. It looks to me that the training phase uses all GPUs while in evaluation phase, I sometimes see `per_device_batch_size * num_gpus` as the batch size and in other times, I see only `per_device_batch_size` as batch size.  
Can someone shed light on if multiple gpus are used during evaluation and whats the effective batch size during evaluation?  
Also, what is the point of using `eval_grad_accumulation_steps` (argument to trainer) during eval? There is no model update based on eval dataset…right?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 22, 2022, 2:04pm UTC](https://discuss.huggingface.co/t/batch-size-in-trainer-eval-loop/17069/2 "2022-04-22T14:04:46Z")

</div>

The evaluation will use all GPUs like the training, so the effective batch size will be the `per_device_batch_size` multiplied by the number of GPUs (it’s logged at the beginning of the evaluation).

Where exactly did you find `eval_grad_accumulation_steps`, I don’t see this anywhere in the Transformers code base.

---

<div class="post-metadata">

**Author:** ![arunwzd](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@arunwzd](https://discuss.huggingface.co/u/arunwzd)\
**Post date:** [April 22, 2022, 2:22pm UTC](https://discuss.huggingface.co/t/batch-size-in-trainer-eval-loop/17069/3 "2022-04-22T14:22:48Z")

</div>

The exact `TrainingArgument` name is `eval_accumulation_steps`. I misunderstood that it is about gradient accumulation. I guess it is about maximum examples to predict before moving the predictions onto CPU. But then wondering, why is `eval_accumulation_steps` important? We could control the number of examples in GPU using `per_device_eval_batch_size`…right?

Thanks

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 22, 2022, 2:37pm UTC](https://discuss.huggingface.co/t/batch-size-in-trainer-eval-loop/17069/4 "2022-04-22T14:37:15Z")

</div>

No, the examples are only passed to the CPU every `eval_accumulation_steps` if provided, not at every step. And when this arument is left unset, they are passed to the CPU at the end of the whole evaluation loop. It’s way faster to only do this transfer once, and most of the time the predictions of the model don’t take much GPU space.