# Hyperparameter tuning practical guide?

**URL:** <https://discuss.huggingface.co/t/hyperparameter-tuning-practical-guide/10297>\
**Category:** Beginners\
**Created:** [September 25, 2021, 10:08am UTC](https://discuss.huggingface.co/t/hyperparameter-tuning-practical-guide/10297 "2021-09-25T10:08:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![moma1820](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moma1820/32/4141_2.png) [@moma1820](https://discuss.huggingface.co/u/moma1820)\
**Post date:** [September 25, 2021, 10:08am UTC](https://discuss.huggingface.co/t/hyperparameter-tuning-practical-guide/10297/1 "2021-09-25T10:08:32Z")

</div>

Hi i have been having problems doing parameter tuning with google colab, where its alawys gpu that runs out of memory.

Is there any practical advice you could give me for tuning bert models? In terms of envoirment settings i need for example number of gpu so i don’t run out of mem

It is to be noted that when doing tuning with CPU it works but takes ages.

I am using trainer api with Optuna

---

<div class="post-metadata">

**Author:** ![Akshayextreme](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Akshayextreme](https://discuss.huggingface.co/u/Akshayextreme)\
**Post date:** [October 6, 2021, 8:03pm UTC](https://discuss.huggingface.co/t/hyperparameter-tuning-practical-guide/10297/2 "2021-10-06T20:03:43Z")

</div>

If your GPU can only take 16 as batch\_size then make sure that multiplication of batch\_size and gradient\_accumulation does not go beyond 16. You need to specify range for both these parameters such that any combination of elements from both ranges does not take the effective batch\_size beyond 16.