# Implement k-fold cross validation for hyperparameter tuning

**URL:** <https://discuss.huggingface.co/t/implement-k-fold-cross-validation-for-hyperparameter-tuning/18425>\
**Category:** Beginners\
**Created:** [May 27, 2022, 11:10am UTC](https://discuss.huggingface.co/t/implement-k-fold-cross-validation-for-hyperparameter-tuning/18425 "2022-05-27T11:10:25Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pascal541](https://avatars.discourse-cdn.com/v4/letter/p/c67d28/32.png) [@pascal541](https://discuss.huggingface.co/u/pascal541)\
**Post date:** [May 27, 2022, 11:10am UTC](https://discuss.huggingface.co/t/implement-k-fold-cross-validation-for-hyperparameter-tuning/18425/1 "2022-05-27T11:10:25Z")

</div>

I’m currently using the Trainer class to finetune a gpt-2 model for a regression task and I would like to tune the hyperparameters of this model.  
Since my dataset is relatively small (about 3500 samples), I wanted to use k-fold cross validation to do that. Unfortunately I couldn’t find this functionality in the documentation of the Transformers library (or any supported hyperparameter backend documentation).

It seems like I would have to override many methods in the Trainer class to achieve that. I’m not quite sure if this is a good idea, since I would also have to override methods like \_inner\_training\_loop().

Is there any way to accomplish this?

Thanks in advance.