# Token classification example script metrics improve despite overfit

**URL:** <https://discuss.huggingface.co/t/token-classification-example-script-metrics-improve-despite-overfit/11161>\
**Category:** 🤗Transformers\
**Created:** [October 28, 2021, 10:14am UTC](https://discuss.huggingface.co/t/token-classification-example-script-metrics-improve-despite-overfit/11161 "2021-10-28T10:14:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![OliverBe](https://avatars.discourse-cdn.com/v4/letter/o/f9ae1b/32.png) [@OliverBe](https://discuss.huggingface.co/u/OliverBe)\
**Post date:** [October 28, 2021, 10:14am UTC](https://discuss.huggingface.co/t/token-classification-example-script-metrics-improve-despite-overfit/11161/1 "2021-10-28T10:14:17Z")

</div>

Hello everyone,

I’d like to compare different models on various NER tasks. I found an example script for token classification here ([notebooks/token\_classification.ipynb at main · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/master/examples/token_classification.ipynb))

This script, however, appears to compute precision/recall/F1 using the training dataset rather than the validation dataset.

Starting with epoch 3, the validation loss increases (overfit?), but the scores continue to improve. The bert-base-cased model trained on the conll2003 dataset looks like this:  
 ![BERT-Base-Overfit-CONLL-metrics](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/a2f4eac49915aaa11e7943375847ea28c031ddb4.jpeg)

Trainer.evaluate() also appears to select the training dataset rather than the evaluation dataset. How can I adjust the settings to calculate the metrics on the evaluation dataset?

Best,  
Oliver