# Use Trainer API with two valiation sets

**URL:** <https://discuss.huggingface.co/t/use-trainer-api-with-two-valiation-sets/5212>\
**Category:** 🤗Transformers\
**Created:** [March 30, 2021, 12:41pm UTC](https://discuss.huggingface.co/t/use-trainer-api-with-two-valiation-sets/5212 "2021-03-30T12:41:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![simonschoe](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/simonschoe/32/20951_2.png) [@simonschoe](https://discuss.huggingface.co/u/simonschoe)\
**Post date:** [March 30, 2021, 12:41pm UTC](https://discuss.huggingface.co/t/use-trainer-api-with-two-valiation-sets/5212/1 "2021-03-30T12:41:05Z")

</div>

Hi everyone,  
right now the `Trainer` API accepts one `eval_dataset`. I am wondering, is it somehow possible to provide two different validation sets that are both evaluated during training? For example, I might want to track my validation loss on a validation set that was previously sampled from my training data and hence shares the same distribution and on a validation set that was sampled from a data set with a presumably different data distribution (e.g., stemming from a different period). The idea stems from [“Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks”](https://arxiv.org/abs/2004.10964).

Thanks in advance 🙂  
Simon

---

<div class="post-metadata">

**Author:** ![deathcrush](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@deathcrush](https://discuss.huggingface.co/u/deathcrush)\
**Post date:** [February 28, 2022, 12:11pm UTC](https://discuss.huggingface.co/t/use-trainer-api-with-two-valiation-sets/5212/2 "2022-02-28T12:11:55Z")

</div>

@patrickvonplaten this a common use case in research (including in mine). I don’t see any examples in the docs, but the docs ([Callbacks](https://huggingface.co/docs/transformers/master/main_classes/callback#trainercallback)) suggest that there should be a way to write a callback function to achieve this? Looking at the code very briefly, I imagine that a callback would simply call the `evaluate` again for all other `eval_dataset`s at the end of the validation loop, optionally changing `metric_key_prefix` so that the logger displays traces of the metrics measured on individual datasets separately?

I’d have to look in more detail at the APIs for implementing a callback to provide more details, but I’m thinking.

UPDATE: I had a look but I can’t see how I could pass the other datasets to the callback, so I assume this is not possible. To promote a productive discussion, I raised a feature request where I summarise the issue and a potential workaround [here.](https://github.com/huggingface/transformers/issues/15857)