# Is it possible to evaluate generations/output while fine-tuning a LLM?

**URL:** <https://discuss.huggingface.co/t/is-it-possible-to-evaluate-generations-output-while-fine-tuning-a-llm/34316>\
**Category:** 🤗Transformers\
**Created:** [March 21, 2023, 5:44pm UTC](https://discuss.huggingface.co/t/is-it-possible-to-evaluate-generations-output-while-fine-tuning-a-llm/34316 "2023-03-21T17:44:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![agademic](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@agademic](https://discuss.huggingface.co/u/agademic)\
**Post date:** [March 21, 2023, 5:44pm UTC](https://discuss.huggingface.co/t/is-it-possible-to-evaluate-generations-output-while-fine-tuning-a-llm/34316/1 "2023-03-21T17:44:58Z")

</div>

Is it possible to run some prompts and generate outputs for these prompts during fine-tuning (with transformers.Trainer), e.g. on each eval step?

I’ve seen some predictions in the w&b report for this blog [here](https://huggingface.co/blog/trl-peft) but I am not sure whether these predictions were made by loading the respective checkpoints.

---

<div class="post-metadata">

**Author:** ![agademic](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@agademic](https://discuss.huggingface.co/u/agademic)\
**Post date:** [April 3, 2023, 1:43pm UTC](https://discuss.huggingface.co/t/is-it-possible-to-evaluate-generations-output-while-fine-tuning-a-llm/34316/2 "2023-04-03T13:43:02Z")

</div>

If someone stumbles upon the same question: [Here](https://huggingface.co/docs/transformers/main_classes/callback) is where to look to implement custom callbacks on evaluation.

---

<div class="post-metadata">

**Author:** ![braintrustdata](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/braintrustdata/32/20834_2.png) [@braintrustdata](https://discuss.huggingface.co/u/braintrustdata)\
**Post date:** [November 1, 2023, 10:07pm UTC](https://discuss.huggingface.co/t/is-it-possible-to-evaluate-generations-output-while-fine-tuning-a-llm/34316/3 "2023-11-01T22:07:12Z")

</div>

Late here, but Braintrust is a great tool / platform for evaluating your LLM. We have a simple library in Python/Typescript for running and logging evaluations so you can use our web UI to dig into the results.

It’s free to use @ [https://braintrustdata.com/](https://braintrustdata.com/)