# Custom evaluation during Llama2 fine tuning

**URL:** <https://discuss.huggingface.co/t/custom-evaluation-during-llama2-fine-tuning/51964>\
**Category:** Beginners\
**Created:** [August 24, 2023, 11:55am UTC](https://discuss.huggingface.co/t/custom-evaluation-during-llama2-fine-tuning/51964 "2023-08-24T11:55:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ddalakoti](https://avatars.discourse-cdn.com/v4/letter/d/898d66/32.png) [@ddalakoti](https://discuss.huggingface.co/u/ddalakoti)\
**Post date:** [August 24, 2023, 11:55am UTC](https://discuss.huggingface.co/t/custom-evaluation-during-llama2-fine-tuning/51964/1 "2023-08-24T11:55:34Z")

</div>

Hi,

I am fine-tuning Llama2 for my particular use case. For this purpose, during evaluation, I want to compute model’s performance on my downstream task. For this I need to simulate `model.generate` as I would use during inference. From browsing the documentation, I can see that one approach is to either create a callback. So this would be something like (pseudo code):

```auto
class MyCallBack(TrainerCallBack):
   def on_evaluate(self, args, state, model, tokenizer):
         tokens = tokenizer("text")
         generated_text = model.generate(tokens["input_ids"], tokens["attention_mask"]

```

I have a few questions regarding this approach:

1. Do I need to do any special treatment to the model before calling generate, i.e. do I need to call `model.eval()` so that gradients are not computed unnecessarily?
2. If I have loaded the model in quantized mode, do I need to take care of this when using `model.generate`?
3. Can I override model’s default generation config here without affecting training?

Another solution which could be helpful is to use the `trainer.predict` which is provided in the `trainer` API. However, I am not sure if this is same as calling `model.generate` i.e does it generate next token based on model’s generated tokens or based on the correct token from input?

---

<div class="post-metadata">

**Author:** ![span11](https://avatars.discourse-cdn.com/v4/letter/s/f4b2a3/32.png) [@span11](https://discuss.huggingface.co/u/span11)\
**Post date:** [January 17, 2024, 11:41pm UTC](https://discuss.huggingface.co/t/custom-evaluation-during-llama2-fine-tuning/51964/2 "2024-01-17T23:41:35Z")

</div>

Hi, I am also trying to customize evolution for the downstream task. I wonder did you figure out what would be the best approach to do so?