# Evaluate model at saved checkpoint

**URL:** <https://discuss.huggingface.co/t/evaluate-model-at-saved-checkpoint/7068>\
**Category:** 🤗Transformers\
**Created:** [June 22, 2021, 2:53pm UTC](https://discuss.huggingface.co/t/evaluate-model-at-saved-checkpoint/7068 "2021-06-22T14:53:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MillyXXX](https://avatars.discourse-cdn.com/v4/letter/m/a9a28c/32.png) [@MillyXXX](https://discuss.huggingface.co/u/MillyXXX)\
**Post date:** [June 22, 2021, 2:53pm UTC](https://discuss.huggingface.co/t/evaluate-model-at-saved-checkpoint/7068/1 "2021-06-22T14:53:36Z")

</div>

Hi, I’m using Seq2seq Trainer for QG and trying to see sample output of model at a checkpoint like so:

model = ProphetNetForConditionalGeneration.from\_pretrained(‘squad\_training\_latest/results/checkpoint-212000’)  
…  
contexts = [tdata[‘context’][i]]  
answers = [tdata[‘answer’][i]]  
questions = [tdata[‘question’][i], tdata[‘question2’][i]]

```
    encoder_inputs, decoder_inputs = preprocess_batch(contexts, questions, answers)
    decoder_inputs = decoder_inputs.contiguous()

    question_ids = model.generate(encoder_inputs, early_stopping=False, return_dict_in_generate=False, eos_token_id=102, min_length = 64)
    rv = tokenizer.batch_decode(question_ids, skip_special_tokens=False)
    print(rv)

```

The result looks very bad, overfitted like [’[SEP] what was the the the the the name of the? [X\_SEP] the the? name of the?’] for every example

however the eval loss was low so wondering if I’m evaluating right, or should I use Trainer.predict? load the model a different way?

(side question: I noticed Trainer.evaluate() returns teacher forced predictions to me and it differs from model.generate() and was wondering why and whether the param predict\_with\_generate has anything to do with it)