# Difference in trainer.predict() and model.generate() for LM

**URL:** <https://discuss.huggingface.co/t/difference-in-trainer-predict-and-model-generate-for-lm/45662>\
**Category:** 🤗Transformers\
**Created:** [July 5, 2023, 1:15am UTC](https://discuss.huggingface.co/t/difference-in-trainer-predict-and-model-generate-for-lm/45662 "2023-07-05T01:15:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![nickzzZzz](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nickzzZzz](https://discuss.huggingface.co/u/nickzzZzz)\
**Post date:** [July 5, 2023, 1:15am UTC](https://discuss.huggingface.co/t/difference-in-trainer-predict-and-model-generate-for-lm/45662/1 "2023-07-05T01:15:21Z")

</div>

Hi, I am training llama with trainer. I woulld like to get generation on training data with trainer.predict(). I apply argmax to the raw predictions for decoding, which I assume should be equivalent to greedy decoding. But the decoded result is different from the model.generate() given the same input. Additionally, the trainer.predict() only generate one more word apart from repeating the input, while model.generate() produces much more words. So how does trainer.predict() actually work?

**related codes for decode trainer.predict()**

```
 def logit2seqGreedy(pred):
   return np.argmax(pred, axis=-1)

  for i in range(predictions.shape[0]):
    decode_pred=logit2seqGreedy(predictions[i])
    pred_tokens = np.where(decode_pred != 0, decode_pred, tokenizer.pad_token_id).reshape(1,-1)
    pred_tokens = tokenizer.batch_decode(pred_tokens, skip_special_tokens=True)

```

**codes for model.generate()**

from transformers import GenerationConfig  
generation\_config = GenerationConfig(  
temperature=0,  
top\_p=1,  
num\_beams=1,  
)  
load\_pretrained=0  
if load\_pretrained:  
model=LLaMAForCausalLM.from\_pretrained(output)

def tokenize(prompt, add\_eos\_token=True):  
result = tokenizer(prompt, return\_tensors=“pt”)  
for i in range(len(prompt)):  
if add\_eos\_token and result[“input\_ids”][i][-1] != tokenizer.eos\_token\_id:  
result[“input\_ids”][i].append(tokenizer.eos\_token\_id)  
result[“attention\_mask”][i].append(1)  
return result

def evaluate(prompt):  
#inputs = tokenizer(prompt, return\_tensors=“pt”)  
inputs=tokenize([prompt],0)  
input\_ids = torch.tensor(inputs[“input\_ids”])

```
generation_output = model.generate(
    input_ids=input_ids,
    generation_config=generation_config,
    return_dict_in_generate=True,
    output_scores=True,
    max_new_tokens=150
)

for s in generation_output.sequences:
    output = tokenizer.decode(s)
    return output

```

**Data input template for trainer.predict() and model.generate()**

```
def getInps_eval(d):
     return f"""### Input:
        {d['que']} ### Response: The answer is ans"""

```