# TFT5ForConditionalGeneration generate returns empty output\_scores

**URL:** <https://discuss.huggingface.co/t/tft5forconditionalgeneration-generate-returns-empty-output-scores/67561>\
**Category:** 🤗Transformers\
**Created:** [December 31, 2023, 3:32pm UTC](https://discuss.huggingface.co/t/tft5forconditionalgeneration-generate-returns-empty-output-scores/67561 "2023-12-31T15:32:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ayalaall](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@ayalaall](https://discuss.huggingface.co/u/ayalaall)\
**Post date:** [December 31, 2023, 3:32pm UTC](https://discuss.huggingface.co/t/tft5forconditionalgeneration-generate-returns-empty-output-scores/67561/1 "2023-12-31T15:32:40Z")

</div>

Hi everyone,  
I’m fine-tuning the TFT5ForConditionalGeneration model (“t5-small”). Before doing model.fit() and save the model I set the output\_score=True as well as the relevant parameters in the generation config. After fine-tuning the model I load it and I can see that in the loaded model the `config.output_scores = True` as well as the `generation_config.output_scores=True`.

However, when I generate the text, the `scores` returns empty.  
How can I fix this please? I read documentation [https://huggingface.co/docs/transformers/internal/generation\_utils#transformers.generation.TFSampleEncoderDecoderOutput](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.generation.TFSampleEncoderDecoderOutput), [https://huggingface.co/docs/transformers/v4.36.1/en/main\_classes/text\_generation#transformers.TFGenerationMixin](https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/text_generation#transformers.TFGenerationMixin) but wasn’t able to figure out how to get the scores in the generated text.

I’m using transformers version 4.36.0

```auto
from transformers import TFAutoModelForSeq2SeqLM, GenerationConfig, AutoConfig

config = AutoConfig.from_pretrained("t5-small")
config.output_scores = True
transformer_model = TFAutoModelForSeq2SeqLM.from_pretrained("t5-small", config=config)

generation_config = GenerationConfig(**transformer_model.generation_config. __dict__ )
generation_config.max_new_tokens = 10
generation_config.return_dict_in_generate = True
generation_config.output_scores = True

transformer_model.save_pretrained(transformer_model_path)
generation_config.save_pretrained(transformer_model_path)

# text generation
input_ids = [[283, 1890, 6, 2733, 18, 715, 621, 29, 32, 106, 4804, 89, 17, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
generated_text = transformer_model.generate(input_ids, generation_config=generation_config, return_dict_in_generate=True, output_scores=True)

print(generated_text.scores) # returns empty tuple 

```

---

<div class="post-metadata">

**Author:** ![omgwenxx](https://avatars.discourse-cdn.com/v4/letter/o/4af34b/32.png) [@omgwenxx](https://discuss.huggingface.co/u/omgwenxx)\
**Post date:** [October 28, 2024, 10:15am UTC](https://discuss.huggingface.co/t/tft5forconditionalgeneration-generate-returns-empty-output-scores/67561/2 "2024-10-28T10:15:11Z")

</div>

Hi!

Don’t know if you are still struggling with this, I stumbled upon this post because my fine-tuned BLIP2 model was returning empty strings.

I fixed it when I realized that I had to set the same data type for my processor and model which in my case was `torch.bfloat16`

My code looks like this

```auto
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
base_model = "path/models/blip2-opt-2.7b"
model = Blip2ForConditionalGeneration.from_pretrained(base_model,torch_dtype=torch.bfloat16)
adapter_model = "path/finetuned_models/finetuned_blip2-opt-2.7b.pt"
peft_model = PeftModel.from_pretrained(model, model_id=adapter_model, device=device)
peft_model.to(device)
processor = AutoProcessor.from_pretrained(base_model)

# generating captions
image = dataset[0]["image"] # using the first dataset item for experimenting
# <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1166x1750>
inputs = processor(image, return_tensors="pt").to(device, torch.bfloat16)
generated_ids = peft_model.generate(**inputs, max_length=50)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0].strip()

```