# Image to text using blip2 gives incorrect answer

**URL:** <https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177>\
**Category:** Beginners\
**Created:** [September 7, 2025, 3:31pm UTC](https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177 "2025-09-07T15:31:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rxshah](https://avatars.discourse-cdn.com/v4/letter/r/a587f6/32.png) [@rxshah](https://discuss.huggingface.co/u/rxshah)\
**Post date:** [September 7, 2025, 3:31pm UTC](https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177/1 "2025-09-07T15:31:05Z")

</div>

Here is code snippet slightly modified from blip2 site:

first prompt “Question: How many cats are there? Answer:” –\> gives correct answer Two

However, second prompt “Question: How many dogs are there? Answer:” –\> gives incorrect answer - Two should be Zero or None.

Is this because the accuracy of the trained model is not 100% we should get incorrect answers? OR AM I doing something incorrectly?

Here is the complete code:

from PIL import Image  
import requests  
from transformers import Blip2Processor, Blip2ForConditionalGeneration  
import torch

device = “cuda” if torch.cuda.is\_available() else “cpu”

processor = Blip2Processor.from\_pretrained(“Salesforce/blip2-opt-2.7b”)  
model = Blip2ForConditionalGeneration.from\_pretrained(  
“Salesforce/blip2-opt-2.7b”, torch\_dtype=torch.float16  
)  
model.to(device)

url = “[http://images.cocodataset.org/val2017/000000039769.jpg”](http://images.cocodataset.org/val2017/000000039769.jpg%E2%80%9D)  
image = Image.open(requests.get(url, stream=True).raw)

prompt = “Question: How many cats are there? Answer:”  
inputs = processor(images=image, text=prompt, return\_tensors=“pt”).to(  
device, torch.float16  
)

outputs = model.generate(\*\*inputs)

text = processor.tokenizer.batch\_decode(outputs, skip\_special\_tokens=True)  
print(text)

Gives correct answer: [‘Question: How many cats are there? Answer: Two\n’]

However, when I change prompt to

prompt2 = "Question: How many dogs are there? Answer: "

inputs2 = processor(images=image, text=prompt2, return\_tensors=“pt”).to(  
device, torch.float16  
)

outputs2 = model.generate(\*\*inputs2)

text2 = processor.tokenizer.batch\_decode(outputs2, skip\_special\_tokens=True)  
print(text2)

[‘Question: How many dogs are there? Answer: Two\n’]

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 7, 2025, 8:48pm UTC](https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177/2 "2025-09-07T20:48:34Z")

</div>

> OR AM I doing something incorrectly?

There’s no problem with the code; [it seems to be a known issue with the model / architecture](https://arxiv.org/pdf/2403.01373). You might want to try [using some fine-tuned version](https://huggingface.co/Salesforce/blip2-opt-2.7b-coco).

---

<div class="post-metadata">

**Author:** ![rxshah](https://avatars.discourse-cdn.com/v4/letter/r/a587f6/32.png) [@rxshah](https://discuss.huggingface.co/u/rxshah)\
**Post date:** [September 8, 2025, 1:14am UTC](https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177/3 "2025-09-08T01:14:33Z")

</div>

Thanks!!

Tried the examples you pointed to. The number of dogs still gave Two. However, following the examples further got following results:

```auto
55.3% that image 0 is 'a photo of a cat'
44.7% that image 0 is 'a photo of a dog'

```

Perhaps this explains why the model cannot distinguish between cats, dogs or anything else?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 8, 2025, 3:51am UTC](https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177/4 "2025-09-08T03:51:52Z")

</div>

Yeah. For example, CLIP can perfectly classify dogs and cats, but [BLIP seems utterly unsuitable for classification](https://huggingface.co/datasets/John6666/forum1/blob/main/blip2_cats_dogs.md)…

---

<div class="post-metadata">

**Author:** ![rxshah](https://avatars.discourse-cdn.com/v4/letter/r/a587f6/32.png) [@rxshah](https://discuss.huggingface.co/u/rxshah)\
**Post date:** [September 8, 2025, 1:52pm UTC](https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177/5 "2025-09-08T13:52:59Z")

</div>

Thanks for the clear explanation!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [September 9, 2025, 1:53am UTC](https://discuss.huggingface.co/t/image-to-text-using-blip2-gives-incorrect-answer/168177/6 "2025-09-09T01:53:46Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.