# Blip-2 as a classification model

**URL:** <https://discuss.huggingface.co/t/blip-2-as-a-classification-model/103383>\
**Category:** Models\
**Created:** [August 21, 2024, 6:29pm UTC](https://discuss.huggingface.co/t/blip-2-as-a-classification-model/103383 "2024-08-21T18:29:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![atomwalk12](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@atomwalk12](https://discuss.huggingface.co/u/atomwalk12)\
**Post date:** [August 21, 2024, 6:29pm UTC](https://discuss.huggingface.co/t/blip-2-as-a-classification-model/103383/1 "2024-08-21T18:29:31Z")

</div>

I was wondering is it even possible to use the Blip-2 model (Blip2ForConditionalGeneration) for classification-like tasks. I have not been able to find any thorough information on how to use this model using a classification head.

Also, if the answer is yes, then which features should be extracted to train the classifier on. I can think of two possibilities:

1. Use the last\_hidden\_layer from the q-former and combine these features with the last\_hidden\_layer of the vision model; or
2. Use the pooled output of the q-former.

I feel like this is an interesting topic, which unfortunately was not able to find much information about.  
Any related tips would be really appreciated. Thanks!

---

<div class="post-metadata">

**Author:** ![AmlEsmail](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@AmlEsmail](https://discuss.huggingface.co/u/AmlEsmail)\
**Post date:** [November 23, 2025, 12:53am UTC](https://discuss.huggingface.co/t/blip-2-as-a-classification-model/103383/2 "2025-11-23T00:53:11Z")

</div>

Hey, any luck on that?

I use it for classifying if an image matches a text or not, I just take the embeddings from image and text, find the cosine similarity then use a threshold to see if it’s matching or not

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 23, 2025, 5:46am UTC](https://discuss.huggingface.co/t/blip-2-as-a-classification-model/103383/3 "2025-11-23T05:46:44Z")

</div>

BLIP-2 can also handle this, but [for the purpose of completing this task, there seems to be no need to limit it to BLIP-2](https://huggingface.co/datasets/John6666/forum2/blob/main/blip2_classification_1.md) today…