# Adapting BLIP2 for zero-shot classification

**URL:** <https://discuss.huggingface.co/t/adapting-blip2-for-zero-shot-classification/46812>\
**Category:** 🤗Transformers\
**Created:** [July 14, 2023, 9:53am UTC](https://discuss.huggingface.co/t/adapting-blip2-for-zero-shot-classification/46812 "2023-07-14T09:53:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dannimassi](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@dannimassi](https://discuss.huggingface.co/u/dannimassi)\
**Post date:** [July 14, 2023, 9:53am UTC](https://discuss.huggingface.co/t/adapting-blip2-for-zero-shot-classification/46812/1 "2023-07-14T09:53:41Z")

</div>

Hi there

I am attempting to adapt the Blip2Model for a zero-shot classification task as follows:

- N text sentences/classes → x = N text embeddings
- 1 test image → y = 1 image embedding
- soft-max(dot-product(x, y)) to get the probabilities over classes

This is my solution so far:

```auto
    def get_img_embedding(images]):
        """
        Turn a list of image inputs into tensor of embedding vectors
        images should be of shape (batch_size, channels, height, width)
        """
        image_tensors = blip2model.preproc([
            Image.open(i.path) # type: ignore
        for i in images], return_tensors='pt') # Dict with 'pixel_values' entry of size batch_size, C, H, W

        image_tensors = image_tensors.to(self.device, torch.float16) # type: ignore

        # pass images through the vision model and then the qformer to get query-conditional image features
        query_outputs = blip2model.get_qformer_features(**image_tensors) # tuple (last_hidden_state, pooler_output)
        query_output = query_outputs['pooler_output'] # (batch_size, hidden_size)
        # project query-conditional image features into language space
        image_features = blip2model.language_projection(query_output) # shape (batch_size, hidden_size)
        image_features /= image_features.norm(dim=-1, keepdim=True)

        return image_features

    def get_text_embedding(texts):
        """
        Turn a list of text inputs into tensor of embedding vectors.
        texts is a list of strings to embed.
        """

        text_tokens = blip2model.text_tokenizer(texts, padding=True, return_tensors='pt')
        text_tokens = text_tokens.to(self.device) 

        text_outputs = blip2model.get_text_features(**text_tokens, output_hidden_states=True) # type: ignore
        text_features = text_outputs['hidden_states'][-1][:, 0, :] # extract [CLS] embedding from last hidden state, shape (batch_size, hidden_size)
        text_features /= text_features.norm(dim=-1, keepdim=True)
        return text_features

```

Then I would take the dot product between the two. Am I on the right track?

Thanks

---

<div class="post-metadata">

**Author:** ![wingz1](https://avatars.discourse-cdn.com/v4/letter/w/8edcca/32.png) [@wingz1](https://discuss.huggingface.co/u/wingz1)\
**Post date:** [August 2, 2023, 10:16pm UTC](https://discuss.huggingface.co/t/adapting-blip2-for-zero-shot-classification/46812/3 "2023-08-02T22:16:37Z")

</div>

Have you had any luck with this? I’m also interested in knowing the best way to get BLIP2 images and texts into the same embedding space for retrieval type of purposes. (Not caption generation per say, but instead, comparing a set of caption embeddings and image embeddings to see which has the best match.

---

<div class="post-metadata">

**Author:** ![tankwell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tankwell/32/18511_2.png) [@tankwell](https://discuss.huggingface.co/u/tankwell)\
**Post date:** [August 13, 2023, 3:41pm UTC](https://discuss.huggingface.co/t/adapting-blip2-for-zero-shot-classification/46812/4 "2023-08-13T15:41:30Z")

</div>

BTW, I don’t know if that is the correct place to ask, but in the original code (lavis package) there is an `itm` (image\_text\_matching) head, see here

[ITM\_head implementation](https://github.com/salesforce/LAVIS/blob/f982acc73288408bceda2d35471a8fcf55aa04ca/lavis/models/blip2_models/blip2_image_text_matching.py#L15)

Is there a similar / parallel class in the 🤗 implementation that can help in this task?  
especially with that head pretrained

---

<div class="post-metadata">

**Author:** ![atomwalk12](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@atomwalk12](https://discuss.huggingface.co/u/atomwalk12)\
**Post date:** [August 8, 2024, 8:13am UTC](https://discuss.huggingface.co/t/adapting-blip2-for-zero-shot-classification/46812/5 "2024-08-08T08:13:03Z")

</div>

Were you able to solve the task? I noticed that you are using a slightly different approach with respect to [[1]](https://discuss.huggingface.co/t/text-classification-using-blip2/100978).  
In the previous post, the output field `qformer_outputs.last_hidden_state` is used to synthesis the information from the `qformer` using the `Blip2ForConditionalGeneration` class. Your approach seems to be using `Blip2Model`.

As far as my understanding goes, the q-former already makes use of the vision model to generate its output. Could anyone with more experience explain which of these two methods is more effective?