# Image Embedding from PaliGemma Model **URL:** \ **Category:** Beginners\ **Created:** [August 20, 2024, 7:34pm UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217 "2024-08-20T19:34:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Samvardhan777](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/samvardhan777/32/30710_2.png) [@Samvardhan777](https://discuss.huggingface.co/u/Samvardhan777)\ **Post date:** [August 20, 2024, 7:34pm UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/1 "2024-08-20T19:34:28Z")
Hey there , I want to extract image embedding from the image. ```auto from transformers import AutoTokenizer, PaliGemmaForConditionalGeneration, PaliGemmaProcessor import torch model_id = "google/paligemma-3b-mix-224" model = PaliGemmaForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.bfloat16) processor = PaliGemmaProcessor.from_pretrained(model_id) from io import BytesIO text = "Describe the scene." image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/bee.JPG?download=true" response = requests.get(image_url) image = Image.open(BytesIO(response.content)) # Preprocess inputs inputs = processor(text=text, images=image, return_tensors="pt").to(device) embeddings = outputs.image_hidden_states print(embeddings) ``` Value for the image\_hidden\_states is null can you please advice on this pls ---
**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\ **Post date:** [August 21, 2024, 5:43am UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/2 "2024-08-21T05:43:40Z")
Hey! Paligemma currently doesn’t return image embeddings in forward call. I will add this to my todo list 🙂 You can also obtain image hidden states by running the following, first run through vision backbone and get the last hidden state. Then run through the MM projector ```python image_outputs = model.vision_tower(pixel_values) selected_image_feature = image_outputs.last_hidden_state image_features = model.multi_modal_projector(selected_image_feature) image_features = image_features / (model.config.hidden_size**0.5) ``` ---
**Author:** ![Samvardhan777](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/samvardhan777/32/30710_2.png) [@Samvardhan777](https://discuss.huggingface.co/u/Samvardhan777)\ **Post date:** [August 21, 2024, 7:07am UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/3 "2024-08-21T07:07:45Z")
Hey Raushan Thanks for this information 😀 I am able to get the image embedding based upon ur suggestion Can you also pls advice me how we can get the multimodal and text embedding from the paligemma model. Currently I thought of using Paligemma for multimodel image search ---
**Author:** ![malba96](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@malba96](https://discuss.huggingface.co/u/malba96)\ **Post date:** [October 21, 2024, 8:27am UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/4 "2024-10-21T08:27:26Z")
Hey, do you have any update on how to get multi-modal embeddings? maybe it is via self.language\_model.get\_output\_embeddings, which for `Gemma` is `self.lm_head`? ---
**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\ **Post date:** [October 21, 2024, 5:34pm UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/5 "2024-10-21T17:34:12Z")
Sorry, I didn’t see the prev reply. Can you elaborate on what you mean by multimodal embedding? Paligemma, in contrast to some other VLMs, was not trained for image-text search thus does not have dedicated embedding layers for multimodal. Rather it has a separate image and a separate text embedding, which are usually concated to feed into an LLM. For image-text search among VLMs I know that BLIP had a similar functionaluty, but you might also prefer to use a more classic way like CLIP or Flava ---
**Author:** ![malba96](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@malba96](https://discuss.huggingface.co/u/malba96)\ **Post date:** [October 21, 2024, 8:03pm UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/6 "2024-10-21T20:03:55Z")
Sorry for the confusion. I meant a single sentence embedding that encodes both image and text at the end of the text decoder. Now that I check in detail the architecture, as you mention there is no embedding for such case. Based on the code you provided I have a few questions: 1. Why do you normalize using `(model.config.hidden_size**0.5)`? 2. The embeddings are shaped `(batch size, 256, 2048)`. In the case I wanted a single image representation per image `(batch size, 2048)`. would you take the mean pooling or the [contextualised embedding of the last image token](https://stackoverflow.com/questions/76926025/sentence-embeddings-from-llama-2-huggingface-opensource)? (maybe you have another suggestion to reduce the dimensionality) ---
**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\ **Post date:** [October 22, 2024, 8:41am UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/7 "2024-10-22T08:41:10Z")
1. The normalization is done because the Paligemma model internally does the same thing before merging image and text embeddings. The code snippet is simply copy-paste from library codebase 2. Yes, since the ViT backbone returns images of `patch_size * image_size` length. You can try to do mean pooling or any other method, but you should note that changing model architecture means we’ll need to train the model. There are no available weights afaik for Paligemma image-text retrieval. That is why I was suggesting as an easy option take one of pre-trained weights on other models and fine-tune on your own domain which will be easier and less resource consuming ---
**Author:** ![ishity](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ishity/32/31154_2.png) [@ishity](https://discuss.huggingface.co/u/ishity)\ **Post date:** [March 5, 2025, 1:37am UTC](https://discuss.huggingface.co/t/image-embedding-from-paligemma-model/103217/8 "2025-03-05T01:37:51Z")
Hi! I want to know how to get both text and image embeddings. I tried the code which are shown in the first and second messages, but I can’t run them. Also, I tried to get them by using another VL model, so if you have some tips for doing these kinds of things, please tell me!