# Combine word embedding with visual features from ViT model

**URL:** <https://discuss.huggingface.co/t/combine-word-embedding-with-visual-features-from-vit-model/25849>\
**Category:** Models\
**Created:** [November 10, 2022, 7:22am UTC](https://discuss.huggingface.co/t/combine-word-embedding-with-visual-features-from-vit-model/25849 "2022-11-10T07:22:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![daeron](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@daeron](https://discuss.huggingface.co/u/daeron)\
**Post date:** [November 10, 2022, 7:22am UTC](https://discuss.huggingface.co/t/combine-word-embedding-with-visual-features-from-vit-model/25849/1 "2022-11-10T07:22:15Z")

</div>

For example I have a text nn.Embedding tensor W with size (1, 7, 768) [batch\_size x seq\_len x emb\_size]  
Using pretrained Vit model with the input image, I get the output with last\_hidden\_state L with size(1,196,768), not use CLS token, and pooler output P with size (1,768). How can I combine the word embedding tensor with the output of ViT model ? Is it correct to torch.cat([W, L],dim=1) to have a representative tensor of size (1, 203, 768) for text and image ?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [November 10, 2022, 9:00am UTC](https://discuss.huggingface.co/t/combine-word-embedding-with-visual-features-from-vit-model/25849/2 "2022-11-10T09:00:36Z")

</div>

Yes that’s indeed the correct way, by concatenating them along the sequence (also called time) dimension.

---

<div class="post-metadata">

**Author:** ![daeron](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@daeron](https://discuss.huggingface.co/u/daeron)\
**Post date:** [November 10, 2022, 4:46pm UTC](https://discuss.huggingface.co/t/combine-word-embedding-with-visual-features-from-vit-model/25849/3 "2022-11-10T16:46:58Z")

</div>

Thank you sir.  
I also wonder in which kind of scenario that pooler\_output would rather be used than the last\_hidden\_state tensor ?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [November 11, 2022, 10:01am UTC](https://discuss.huggingface.co/t/combine-word-embedding-with-visual-features-from-vit-model/25849/4 "2022-11-11T10:01:35Z")

</div>

The `pooler_output` can be seen as a “summary” of an entire sequence of tokens. There are typically 2 ways to obtain a summary of all tokens, either by using the `pooler_output` or by averaging the final hidden states of all tokens.