# VivitModel last hidden states dimension Problem

**URL:** <https://discuss.huggingface.co/t/vivitmodel-last-hidden-states-dimension-problem/96852>\
**Category:** 🤗Transformers\
**Created:** [July 11, 2024, 4:02pm UTC](https://discuss.huggingface.co/t/vivitmodel-last-hidden-states-dimension-problem/96852 "2024-07-11T16:02:48Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![BeefWellington](https://avatars.discourse-cdn.com/v4/letter/b/bc79bd/32.png) [@BeefWellington](https://discuss.huggingface.co/u/BeefWellington)\
**Post date:** [July 11, 2024, 4:02pm UTC](https://discuss.huggingface.co/t/vivitmodel-last-hidden-states-dimension-problem/96852/1 "2024-07-11T16:02:48Z")

</div>

In the example document of VivitModel [Video Vision Transformer (ViViT) (huggingface.co)](https://huggingface.co/docs/transformers/model_doc/vivit), why the last\_hidden\_states.shape is 3137 but not 3136? What is the additional 1 dimension?

Based on the default parameters, image\_size=224, num\_frames=32, tubelet\_size=[2,16,16], then number of tubelets is (224/16)x(224/16)x(32/2) = 14x14x16 = 3136.  
🤯

```auto
# forward pass
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state
list(last_hidden_states.shape)
[1, 3137, 768]

```