# Issues with Whisper Encoder: Positional Encoding

**URL:** <https://discuss.huggingface.co/t/issues-with-whisper-encoder-positional-encoding/26100>\
**Category:** 🤗Transformers\
**Created:** [November 15, 2022, 11:22am UTC](https://discuss.huggingface.co/t/issues-with-whisper-encoder-positional-encoding/26100 "2022-11-15T11:22:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sahuamrit](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@sahuamrit](https://discuss.huggingface.co/u/sahuamrit)\
**Post date:** [November 15, 2022, 11:22am UTC](https://discuss.huggingface.co/t/issues-with-whisper-encoder-positional-encoding/26100/1 "2022-11-15T11:22:51Z")

</div>

1. The positional embedding in [hugging face implementation](https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py#:~:text=self.embed_positions%20%3D%20nn.Embedding(self.max_source_positions%2C%20embed_dim)) of whisper is a nn.Embedding layer (learnable) as opposed to [OpenAI’s implementation](https://github.com/openai/whisper/blob/9f70a352f9f8630ab3aa0d06af5cb9532bd8c21d/whisper/model.py#:~:text=self.register_buffer(%22positional_embedding%22%2C%20sinusoids(n_ctx%2C%20n_state))) of a standard sinusoid positional embedding.
2. The input to the model is assumed to be padded to 3000 frames (both in hugging face and OpenAI’s implementation). Isn’t it non-optimal? Rather, can we pad it to the largest sequence in a batch?

@sanchit-gandhi can you help here?

---

<div class="post-metadata">

**Author:** ![sanchit-gandhi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sanchit-gandhi/32/21280_2.png) [@sanchit-gandhi](https://discuss.huggingface.co/u/sanchit-gandhi)\
**Post date:** [November 15, 2022, 11:56am UTC](https://discuss.huggingface.co/t/issues-with-whisper-encoder-positional-encoding/26100/2 "2022-11-15T11:56:27Z")

</div>

Hey @sahuamrit,

1. The models are equivalent - maybe @ArthurZ can shed light on why we use a learnable embed layer in Transformers?
2. This is how OpenAI trained the model (pad/truncate audio inputs to 30s, then compute log-Mel filter bank features). You can read here why this works: [Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper#load-whisperfeatureextractor). Training the model by padding/truncating to max length would require an attention mask, which is not how the model is trained. Hence, we pad/truncate to a fixed length.

---

<div class="post-metadata">

**Author:** ![ArthurZ](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arthurz/32/26972_2.png) [@ArthurZ](https://discuss.huggingface.co/u/ArthurZ)\
**Post date:** [November 15, 2022, 3:09pm UTC](https://discuss.huggingface.co/t/issues-with-whisper-encoder-positional-encoding/26100/3 "2022-11-15T15:09:13Z")

</div>

Hey!

1. I think it was a choice of simplicity; it is equivalent at inference but might indeed be problematic for training and should be frozen. We can open an issue on this to use `register_buffer`.  
Nice catch 🤗

---

<div class="post-metadata">

**Author:** ![sahuamrit](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@sahuamrit](https://discuss.huggingface.co/u/sahuamrit)\
**Post date:** [November 16, 2022, 5:25am UTC](https://discuss.huggingface.co/t/issues-with-whisper-encoder-positional-encoding/26100/4 "2022-11-16T05:25:40Z")

</div>

Hi @ArthurZ ,

I dont think it will be same even during inference. When you load official whisper weights, self.positional\_encoding won’t have anything and will be random.

I think since the demo had to do with finetuning, the nn.Embedding got learnt.

---

<div class="post-metadata">

**Author:** ![ArthurZ](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arthurz/32/26972_2.png) [@ArthurZ](https://discuss.huggingface.co/u/ArthurZ)\
**Post date:** [November 16, 2022, 11:06am UTC](https://discuss.huggingface.co/t/issues-with-whisper-encoder-positional-encoding/26100/5 "2022-11-16T11:06:20Z")

</div>

Not really sure I follow, the weights are not random for the 2 different positional embeddings.

- In the audio encoder we have `self.register_buffer("positional_embedding", sinusoids(n_ctx, n_state))`. Which uses the [`sinusoids`](https://github.com/openai/whisper/blob/9f70a352f9f8630ab3aa0d06af5cb9532bd8c21d/whisper/model.py#L48) function that has no randomness.
- In the text encoder, the `positional_embedding` is learned.