# Using BERT embeddings as input for transformer architecture

**URL:** <https://discuss.huggingface.co/t/using-bert-embeddings-as-input-for-transformer-architecture/19547>\
**Category:** 🤗Transformers\
**Created:** [June 23, 2022, 6:53pm UTC](https://discuss.huggingface.co/t/using-bert-embeddings-as-input-for-transformer-architecture/19547 "2022-06-23T18:53:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![canovich](https://avatars.discourse-cdn.com/v4/letter/c/aeb1de/32.png) [@canovich](https://discuss.huggingface.co/u/canovich)\
**Post date:** [June 23, 2022, 6:53pm UTC](https://discuss.huggingface.co/t/using-bert-embeddings-as-input-for-transformer-architecture/19547/1 "2022-06-23T18:53:35Z")

</div>

I will use BERT’s embedding weights(as discussed [here](https://discuss.huggingface.co/t/how-to-get-embedding-matrix-of-bert-in-hugging-face/10261/4)) for embedding in embedding layers of the transformer model. But my question is: Doesn’t embeddings of bert already go through the whole encoding layer and got that matrix? Why I shouldn’t just remove-freeze the encoding layer and use bert embedding vectors as input for the decoding layer? And also I will use bert embeddings in the input of the decoding layer. Why should I not freeze attention layers in decoder layer too? Because embeddings of output text already have attention information? Thanks in advance.