# How create BERT2Rand Encoder-Decoder model

**URL:** <https://discuss.huggingface.co/t/how-create-bert2rand-encoder-decoder-model/4399>\
**Category:** Models\
**Created:** [March 15, 2021, 6:23am UTC](https://discuss.huggingface.co/t/how-create-bert2rand-encoder-decoder-model/4399 "2021-03-15T06:23:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kaushal](https://avatars.discourse-cdn.com/v4/letter/k/a88e4f/32.png) [@kaushal](https://discuss.huggingface.co/u/kaushal)\
**Post date:** [March 15, 2021, 6:23am UTC](https://discuss.huggingface.co/t/how-create-bert2rand-encoder-decoder-model/4399/1 "2021-03-15T06:23:36Z")

</div>

There are multiple helpful references ([https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44\_VAyKE?usp=sharing#scrollTo=6r2-M5hYt-Vw](https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=6r2-M5hYt-Vw) ) for creating instances of BERT2BERT and BERT2Share models. I was wondering how to create a BERT2Rand Encoder-Decoder model where the encoder parameters will be loaded from pre-trained checkpoint and decoder parameters should be randomly initialized?

By reading document and codes, I tried this  
`EncoderDecoderModel.from_encoder_decoder_pretrained(bert-base-multilingual-cased, None)`  
which gave fallowing error:  
`Huggingface AssertionError: If *decoder_model* is not defined as an argument, a *decoder pretrained model_name_or_path* has to be define`

I am not sure how to fix this. Please let me in this. Thank you!

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [March 15, 2021, 7:56am UTC](https://discuss.huggingface.co/t/how-create-bert2rand-encoder-decoder-model/4399/2 "2021-03-15T07:56:09Z")

</div>

Hi there,

if you want a randomly initialized decoder then you can create the decoder separately, save it, and then pass that to the `from_encoder_decoder_pretrained`. Following code snipper shows how you can init the encoder using pre-tarined `bert-base` and a randomly initialized decoder of bert-base size.

```python
from transformers import BertConfig, BertLMHeadModel, EncoderDecoderModel

decoder_config = BertConfig(is_decoder=True)# bert-base size
decoder = BertLMHeadModel(decoder_config)
decoder.save_pretrained("decoder") # save the decoder

model = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "decoder")

# verify the decoder
model.decoder.config.is_decoder
model.decoder.config.add_cross_attention

```

---

<div class="post-metadata">

**Author:** ![kaushal](https://avatars.discourse-cdn.com/v4/letter/k/a88e4f/32.png) [@kaushal](https://discuss.huggingface.co/u/kaushal)\
**Post date:** [March 16, 2021, 10:26am UTC](https://discuss.huggingface.co/t/how-create-bert2rand-encoder-decoder-model/4399/3 "2021-03-16T10:26:51Z")

</div>

Thank you @valhalla! It is helpful information.