# BERT2RND EncoderDecoderModel predicts random words for Translation tasks

**URL:** <https://discuss.huggingface.co/t/bert2rnd-encoderdecodermodel-predicts-random-words-for-translation-tasks/18538>\
**Category:** 🤗Transformers\
**Created:** [May 30, 2022, 6:03pm UTC](https://discuss.huggingface.co/t/bert2rnd-encoderdecodermodel-predicts-random-words-for-translation-tasks/18538 "2022-05-30T18:03:46Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![salma-elshafey](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@salma-elshafey](https://discuss.huggingface.co/u/salma-elshafey)\
**Post date:** [May 30, 2022, 6:03pm UTC](https://discuss.huggingface.co/t/bert2rnd-encoderdecodermodel-predicts-random-words-for-translation-tasks/18538/1 "2022-05-30T18:03:46Z")

</div>

Hello! I am using the EncoderDecoderModel for translating from Arabic to English by initializing the encoder with pretrained AraBERT checkpoint (a BERT model pre-trained on Arabic language) and a randomly initialized decoder. However, the BLEU score does not improve at all throughout the evaluation after each epoch, and the predicted sentences are very random and do not make sense.  
 ![bleu scores](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/864d58b26c690fb591a73c005a54bb079aecb329.jpeg)

Can anybody please help me? Here’s my colab notebook for the code: [Google Colab](https://colab.research.google.com/drive/1Cm53n2zoOqClONu8hqZ9HLi0TkILsPjw?usp=sharing)