# How to train an EncoderDecoderModel with different pretrained encoder and decoder?

**URL:** <https://discuss.huggingface.co/t/how-to-train-an-encoderdecodermodel-with-different-pretrained-encoder-and-decoder/40358>\
**Category:** 🤗Transformers\
**Created:** [May 19, 2023, 9:01am UTC](https://discuss.huggingface.co/t/how-to-train-an-encoderdecodermodel-with-different-pretrained-encoder-and-decoder/40358 "2023-05-19T09:01:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [May 19, 2023, 9:01am UTC](https://discuss.huggingface.co/t/how-to-train-an-encoderdecodermodel-with-different-pretrained-encoder-and-decoder/40358/1 "2023-05-19T09:01:08Z")

</div>

When trying to initialize an `EncoderDecoderModel` with different pre-trained models, this kinda works without error:

```auto
import evaluate
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, DataCollatorForSeq2Seq
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments
from datasets import load_dataset, Dataset

import torch
from transformers import EncoderDecoderModel

src_tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-uncased")
tgt_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

multibert = EncoderDecoderModel.from_encoder_decoder_pretrained(
    "dbmdz/bert-base-german-uncased", "bert-base-uncased",
)

```

When we check the tokenizers, they are not aligned, esp. for the special tokens, e.g.

```auto
src_tokenizer.bos_token = src_tokenizer.cls_token
src_tokenizer.eos_token = src_tokenizer.sep_token
src_tokenizer.add_special_tokens({'pad_token': '[PAD]'})

tgt_tokenizer.bos_token = tgt_tokenizer.cls_token
tgt_tokenizer.eos_token = tgt_tokenizer.sep_token
tgt_tokenizer.add_special_tokens({'pad_token': '[PAD]'})

print(tgt_tokenizer.special_tokens_map, tgt_tokenizer.all_special_tokens, tgt_tokenizer.all_special_ids)

print(src_tokenizer.special_tokens_map, src_tokenizer.all_special_tokens, src_tokenizer.all_special_ids)

```

[out]:

```auto
({'bos_token': '[CLS]',
  'eos_token': '[SEP]',
  'unk_token': '[UNK]',
  'sep_token': '[SEP]',
  'pad_token': '[PAD]',
  'cls_token': '[CLS]',
  'mask_token': '[MASK]'},
 ['[CLS]', '[SEP]', '[UNK]', '[PAD]', '[MASK]'],
 [101, 102, 100, 0, 103])

({'bos_token': '[CLS]',
  'eos_token': '[SEP]',
  'unk_token': '[UNK]',
  'sep_token': '[SEP]',
  'pad_token': '[PAD]',
  'cls_token': '[CLS]',
  'mask_token': '[MASK]'},
 ['[CLS]', '[SEP]', '[UNK]', '[PAD]', '[MASK]'],
 [102, 103, 101, 0, 104])

```

Then it comes to the part when we need to set the tokenizers’ special tokens, **what should we set for the EOS when it’s different ids for the encoders and decoders?**

e.g.

```auto
# set special tokens

# I guess this must be the decoder's
multibert.config.decoder_start_token_id = tgt_tokenizer.bos_token_id

# And these are the encoder's tokenizer or decoder's?
multibert.config.eos_token_id = ???_tokenizer.eos_token_id

# This is the same so it doesn't matter, I guess.
multibert.config.pad_token_id = ???_tokenizer.pad_token_id

```

**And when we have the collator, which tokenizer should we use?**

```auto
data_collator = DataCollatorForSeq2Seq(???_tokenizer)

```

**Finally when initializing the Seq2SeqTrainer, it asks for the tokenizer, which should it be encoder’s or decoder’s?**

```auto
# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=???_tokenizer,
    args=training_args,
    train_dataset=ds_train.with_format("torch"),
    eval_dataset=ds_valid.with_format("torch"),
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    
)

```

---

<div class="post-metadata">

**Author:** ![gustavewu](https://avatars.discourse-cdn.com/v4/letter/g/839c29/32.png) [@gustavewu](https://discuss.huggingface.co/u/gustavewu)\
**Post date:** [September 6, 2023, 8:17am UTC](https://discuss.huggingface.co/t/how-to-train-an-encoderdecodermodel-with-different-pretrained-encoder-and-decoder/40358/2 "2023-09-06T08:17:13Z")

</div>

Hello, I encountered the same issue. Did you manage to resolve it?

---

<div class="post-metadata">

**Author:** ![AbderrahmanSkiredj1](https://avatars.discourse-cdn.com/v4/letter/a/8baadc/32.png) [@AbderrahmanSkiredj1](https://discuss.huggingface.co/u/AbderrahmanSkiredj1)\
**Post date:** [April 2, 2024, 5:11pm UTC](https://discuss.huggingface.co/t/how-to-train-an-encoderdecodermodel-with-different-pretrained-encoder-and-decoder/40358/3 "2024-04-02T17:11:14Z")

</div>

Any news? I am encoutering the same issue