# Replacing the decoder of an xxxEncoderDecoderModel

**URL:** <https://discuss.huggingface.co/t/replacing-the-decoder-of-an-xxxencoderdecodermodel/13367>\
**Category:** 🤗Transformers\
**Created:** [January 4, 2022, 2:09pm UTC](https://discuss.huggingface.co/t/replacing-the-decoder-of-an-xxxencoderdecodermodel/13367 "2022-01-04T14:09:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [January 4, 2022, 2:09pm UTC](https://discuss.huggingface.co/t/replacing-the-decoder-of-an-xxxencoderdecodermodel/13367/1 "2022-01-04T14:09:26Z")

</div>

A question someone had was how to replace the decoder of an existing [VisionEncoderDecoderModel](https://huggingface.co/docs/transformers/model_doc/visionencoderdecoder) from the hub. Namely, the [TrOCR](https://huggingface.co/docs/transformers/model_doc/trocr) model currently only has checkpoints on the hub with an English-only language model (RoBERTa) as decoder - how to replace it with a multilingual XLMRoBERTa model?

Here’s the answer:

```auto
from transformers import VisionEncoderDecoderModel, RobertaForCausalLM

model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")

# replace decoder
model.decoder = RobertaForCausalLM.from_pretrained("xlm-roberta-base", is_decoder=True, add_cross_attention=True)

```

As you can see, we first initialize the entire model from the [hub](https://huggingface.co/microsoft/trocr-base-handwritten), after which we replace the decoder with a custom one. Also note that we are initializing a `RobertaForCausalLM` model, which includes the language modeling head on top (as opposed to `RobertaModel`). We also set the `is_decoder` and `add_cross_attention` attributes of the model’s configuration to make sure cross-attention is added between the encoder and decoder. A warning will be printed when we initialize the model, indicating that the weights of the cross-attention layers are randomly initialized.

### Preparing the data

Also note that, in case you are going to prepare data for the model, one must use the appropriate tokenizer to create the labels for the model (in this case, one should use `XLMRobertaTokenizer`). Let’s define a `TrOCRProcessor` (which wraps a feature extractor and a tokenizer into a single object), by first using the one from the corresponding checkpoint, and then replace the tokenizer part:

```auto
from transformers import TrOCRProcessor, XLMRobertaTokenizer

processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
processor.tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base")

```

One can then prepare a single (image, text) pair for the model as follows:

```auto
from PIL import Image
import torch

image = Image.open("...").convert("RGB")
text = "..."

pixel_values = processor(image, return_tensors="pt").pixel_values
# add labels (input_ids) by encoding the text
labels = processor.tokenizer(text, padding="max_length", truncation=True).input_ids
# important: make sure that PAD tokens are ignored by the loss function
labels = [label if label != processor.tokenizer.pad_token_id else -100 for label in labels]

encoding = {"pixel_values": pixel_values.squeeze(), "labels": torch.tensor(labels)}

```

---

<div class="post-metadata">

**Author:** ![sergak0](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sergak0](https://discuss.huggingface.co/u/sergak0)\
**Post date:** [January 7, 2022, 5:32pm UTC](https://discuss.huggingface.co/t/replacing-the-decoder-of-an-xxxencoderdecodermodel/13367/3 "2022-01-07T17:32:59Z")

</div>

I also tried to use trocr-small-handwritten encoder and get this error, when tried to predict image by model:  
‘VisionEncoderDecoderModel’ object has no attribute ‘enc\_to\_dec\_proj’

```auto
from transformers import RobertaForCausalLM, VisionEncoderDecoderModel

model = VisionEncoderDecoderModel.from_pretrained(f'microsoft/trocr-small-handwritten')
model.decoder = RobertaForCausalLM.from_pretrained('xlm-roberta-base', is_decoder=True, add_cross_attention=True)

```

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/0b3714e057b4da62c947c319ee7ba343b53ffa39.png)

---

<div class="post-metadata">

**Author:** ![magistermilitum](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/magistermilitum/32/20188_2.png) [@magistermilitum](https://discuss.huggingface.co/u/magistermilitum)\
**Post date:** [December 16, 2023, 11:03pm UTC](https://discuss.huggingface.co/t/replacing-the-decoder-of-an-xxxencoderdecodermodel/13367/4 "2023-12-16T23:03:03Z")

</div>

Hi Nielsr, I’ve been training using this approach over the last few days. In my case (i have 50k training pairs), replacing the decoder while keeping the weights from trocr-base-handwritten has resulted in better performance during training, especially in WER, than when using this which seems needing lots of training pairs:

```auto
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(encoder, decoder)

```

But I was wondering how exactly we should use the model generated in the way that you propose during inference. Even when I save the processor and model, the predictions on inference seem to be absurd (I suppose it’s a vocabulary alignment problem). For instance, when I switch to a BERT multilingual model, I achieve CER = 0.09 and WER = 0.17 on the development set, which are overall good results using this code:

```auto
processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
processor.tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
processor.save_pretrained('./processor')

model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")
model.decoder = AutoModelForCausalLM.from_pretrained("bert-base-multilingual-cased", is_decoder=True, add_cross_attention=True)

model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
model.config.vocab_size = model.config.decoder.vocab_size

model.save_pretrained("trainer_trocr/model")

```

But when using the trained model on inferences using this:

```auto
device=torch.device("cpu")
processor = TrOCRProcessor.from_pretrained("./processor")
model = VisionEncoderDecoderModel.from_pretrained("trainer_trocr/model").to(device)

# load image 
url = "D:\\trocr\\1538983940018_363.png"
image = Image.open(url).convert("RGB")

pixel_values = processor(image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(generated_ids)
print("OCR output:", generated_text)

```

The model take ages to load and I get results as : död död död 鈦 鈦 鈦ਿਕਿਕਿਕ Première Première Premièreчиличиличили 锢 锢 锢:

I’ve tried different combinations, but I can’t get the model to perform as well as during training. If you have any suggestions, I would be really grateful. Have a nice day, and thank you for all the work you’ve put into trocr.