# Train new VisonEncoderDecoder model for new languages

**URL:** <https://discuss.huggingface.co/t/train-new-visonencoderdecoder-model-for-new-languages/14359>\
**Category:** Intermediate\
**Created:** [February 3, 2022, 2:29am UTC](https://discuss.huggingface.co/t/train-new-visonencoderdecoder-model-for-new-languages/14359 "2022-02-03T02:29:05Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Su-Youn](https://avatars.discourse-cdn.com/v4/letter/s/e8c25b/32.png) [@Su-Youn](https://discuss.huggingface.co/u/Su-Youn)\
**Post date:** [February 3, 2022, 2:29am UTC](https://discuss.huggingface.co/t/train-new-visonencoderdecoder-model-for-new-languages/14359/1 "2022-02-03T02:29:05Z")

</div>

Hello.  
I am trying to train a new VisionEncoderDecoder model for new language (Bahasa-Indonesian).  
The initial performance is pretty bad (72% CER). I am wondering whether I can get advices from anyone who tried similar tasks.

Based on my understanding, Bahasa-Indonesian uses Alphabet characters (without special characters such as German Epsilon). So my plan is as follows:

1. Encoder: Pretrained ViT model (“google/vit-base-patch16-224”)
2. Decoder: Pretrained Indonesian Roberta Language model (“'cahya/roberta-base-indonesian-1.5G”)

For updating decoder weights using Pertained Roberta model, I have a few questions.  
Currently, the parameter names from Roberta models are different from Decoder model parameters, so we need some mapping process. I did the following steps, and i am wondering whether there are some errors.

# encoder

encoder = ViTModel.from\_pretrained(“google/vit-base-patch16-224”)

# decoder

lmconfig = TrOCRConfig(vocab\_size=indonesian\_lm\_vocab\_size, …, ) # update accordingly by indonesian lm  
decoder = CausualLM(lmcfg)  
lm = RobertaForCausalLM.from\_pretrained(“cahya/roberta-base-indonesian-1.5G”)  
decoder = load\_wts(decoder, lm)

def load\_wts(decoder, lm):  
param\_name\_dict = {  
‘attention.self.query.weight’: ‘self\_attn.q\_proj.weight’,  
‘attention.self.query.bias’: ‘self\_attn.q\_proj.bias’,  
‘attention.self.key.weight’: ‘self\_attn.k\_proj.weight’,  
‘attention.self.key.bias’: ‘self\_attn.k\_proj.bias’,  
‘attention.self.value.weight’: ‘self\_attn.v\_proj.weight’,  
‘attention.self.value.bias’: ‘self\_attn.v\_proj.bias’,  
‘attention.output.dense.weight’: ‘self\_attn.out\_proj.weight’,  
‘attention.output.dense.bias’: ‘self\_attn.out\_proj.bias’,  
‘attention.output.LayerNorm.weight’: ‘self\_attn\_layer\_norm.weight’,  
‘attention.output.LayerNorm.bias’: ‘self\_attn\_layer\_norm.bias’,  
‘output.LayerNorm.weight’: ‘final\_layer\_norm.weight’,  
‘output.LayerNorm.bias’:‘final\_layer\_norm.bias’,  
‘word\_embeddings.weight’:‘embed\_tokens.weight’,  
‘position\_embeddings.weight’: ‘embed\_positions.weight’,  
‘LayerNorm.weight’:‘layernorm\_embedding.weight’,  
‘LayerNorm.bias’: ‘layernorm\_embedding.bias’  
}  
wts = lm.state\_dict()  
new\_wts = {}  
dwts = decoder.state\_dict()  
for key in wts.keys():  
nkey = rename\_param(key)  
if nkey:  
new\_wts[nkey] = wts[key]  
decoder.load\_state\_dict(new\_wts, strict=False)  
return decoder

As paper mentioned, this does not update weights in the encoder-decoder attention layers  
since they do not exist in Roberta language model.

If you have any advice, please let me know. Thank you a lot.