# How to parameter efficient finetune Decoder in encoder-decoder model?

**URL:** <https://discuss.huggingface.co/t/how-to-parameter-efficient-finetune-decoder-in-encoder-decoder-model/81874>\
**Category:** 🤗Transformers\
**Created:** [April 15, 2024, 3:13pm UTC](https://discuss.huggingface.co/t/how-to-parameter-efficient-finetune-decoder-in-encoder-decoder-model/81874 "2024-04-15T15:13:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhishekkgupta](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abhishekkgupta/32/25855_2.png) [@Abhishekkgupta](https://discuss.huggingface.co/u/Abhishekkgupta)\
**Post date:** [April 15, 2024, 3:13pm UTC](https://discuss.huggingface.co/t/how-to-parameter-efficient-finetune-decoder-in-encoder-decoder-model/81874/1 "2024-04-15T15:13:46Z")

</div>

I have been trying to finetune the decoder by inserting an adapter after the ffn block of a large encoder-decoder model for the MT task, but the blue score is bad when compared to full finetuning.  
Please, can anyone suggest a better method to finetune or refer me to some resources to finetune the decoder of a model

---

<div class="post-metadata">

**Author:** ![yasmineee](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yasmineee/32/29763_2.png) [@yasmineee](https://discuss.huggingface.co/u/yasmineee)\
**Post date:** [July 27, 2024, 7:45pm UTC](https://discuss.huggingface.co/t/how-to-parameter-efficient-finetune-decoder-in-encoder-decoder-model/81874/2 "2024-07-27T19:45:29Z")

</div>

have you tried fixing it? i faced the same exact problem

---

<div class="post-metadata">

**Author:** ![Abhishekkgupta](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abhishekkgupta/32/25855_2.png) [@Abhishekkgupta](https://discuss.huggingface.co/u/Abhishekkgupta)\
**Post date:** [July 27, 2024, 7:57pm UTC](https://discuss.huggingface.co/t/how-to-parameter-efficient-finetune-decoder-in-encoder-decoder-model/81874/3 "2024-07-27T19:57:40Z")

</div>

yes, it was an issue with the initialization of layer norms in the network. I had to manually initialize the layer norms which fixed the issue

---

<div class="post-metadata">

**Author:** ![yasmineee](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yasmineee/32/29763_2.png) [@yasmineee](https://discuss.huggingface.co/u/yasmineee)\
**Post date:** [July 27, 2024, 8:00pm UTC](https://discuss.huggingface.co/t/how-to-parameter-efficient-finetune-decoder-in-encoder-decoder-model/81874/4 "2024-07-27T20:00:30Z")

</div>

could you please share with me the notebook?

---

<div class="post-metadata">

**Author:** ![Abhishekkgupta](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/abhishekkgupta/32/25855_2.png) [@Abhishekkgupta](https://discuss.huggingface.co/u/Abhishekkgupta)\
**Post date:** [July 27, 2024, 8:12pm UTC](https://discuss.huggingface.co/t/how-to-parameter-efficient-finetune-decoder-in-encoder-decoder-model/81874/5 "2024-07-27T20:12:43Z")

</div>

I am not using notebook but I can share the code to initialize new layernorm params

```auto
    for name, param in model.named_parameters():
             if "language_adaptor" in name and "norm" in name:
                if "weight" in name: # Layer normalization scale
                    print("Initializing {} with mean=0 and std=1".format(name))
                    init.ones_(param.data)
                elif "bias" in name: # Layer normalization bias
                    print("Initializing {} with mean=0".format(name))
                    init.zeros_(param.data)

```