# BART from finetuned BERT **URL:** \ **Category:** Intermediate\ **Created:** [April 1, 2021, 5:56am UTC](https://discuss.huggingface.co/t/bart-from-finetuned-bert/5246 "2021-04-01T05:56:31Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![gchaperon](https://avatars.discourse-cdn.com/v4/letter/g/f475e1/32.png) [@gchaperon](https://discuss.huggingface.co/u/gchaperon)\ **Post date:** [April 1, 2021, 5:56am UTC](https://discuss.huggingface.co/t/bart-from-finetuned-bert/5246/1 "2021-04-01T05:56:31Z")
Hi all! Is it possible to use a pretrained BERT model to initialize the encoder part of an encoder-decoder model like BART, leaving the decoder uninitialized (or random), and then do fintuning on some seq2seq task? How should I proceed if its possible? Does someone know of previous instances where something like that has been tried? Thanks in advance! Best, Gabriel. ---
**Author:** ![Zhylkaaa](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@Zhylkaaa](https://discuss.huggingface.co/u/Zhylkaaa)\ **Post date:** [September 9, 2021, 7:39pm UTC](https://discuss.huggingface.co/t/bart-from-finetuned-bert/5246/2 "2021-09-09T19:39:09Z")
I am not entirely sure about Bart but you can check out this: [transformers/modeling\_encoder\_decoder.py at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/master/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py) Also you can read the publication linked in comments, I think this is similar to what you want to achieve ---
**Author:** ![gchaperon](https://avatars.discourse-cdn.com/v4/letter/g/f475e1/32.png) [@gchaperon](https://discuss.huggingface.co/u/gchaperon)\ **Post date:** [September 9, 2021, 8:01pm UTC](https://discuss.huggingface.co/t/bart-from-finetuned-bert/5246/3 "2021-09-09T20:01:13Z")
Hi! yeah i eventually found that as well, it is indeed what I had in mind, plus the linked papers where super insightful.