# BERT2BERT Notebook for Models without GenerationMixin

**URL:** <https://discuss.huggingface.co/t/bert2bert-notebook-for-models-without-generationmixin/2015>\
**Category:** 🤗Transformers\
**Created:** [November 12, 2020, 9:16am UTC](https://discuss.huggingface.co/t/bert2bert-notebook-for-models-without-generationmixin/2015 "2020-11-12T09:16:48Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\
**Post date:** [November 12, 2020, 9:16am UTC](https://discuss.huggingface.co/t/bert2bert-notebook-for-models-without-generationmixin/2015/1 "2020-11-12T09:16:48Z")

</div>

Hello,

I have just stambled upon the “Warm-starting BERT2BERT for CNN/Dailymail” Notebook @patrickvonplaten kindly shared with us: [https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT\_for\_CNN\_Dailymail.ipynb](https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb)

From my understanding, one can use this notebook for fine-tuning any model _ForConditionalGeneration_, such as T5, Pegasus, ProphetNet, etc. Can you please confirm this?

Moreover, I am interested in the new Longformer and Reformer models, which you can feed much longer sequences. These two models do not have a _ForConditionalGeneration_ class. However, I was wondering if they could be fine-tuned on summarization tasks using the same script, e.g. changing

> from transformers import EncoderDecoderModel
> 
> bert2bert = EncoderDecoderModel.from\_encoder\_decoder\_pretrained(“bert-base-uncased”, “bert-base-uncased”)

With

> from transformers import LongformerModel, ReformerModel
> 
> longformer = LongformerModel.from\_pretrained(“allenai/longformer-base-4096”)  
> reformer = ReformerModel.from\_pretrained(“google/reformer-enwik8”)

Thank you for your help! 🙂