# Leveraging pre-trained checkpoints for summarization

**URL:** <https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835>\
**Category:** Models\
**Created:** [August 23, 2020, 10:51pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835 "2020-08-23T22:51:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\
**Post date:** [August 23, 2020, 10:51pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/1 "2020-08-23T22:51:48Z")

</div>

The effectiveness of initializing Encoder-Decoder models from pre-trained encoder-only models, such as BERT and RoBERTa, for sequence-to-sequence tasks has been shown in: [https://arxiv.org/abs/1907.12461](https://arxiv.org/abs/1907.12461).

Similarly, the `EncoderDecoderModel` framework of 🤗Transformers can be used to leverage initialize Encoder-Decoder models from “bert-base-cased” or “roberta-base” for summarization.

One can initialize such a model with weights from pre-trained checkpoints via:

```
from transformers import EncoderDecoderModel
bert2bert = EncoderDecoderModel.from_encoder_decoder_pretrained("bert-base-uncased", "bert-base-uncased")

```

A couple of models based on “bert-base-cased” or “roberta-base” have been trained this way for the CNN/Daily-Mail summarization task with the purpose of verifying that the `EncoderDecoderModel` framework is functional.

Below the Rouge2 - fmeasure results on the test set of CNN/Daily-Mai:

**Bert2GPT2** : _15.19_ [https://huggingface.co/patrickvonplaten/bert2gpt2-cnn\_dailymail-fp16](https://huggingface.co/patrickvonplaten/bert2gpt2-cnn_dailymail-fp16)  
**Bert2Bert** : _16.1_ - [https://huggingface.co/patrickvonplaten/bert2bert-cnn\_dailymail-fp16](https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16)  
**Roberta2Roberta** : _16.79_: [https://huggingface.co/patrickvonplaten/roberta2roberta-cnn\_dailymail-fp16](https://huggingface.co/patrickvonplaten/roberta2roberta-cnn_dailymail-fp16)  
**Roberta2Roberta (shared)**: _16.59_: [https://huggingface.co/patrickvonplaten/roberta2roberta-share-cnn\_dailymail-fp16](https://huggingface.co/patrickvonplaten/roberta2roberta-share-cnn_dailymail-fp16)

**Note** : The models below were trained without any hyper-parameter search and fp16 precision. For more detail, please refer to the respective model card.

UPDATE:

Better models using the Seq2Seq Trainer and code on the current master give the following results:

**BERT2BERT** on CNN/Dailymail: _18.22_ - [https://huggingface.co/patrickvonplaten/bert2bert\_cnn\_daily\_mail](https://huggingface.co/patrickvonplaten/bert2bert_cnn_daily_mail)  
**Roberta2Roberta (shared)** on BBC/XSum: _16.89_ - [https://huggingface.co/patrickvonplaten/roberta\_shared\_bbc\_xsum](https://huggingface.co/patrickvonplaten/roberta_shared_bbc_xsum)

Also two notebooks are attached to the model cards showing how Encoder-Decoder models can be trained using master.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 25, 2020, 2:22am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/2 "2020-08-25T02:22:10Z")

</div>

> [@patrickvonplaten](#):
>
> **Note** : The models below were trained without any hyper-parameter search and fp16 precision. For more detail, please refer to the respective model card.

Interesting results!

1. would love to know how finetune times/inference times compare to bart-base/bart-large. These are roughly bart-base size, right?

2. Would also love to know on xsum where gaps between good and worse models get magnified in ROUGE space.

3. Feels like we desperately need some sort of lb/aggregator, like the one you tried to get going for benchmarking. I know bart-large takes ~24h to get to ~ 21 ROUGE on cnn. @VictorSanh got 15.5 ROUGE2 with bart-base **on xsum** which felt a little low to me.

1. Are you using `pip install wandb`? Share your logs?

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 25, 2020, 4:44am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/3 "2020-08-25T04:44:15Z")

</div>

Does this mean we can delete bertabs!?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 25, 2020, 9:10am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/4 "2020-08-25T09:10:05Z")

</div>

1. In my Roberta2Roberta experiment for inference on cnn test dataset on P100, it took 2 hours , 22 minutes.  
I fine-tuned for 16 hours but got much worse results than Patrick. ROUGE-2 F-measure was just 9.9

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 25, 2020, 3:49pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/5 "2020-08-25T15:49:55Z")

</div>

I got 16.6 ROUGE 2 finetuning bart-base on XSUM, in 3 epochs/ 7.5 hrs

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/5/5b15897c6a6428864f613e30598f59446c67bdd7.png) Still way worse than distilbart-xsum-6-6 (20.92) and not that much faster.

---

<div class="post-metadata">

**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\
**Post date:** [September 8, 2020, 11:24am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/6 "2020-09-08T11:24:27Z")

</div>

Hi @patrickvonplaten,  
I tried to reproduce your Bert2GPT2-CNN\_dailymail, but when I train it I get following error message  
`TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'`. The gist of my notebook: [https://gist.github.com/cahya-wirawan/b36e91cae21a6a7f9a10e1c85f59d9ae](https://gist.github.com/cahya-wirawan/b36e91cae21a6a7f9a10e1c85f59d9ae)  
I use also the branch bert2gpt2-cnn\_dailymail-fp16 as suggested. Would be nice if you could point me where I did it wrongly.  
Thanks.

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 27, 2020, 9:40am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/7 "2020-10-27T09:40:32Z")

</div>

Hey @patrickvonplaten ,it seems like the result doesn’t match this paper’s result .

> **[1907.12461.pdf](https://arxiv.org/pdf/1907.12461.pdf)**
>
> 214.58 KB

  
for example ,the roberta2roberta model  
the rouge2 fmeasure in the paper for cnn dataset is 18.5  
but your result is 16.79.

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 27, 2020, 9:45am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/8 "2020-10-27T09:45:58Z")

</div>

The models below were trained without any hyper-parameter search .Does that claim means any other parameter experiment has not been compared ? I also wonder fp16 or fp32 significantly affect the performance of the model ?

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 28, 2020, 10:13am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/9 "2020-10-28T10:13:09Z")

</div>

Hey , do you know why your result is worse than  
[patrickvonplaten/roberta2roberta-cnn\_dailymail-fp16](https://huggingface.co/patrickvonplaten/roberta2roberta-cnn_dailymail-fp16) ?  
I also want to reproduce the result. And I got similar rouge-2 score such as 9.6 or 9.7.  
But in the original paper , the rouge-2 score is 18.9.  
It is weird.  
Do we need to increase the batch size or train more steps ?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 29, 2020, 5:17pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/10 "2020-10-29T17:17:12Z")

</div>

I didn’t investigate it much, @patrickvonplaten will have some ideas about this

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 30, 2020, 5:09am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/11 "2020-10-30T05:09:49Z")

</div>

it seems that tokenizer.batch\_decode  
this method is removed  
I can’t find it in the document

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 31, 2020, 2:42pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/12 "2020-10-31T14:42:15Z")

</div>

Seems like it’s not included in the docs, but it’s available. See

> <https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L2943>

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\
**Post date:** [November 2, 2020, 7:26pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/13 "2020-11-02T19:26:33Z")

</div>

I just added two notebooks showing how to reproduce the results in the paper.  
Here one for Bert2Bert on CNN/Dailymail:

> <https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb>

  
Here one for RobertaShared on BBC XSum:  

> <https://github.com/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb>

The Bert2Bert model actually performs a bit better than reported in the paper, the roberta\_shared model a bit worse (but training roberta\_shared a bit longer would probably close that gap).

The motivation of doing this is to provide some educational material on how to use the `EncoderDecoderModel` - the exact performance was less important here.

I’m planning on making two short notebooks on Roberta2GPT2 for sentence fusion (DiscoFuse) and a Bert2Rnd for WMT en-de. Hope this is useful!

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\
**Post date:** [November 9, 2020, 5:17pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/14 "2020-11-09T17:17:59Z")

</div>

A longer blog post on this topic is online now: [https://huggingface.co/blog/warm-starting-encoder-decoder](https://huggingface.co/blog/warm-starting-encoder-decoder)

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\
**Post date:** [November 10, 2020, 12:30pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/15 "2020-11-10T12:30:06Z")

</div>

Thanks so much for your blog @patrickvonplaten, you are the hero!

---

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [November 17, 2020, 6:30am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/17 "2020-11-17T06:30:33Z")

</div>

How many epoches did you run for full training ?

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\
**Post date:** [November 17, 2020, 2:42pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/18 "2020-11-17T14:42:33Z")

</div>

ran for 3 epochs

---

<div class="post-metadata">

**Author:** ![ncoop57](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ncoop57/32/3107_2.png) [@ncoop57](https://discuss.huggingface.co/u/ncoop57)\
**Post date:** [November 17, 2020, 8:11pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/19 "2020-11-17T20:11:05Z")

</div>

Is it possible to use a pretrained encoder and then an untrained decoder that is just defined by a config (and also vice versa)?

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\
**Post date:** [November 18, 2020, 7:28am UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/20 "2020-11-18T07:28:30Z")

</div>

Yes! In this case I’d recommend loading the encoder / decoder directly.

```python
encoder = AutoModel.from_pretrained(...)
decoder = BertLMHeadModel(BertConfig())

enc_dec_model = EncoderDecoderModel(encoder=encoder, decoder=decoder)

```

---

<div class="post-metadata">

**Author:** ![ncoop57](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ncoop57/32/3107_2.png) [@ncoop57](https://discuss.huggingface.co/u/ncoop57)\
**Post date:** [November 19, 2020, 6:33pm UTC](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/21 "2020-11-19T18:33:48Z")

</div>

Amazing!! Thanks so much!

[Next page](https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835.md?page=2)