# T5 outperforms BART when fine-tuned for summarization task

**URL:** <https://discuss.huggingface.co/t/t5-outperforms-bart-when-fine-tuned-for-summarization-task/20009>\
**Category:** Intermediate\
**Created:** [July 5, 2022, 9:38am UTC](https://discuss.huggingface.co/t/t5-outperforms-bart-when-fine-tuned-for-summarization-task/20009 "2022-07-05T09:38:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sarthak09](https://avatars.discourse-cdn.com/v4/letter/s/e0b2c6/32.png) [@sarthak09](https://discuss.huggingface.co/u/sarthak09)\
**Post date:** [July 5, 2022, 9:38am UTC](https://discuss.huggingface.co/t/t5-outperforms-bart-when-fine-tuned-for-summarization-task/20009/1 "2022-07-05T09:38:43Z")

</div>

I was observing a strange behaviour with the fine-tuned model of BART and T5 on the summarization task.  
I am referring to the following repository:

> **[transformers/examples/pytorch/summarization at main · huggingface/transformers](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization)**
>
> main/examples/pytorch/summarization

**Dataset:**

```auto
It is a collection of dictionaries. Example:-
{"text": "Who is {champion} of {nominee for} {Graduation} ?", "summary": "select ?vr1 where { @@entbegin wd: || graduation @@entend @@relbegin wdt: || nominated for @@relend ?vr0 . ?vr0 @@relbegin wdt: || winner @@relend ?vr1 } }

```

**T5** :  
I used the exact code in the above posted github link.  
Python code for running:  
python examples/pytorch/summarization/run\_summarization.py   
–model\_name\_or\_path t5-small   
–do\_train   
–do\_eval   
–do\_predict   
–train\_file path\_to\_csv\_or\_jsonlines\_file   
–validation\_file path\_to\_csv\_or\_jsonlines\_file   
–source\_prefix "summarize: "   
–output\_dir /tmp/tst-summarization   
–overwrite\_output\_dir   
–per\_device\_train\_batch\_size=4   
–per\_device\_eval\_batch\_size=4   
–predict\_with\_generate

**T5\_result** :

```auto
{

"text": "Who is {champion} of {nominee for} {Graduation}?",

"summary": "select ?vr1 where { @@entbegin wd: || graduation @@entend @@relbegin wdt: || nominated for @@relend ?vr0 . ?vr0 @@relbegin wdt: || winner @@relend ?vr1 }",

"output": "select ?vr1 where { @@entbegin wd: || graduation @@entend @@relbegin wdt: || nominated for @@relend ?vr0 . ?vr0 @@relbegin wdt: || winner @@relend ?vr1 }",

"score": 1.0

}

```

**BART** :  
In the above python code I used the model facebook/bart-base of the huggingface models and so the code to run looks like this:  
python examples/pytorch/summarization/run\_summarization.py   
–model\_name\_or\_path facebook/bart-base   
–do\_train   
–do\_eval   
–do\_predict   
–train\_file path\_to\_csv\_or\_jsonlines\_file   
–validation\_file path\_to\_csv\_or\_jsonlines\_file   
–output\_dir /tmp/tst-summarization   
–overwrite\_output\_dir   
–per\_device\_train\_batch\_size=4   
–per\_device\_eval\_batch\_size=4   
–predict\_with\_generate

(I also removed the source\_prefix argument from the python running code compared to what was used in T5’s)

**BART\_result on the same example** :

```auto
{'text': 'Who is {champion} of {nominee for} {Graduation} ?',
 'summary': 'select ?vr1 where { @@entbegin wd: || graduation @@entend @@relbegin wdt: || nominated for @@relend ?vr0 . ?vr0 @@relbegin wdt: || winner @@relend ?vr1 } ', 
 'output': 'voy go devoidptper n number now when<unk> market said a Services Ambassador<unk> imagine SmithuptperEverybodyptper," will number now while<unk> women byptper, will leave Elev<unk> the Elev<unk> we said there to a will Aviation Mori<unk> the soccer an nice an<unk> will reg an Mori<unk> which<unk>,<unk> from market from<unk> which Golden<unk> from: from<unk> thatVict whichVict identified Co which<unk> the Soccer an adjacent an Mori which GoldenVict identified Co<unk> whichVict the soccer AN Mori<unk> we declared<unk> to an<unk> which identified W which', 
'score': 0.0} 

```

As evident bart results are gibberish. I don’t understand what’s the reason behind this. Is there some other way to fine-tune BART model compared to T5?  
Any lead would be helpful.  
TIA!

---

<div class="post-metadata">

**Author:** ![chrisdoyle](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyle/32/5435_2.png) [@chrisdoyle](https://discuss.huggingface.co/u/chrisdoyle)\
**Post date:** [July 20, 2022, 10:58am UTC](https://discuss.huggingface.co/t/t5-outperforms-bart-when-fine-tuned-for-summarization-task/20009/2 "2022-07-20T10:58:48Z")

</div>

First observation - gradients are likely being updated somewhere because a default BART mightn’t spurt out a load of gibberish. I’d start by double checking that all the special characters are recongised as tokens by the tokeniser - you might need to add them 🙂

---

<div class="post-metadata">

**Author:** ![sarthak09](https://avatars.discourse-cdn.com/v4/letter/s/e0b2c6/32.png) [@sarthak09](https://discuss.huggingface.co/u/sarthak09)\
**Post date:** [July 20, 2022, 11:39am UTC](https://discuss.huggingface.co/t/t5-outperforms-bart-when-fine-tuned-for-summarization-task/20009/3 "2022-07-20T11:39:16Z")

</div>

Hi I later on used the fariseq repository for summarization on bart model which gives better results than the huggingface but still T5 is more accurate. I wonder what can be the reason for this.  
Check the latest bart result:  
“text”: “Who is {champion} of {nominee for} {Graduation}?”,

“summary”: “select ?vr1 where { @@entbegin wd: || graduation @@entend @@relbegin wdt: || nominated for @@relend ?vr0 . ?vr0 @@relbegin wdt: || winner @@relend ?vr1 }”,

“BART-output”: ?vr1 where { @@entbegin wd: || josephe @@entend @@relbegin wdt: || nominated for @@relend ?vr0 . ?vr0 @@relbegin wdt: || winner @@relend ?vr1 }

Any clue why T5 is having a better output than bart?

---

<div class="post-metadata">

**Author:** ![chrisdoyle](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyle/32/5435_2.png) [@chrisdoyle](https://discuss.huggingface.co/u/chrisdoyle)\
**Post date:** [August 8, 2022, 4:15pm UTC](https://discuss.huggingface.co/t/t5-outperforms-bart-when-fine-tuned-for-summarization-task/20009/4 "2022-08-08T16:15:00Z")

</div>

> [@chrisdoyle](#):
>
> I’d start by double checking that all the special characters are recongised as tokens by the tokeniser - you might need to add them 🙂

Different tokenisers maybe - I’d start by double checking that all the special characters are recongised as tokens by the tokeniser - you might need to add them to the BART tokenizer  
🙂