# Failed to train bart-cnn from bart-base using my own code

**URL:** <https://discuss.huggingface.co/t/failed-to-train-bart-cnn-from-bart-base-using-my-own-code/1393>\
**Category:** Beginners\
**Created:** [October 7, 2020, 2:56am UTC](https://discuss.huggingface.co/t/failed-to-train-bart-cnn-from-bart-base-using-my-own-code/1393 "2020-10-07T02:56:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![LiuYangyang](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@LiuYangyang](https://discuss.huggingface.co/u/LiuYangyang)\
**Post date:** [October 7, 2020, 2:56am UTC](https://discuss.huggingface.co/t/failed-to-train-bart-cnn-from-bart-base-using-my-own-code/1393/1 "2020-10-07T02:56:27Z")

</div>

**Update: problem solved. caused by a bug in model saving function**

There’re not much difference on Rouge score before and after training.  
The generated summarization of same article is almost same.  
Need some suggestion.

I used original setting in [fairseq](https://github.com/pytorch/fairseq/blob/master/examples/bart/README.summarization.md).

- `label_smoothed_cross_entropy`,
- `label-smoothing` 0.1,
- `attention-dropout` 0.1,
- `weight-decay` 0.01,
- `lr-scheduler` polynomial\_decay,
- `adam-betas` “(0.9, 0.999)”
- `adam-eps` 1e-08,
- `LR`=3e-05

**some difference:**

- batch-size 3,
- gradient\_accumulation\_steps 16,
- adamW,
- fp32,
- clip-norm 1.0,

trained for 7 hours on 2 tesla P4.

Loss (my fault made it hard to read):

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/b/bdf784d6aac018f146d7676403e91bbbe8b90565.png)

Model seems not converge .

Rouge score before training:

> “rouge1”: 37.325248080026086,  
> “rouge2”: 18.03751262341448,  
> “rougeL”: 25.82438158757282

Rouge score after training:

> “rouge1”: 37.818,  
> “rouge2”: 17.246,  
> “rougeL”: 23.7038

**Update: problem solved. caused by a bug in model saving function**

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 7, 2020, 4:15pm UTC](https://discuss.huggingface.co/t/failed-to-train-bart-cnn-from-bart-base-using-my-own-code/1393/2 "2020-10-07T16:15:12Z")

</div>

AFAIK there isn’t bart-base checkpoint trained for summarization, how did you get those metrics without training ?

---

<div class="post-metadata">

**Author:** ![LiuYangyang](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@LiuYangyang](https://discuss.huggingface.co/u/LiuYangyang)\
**Post date:** [October 8, 2020, 1:55am UTC](https://discuss.huggingface.co/t/failed-to-train-bart-cnn-from-bart-base-using-my-own-code/1393/3 "2020-10-08T01:55:30Z")

</div>

I directly calculate rouge score on cnn/mail before training.

---

<div class="post-metadata">

**Author:** ![LiuYangyang](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@LiuYangyang](https://discuss.huggingface.co/u/LiuYangyang)\
**Post date:** [October 8, 2020, 1:56am UTC](https://discuss.huggingface.co/t/failed-to-train-bart-cnn-from-bart-base-using-my-own-code/1393/4 "2020-10-08T01:56:26Z")

</div>

I think there’re bugs in my code, but have no idea what kind of bug would cause the problem.