# Implementation difference between Bert and Roberta ForSequenceClassification?

**URL:** <https://discuss.huggingface.co/t/implementation-difference-between-bert-and-roberta-forsequenceclassification/7216>\
**Category:** 🤗Transformers\
**Created:** [June 24, 2021, 6:52am UTC](https://discuss.huggingface.co/t/implementation-difference-between-bert-and-roberta-forsequenceclassification/7216 "2021-06-24T06:52:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vijayabhaskar](https://avatars.discourse-cdn.com/v4/letter/v/df788c/32.png) [@Vijayabhaskar](https://discuss.huggingface.co/u/Vijayabhaskar)\
**Post date:** [June 24, 2021, 6:52am UTC](https://discuss.huggingface.co/t/implementation-difference-between-bert-and-roberta-forsequenceclassification/7216/1 "2021-06-24T06:52:17Z")

</div>

Hi everyone, I was looking into the source code for `BertForSequenceClassification` and `RobertaForSequenceClassification` and found that the `BertForSequenceClassification` uses the `pooled_output` produced by the `BertPooler` to generate the head, on the other hand `RobertaForSequenceClassification` uses the `sequence_output` instead of using the `pooled_output` but reimplements the same with an additional dropout layer to generate the head.  
If I unravel and compare the code, you can see an extra dropout layer.  
 ![bertcompare](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/a439a3dc98918eddd1cd1d6a72aa8e6721d3e083.png)

Why not simply use the `RobertaPooler`/`pooled_output` like Bert, What is the significance of this extra dropout layer?