# Classifier Dropout for \*DecoderModel\*ForSequenceClassification Classes

**URL:** <https://discuss.huggingface.co/t/classifier-dropout-for-decodermodel-forsequenceclassification-classes/113989>\
**Category:** 🤗Transformers\
**Created:** [October 25, 2024, 5:56pm UTC](https://discuss.huggingface.co/t/classifier-dropout-for-decodermodel-forsequenceclassification-classes/113989 "2024-10-25T17:56:45Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![alphacentauri22](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@alphacentauri22](https://discuss.huggingface.co/u/alphacentauri22)\
**Post date:** [October 25, 2024, 5:56pm UTC](https://discuss.huggingface.co/t/classifier-dropout-for-decodermodel-forsequenceclassification-classes/113989/1 "2024-10-25T17:56:45Z")

</div>

A common practice for doing classification on top of encoder only model is to add a classification head on top of pooled outputs from the encoder. The pooled outputs usually go through a dropout layer before the linear classification layer and the dropout probability is specified by `classifier_dropout`.

However, I see that across a few common decoder models ([Mistral](https://github.com/huggingface/transformers/blob/v4.46.0/src/transformers/models/mistral/modeling_mistral.py#L1126), [Llama](https://github.com/huggingface/transformers/blob/v4.46.0/src/transformers/models/llama/modeling_llama.py#L1244), [Phi3](https://github.com/huggingface/transformers/blob/main/src/transformers/models/phi3/modeling_phi3.py#L1346)), the sequence classification head (in the \*Model\*ForSequenceClassification class) does not have a dropout layer. Moreover, dropout IS available for the token\_classification heads though.

Is there any underlying reason why dropout is not used for sequence classification but is available for token classification?