# BertModel.forward() output caveat removed? **URL:** \ **Category:** Models\ **Created:** [September 4, 2020, 12:14am UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983 "2020-09-04T00:14:04Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![facehugger2020](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/facehugger2020/32/3052_2.png) [@facehugger2020](https://discuss.huggingface.co/u/facehugger2020)\ **Post date:** [September 4, 2020, 12:14am UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983/1 "2020-09-04T00:14:04Z")
In Transformers 3.0.2 and prior, there used to be a small _caveat_ accompanying the description of [3.0.2 BertModel.forward()](https://huggingface.co/transformers/v3.0.2/model_doc/bert.html#transformers.BertModel.forward): > **Returns** > > pooler\_output (`torch.FloatTensor`: of shape `(batch_size, hidden_size)`): > > Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pre-training. > > _This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence_. However, the caveat was removed in 3.1.0 (current master). The description of [3.1.0 BertModel.forward()](https://huggingface.co/transformers/model_doc/bert.html#transformers.BertModel.forward) now just says: > **Returns** > > pooler\_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) - Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining. Is there any deeply meaningful reason why that line was removed? Is there a new important secret to that `[CLS]` token that you’re not telling us? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [September 4, 2020, 3:03am UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983/2 "2020-09-04T03:03:08Z")
That caveat was removed because this output is actually what is used for classification in the `Sequence Classification` model and experiments showed it was given the same kinds of results as the mean/average. ---
**Author:** ![facehugger2020](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/facehugger2020/32/3052_2.png) [@facehugger2020](https://discuss.huggingface.co/u/facehugger2020)\ **Post date:** [September 4, 2020, 10:23pm UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983/3 "2020-09-04T22:23:22Z")
@sgugger May I ask why that statement was written in the first place? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [September 5, 2020, 1:49pm UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983/4 "2020-09-05T13:49:34Z")
No one seems to remember… ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [September 5, 2020, 2:49pm UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983/5 "2020-09-05T14:49:04Z")
It’s probably because of the original paper. In the BERT paper, they discuss how NSP is beneficial for QA and NLI but that “[t]he vector C is not a meaningful sentence representation without fine-tuning, since it was trained with NSP”, where C is “the final hidden vector of the special [CLS] token” (p. 4). In other words, CLS token without finetuning might not be a semantic representation of a given sentence because it was pretrained only on the NSP task. To use it in your own downstream task, you should finetune it. If you only use BERT as a feature extractor (e.g. for sentence embeddings), then you are probably better off by using mean/avg pooling. See this Twitter thread and the paper reference in there: [https://twitter.com/JohnMGiorgi/status/1295472684353105920](https://twitter.com/JohnMGiorgi/status/1295472684353105920) ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [September 5, 2020, 7:05pm UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983/6 "2020-09-05T19:05:29Z")
Not sure how a pooling would give you descent results without fine-tuning either, but this makes sense. ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [September 5, 2020, 7:42pm UTC](https://discuss.huggingface.co/t/bertmodel-forward-output-caveat-removed/983/7 "2020-09-05T19:42:01Z")
Well, intuitively you would think that taking the average over all the last hidden states of all tokens captures a bit better the essence of the sentence than just taking the CLS token. The meaning of tokens after pretraining are usable (just take average and you have a relatively decent semantic representation), but CLS is specific to the NSP task.