# TFBertModel for classification task with no CLS token

**URL:** <https://discuss.huggingface.co/t/tfbertmodel-for-classification-task-with-no-cls-token/33594>\
**Category:** Beginners\
**Created:** [March 11, 2023, 6:21pm UTC](https://discuss.huggingface.co/t/tfbertmodel-for-classification-task-with-no-cls-token/33594 "2023-03-11T18:21:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![BaroneRampante](https://avatars.discourse-cdn.com/v4/letter/b/3e96dc/32.png) [@BaroneRampante](https://discuss.huggingface.co/u/BaroneRampante)\
**Post date:** [March 11, 2023, 6:21pm UTC](https://discuss.huggingface.co/t/tfbertmodel-for-classification-task-with-no-cls-token/33594/1 "2023-03-11T18:21:24Z")

</div>

Hello, I’m reading a paper where BERT (`TFBertModel`) and RoBerta (`TFRobertaModel`) are used to solve a text classification task.

**Preamble**  
Going through the implementation, I noticed that each text sample is tokenized with no special tokens `tokenizer.encode(sentence, add_special_tokens=False)`.

Later on the outputs of the tokenizers are passed to the respective models and the _pooled output_ is retrieved, as follows:

```auto
embedding_BERT = encoder_BERT(
    input_ids_BERT,
    token_type_ids=token_type_ids_BERT,
    attention_mask=attention_mask_BERT
)['pooler_output']

```

**Questions**

1. The authors claim to be using the `[CLS]` tokens produced by both models. However, how can this be the case if the tokenizers encoded the text samples without including the special tokens?
2. If `add_special_tokens` is `False`, does the first token of each text sample still encode knowledge about the whole sequence as it usually is the case with `[CLS]`?
3. The authors actually use the _pooled output_, which is produced by `BertPooler`. Can its output still be considered as the `CLS` token?

**References**  
paper: [https://ceur-ws.org/Vol-3202/politices-paper1.pdf](https://ceur-ws.org/Vol-3202/politices-paper1.pdf)  
code: [PoliticES2022/PoliticES.ipynb at main · ssantamaria94/PoliticES2022 · GitHub](https://github.com/ssantamaria94/PoliticES2022/blob/main/PoliticES.ipynb)