# Sentence embeddings generated by RoBERTa

**URL:** <https://discuss.huggingface.co/t/sentence-embeddings-generated-by-roberta/24894>\
**Category:** 🤗Transformers\
**Created:** [October 24, 2022, 4:01pm UTC](https://discuss.huggingface.co/t/sentence-embeddings-generated-by-roberta/24894 "2022-10-24T16:01:41Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![payamdig](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/payamdig/32/12216_2.png) [@payamdig](https://discuss.huggingface.co/u/payamdig)\
**Post date:** [October 24, 2022, 4:01pm UTC](https://discuss.huggingface.co/t/sentence-embeddings-generated-by-roberta/24894/1 "2022-10-24T16:01:41Z")

</div>

Hi, I am trying to understand how sentence embeddings are generated in RoBERTa. What I’ve understood so far is that in BERT, sentence embeddings are created by training a CLS token using the Next Sentence Prediction (NSP) task. However, in RoBERTa the NSP objective is dropped, but we still have sentence embeddings. I have failed to find a resource that explains how we can still have sentence embeddings while that objective is dropped. I am new to the subject, so there might be many misunderstandings on my part, so any clarification will be helpful and appreciated 🙂