# Using Roberta for Sentence2Vec

**URL:** <https://discuss.huggingface.co/t/using-roberta-for-sentence2vec/5413>\
**Category:** Intermediate\
**Created:** [April 11, 2021, 9:26am UTC](https://discuss.huggingface.co/t/using-roberta-for-sentence2vec/5413 "2021-04-11T09:26:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](https://discuss.huggingface.co/u/Maimonator)\
**Post date:** [April 11, 2021, 9:26am UTC](https://discuss.huggingface.co/t/using-roberta-for-sentence2vec/5413/1 "2021-04-11T09:26:07Z")

</div>

Hey, I’ve been trying to train Sentence2Vec embeddings and I’ve been wondering what do you think about my approach.  
I would be glad to learn about different possible pitfalls in my approach and how to solve them.

#### What do I have?

1. I have a small unique corpus of about 4 million sentences in my test language
2. I have a smaller subset (150k) of labeled sentences of whether the sentence is toxic or not.

For this question’s sake assume there isn’t any pretrained model or another existing corpus I can use.

#### What am I trying to achieve?

The target is to be able to cluster sentences that have similar meanings together.

#### Way of Action

1. Train a RoBERTa language model
2. Fine tune it for classification of toxic or not
3. Use a hidden layer of that classifier as an embedding model
4. Cluster using embeddings and HDBSCAN.

That’s about it, I tried to keep it as clear as I can.  
Thanks for anyone that read up until here!!

---

<div class="post-metadata">

**Author:** ![Simpan](https://avatars.discourse-cdn.com/v4/letter/s/dfb087/32.png) [@Simpan](https://discuss.huggingface.co/u/Simpan)\
**Post date:** [April 11, 2021, 9:42am UTC](https://discuss.huggingface.co/t/using-roberta-for-sentence2vec/5413/2 "2021-04-11T09:42:00Z")

</div>

What is your main goal? Is it to have a classifier for toxic vs non-toxic sentences? Or to cluster sentence embeddings by semantic meaning?

For creating sentence embeddings I would recommend [sentence transformers](https://github.com/UKPLab/sentence-transformers). It’s an extensions of regular huggingface transformers but optimized for creating text embeddings.

---

<div class="post-metadata">

**Author:** ![Maimonator](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maimonator/32/1452_2.png) [@Maimonator](https://discuss.huggingface.co/u/Maimonator)\
**Post date:** [April 11, 2021, 10:07am UTC](https://discuss.huggingface.co/t/using-roberta-for-sentence2vec/5413/3 "2021-04-11T10:07:53Z")

</div>

Cool!  
Will take a look at that!  
Is there a recommended way to finetune embeddings created by `sentence-transofmers`?

---

<div class="post-metadata">

**Author:** ![Simpan](https://avatars.discourse-cdn.com/v4/letter/s/dfb087/32.png) [@Simpan](https://discuss.huggingface.co/u/Simpan)\
**Post date:** [April 11, 2021, 10:31am UTC](https://discuss.huggingface.co/t/using-roberta-for-sentence2vec/5413/4 "2021-04-11T10:31:03Z")

</div>

Yes there is sample code in the repo 🙂 Would also recommend to check out the repo homepage [https://www.sbert.net/](https://www.sbert.net/) for good explanations of different use cases.