# Generating sentence embeddings from pretrained transformers model

**URL:** <https://discuss.huggingface.co/t/generating-sentence-embeddings-from-pretrained-transformers-model/3314>\
**Category:** Intermediate\
**Created:** [January 20, 2021, 7:45pm UTC](https://discuss.huggingface.co/t/generating-sentence-embeddings-from-pretrained-transformers-model/3314 "2021-01-20T19:45:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![prithvisrinivasan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prithvisrinivasan/32/620_2.png) [@prithvisrinivasan](https://discuss.huggingface.co/u/prithvisrinivasan)\
**Post date:** [January 20, 2021, 7:45pm UTC](https://discuss.huggingface.co/t/generating-sentence-embeddings-from-pretrained-transformers-model/3314/1 "2021-01-20T19:45:10Z")

</div>

Hi, I have a pretrained BERT based model hosted on huggingface.

> **[microsoft/SportsBERT · Hugging Face](https://huggingface.co/microsoft/SportsBERT)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

How do I generate sentence vectors using this model? I have explored sentence bert but it doesn’t allow you to use custom trained models. I have also seen Bert as a client. It works but for my current scenario, I was wondering if there’s something which could be done without running a server for converting to vectors.

---

<div class="post-metadata">

**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\
**Post date:** [January 22, 2021, 8:31am UTC](https://discuss.huggingface.co/t/generating-sentence-embeddings-from-pretrained-transformers-model/3314/2 "2021-01-22T08:31:42Z")

</div>

I think the best method is to go with sentence-bert. Indeed, you can use your own model, just try reproducing what they do in the [paper](https://arxiv.org/abs/1908.10084).  
You add a pooling layer at the output of your model, from the paper:

> We experiment with three pooling strategies: Using the output of the CLS-token,computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of the output vectors (MAX-strategy). The default configuration is MEAN.

Finally, you might want to fine-tune your model for comparison, too:

> In order to fine-tune BERT / RoBERTa **(your model)**, we create siamese and triplet networks (Schroff et al.,2015) to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity.

Hope this help!