# Info regarding sentence-transformers

**URL:** <https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808>\
**Category:** Models\
**Created:** [August 21, 2020, 6:35am UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808 "2020-08-21T06:35:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 21, 2020, 6:35am UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/1 "2020-08-21T06:35:44Z")

</div>

`sentence-transformers` were recently added in the model hub. I wanted to know the following if :

- all these models were trained on NLI (which dataset ? MNLI)
- for how many epochs ?
- If I want to use it for inference on NLI task, do I need to train an additional linear layer ? So freeze the `AutoModel`, train a `nn.Linear` for 1 epoch on the `mean` output and then perform inference. Can you please confirm this usage ?

```auto
tfmr = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")

class Model(nn.Module):
    def __init__ (self, model):
        super(model, self). __init__ ()
        self.encoder = model
        for param in self.encoder.parameters():
            param.requires_grad = False
        self.classifier = nn.Linear(768, 3)
        self.criterion = nn.CrossEntropy()

    def forward(self, inputs): # x will come from dataloader using default_data_collator

        labels = inputs.pop('labels')
        model_output = self.encoder(**inputs)
        sentence_embeddings = mean_pooling(model_output, inputs['attention_mask'])
        logits = self.classifier(sentence_embeddings)
        loss = self.criterion(logits, labels)
        return loss, logits

```

```auto
model = Model(tfmr)

```

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 21, 2020, 1:20pm UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/2 "2020-08-21T13:20:00Z")

</div>

I trained it on MNLI with the model frozen, it gets around 46% accuracy after epoch 1. Seems like it has not been trained on MNLI. Can anyone confirm ?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 21, 2020, 3:57pm UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/3 "2020-08-21T15:57:35Z")

</div>

from the paper, section 3.1

> We train SBERT on the combination of the SNLI  
> (Bowman et al., 2015) and the Multi-Genre NLI dataset.

So yes, it’s trained on MNLI .

My guess is as it’s trained using Triplet loss to produce embeddings that are semantically meaningful  
and can be compared with cosine-similarity, it might nor perform well on just classification .

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 22, 2020, 2:43am UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/4 "2020-08-22T02:43:01Z")

</div>

Thanks for the information. When I read the paper, I initially had the impression that it was trained on STS task. Given that it has been trained on both SNLI and MNLI (not sure about the percentage), I’m getting 46% accuracy when I train a MLP for one epoch. I know the objective is a bit different (not classification) but I’d expect it to do well (in frozen condition) given that it has seen both NLI datasets. Things don’t improve much even after 3 epochs.

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 23, 2020, 11:18am UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/5 "2020-08-23T11:18:40Z")

</div>

On what objective is this model trained on ? Section 3 of the paper mentions three objective functions `classification`, `regression` and `triplet`. As per Fig 2. they seem to be using regression for computing similarity scores at inference. As per Fig 1. it says that they used `classification` objective for fine-tuning. Based on the performance I’m getting, I doubt that these weights are coming from model trained on `classification`. Can someone please clarify ?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 23, 2020, 4:57pm UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/6 "2020-08-23T16:57:08Z")

</div>

I have the same confusion, it’s not clear what was the final objective from those 3

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 25, 2020, 6:16am UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/7 "2020-08-25T06:16:06Z")

</div>

@joeddav If you’ve some idea, can you please confirm this ?

---

<div class="post-metadata">

**Author:** ![joeddav](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joeddav/32/53881_2.png) [@joeddav](https://discuss.huggingface.co/u/joeddav)\
**Post date:** [August 25, 2020, 2:28pm UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/8 "2020-08-25T14:28:16Z")

</div>

I’m afraid I don’t have any extra insight here. I might head over to the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) repo and see if you can find an answer or open an issue there (and then loop us back in over here once you have an answer 🤗)

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 26, 2020, 4:21am UTC](https://discuss.huggingface.co/t/info-regarding-sentence-transformers/808/9 "2020-08-26T04:21:21Z")

</div>

I’ve opened an [issue](https://github.com/UKPLab/sentence-transformers/issues/378). You can follow it if interested.