# Using Accelerated Inference API to produce sentense embeddings

**URL:** <https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223>\
**Category:** 🤗Transformers\
**Created:** [May 18, 2021, 4:39am UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223 "2021-05-18T04:39:38Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitali](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vitali](https://discuss.huggingface.co/u/vitali)\
**Post date:** [May 18, 2021, 4:39am UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/1 "2021-05-18T04:39:38Z")

</div>

Is it possible to use [Accelerated Inference API](https://api-inference.huggingface.co/docs/python/html/index.html#accelerated-inference-api) to produce sentense embeddings as described [here](https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens)?

```
from transformers import AutoTokenizer, AutoModel
import torch

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
    return sum_embeddings / sum_mask

#Sentences we want sentence embeddings for
sentences = ['This framework generates embeddings for each input sentence',
             'Sentences are passed as a list of string.',
             'The quick brown fox jumps over the lazy dog.']

#Load AutoModel from huggingface model repository
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")
model = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")

#Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')

#Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

#Perform pooling. In this case, mean pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
```

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [May 18, 2021, 1:08pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/2 "2021-05-18T13:08:39Z")

</div>

hey @vitali i believe integration with `sentence-transformers` in the inference API is currently in progress, so maybe @osanseviero can share some details (or whether it’s currently possible)

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [May 18, 2021, 2:23pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/3 "2021-05-18T14:23:04Z")

</div>

Hi @vitali!

Your question comes in good time. You can already do this by calling  
`https://api-inference.huggingface.co/pipeline/feature-extraction/MODEL_ID`. This endpoint is in **experimental** state at the moment, so things might not be stable.

Note that, as of now, we’re working on deeply integrating `sentence-transformers` with the Hub. This will be part of the v2 release of the library. Some details:

- Allow downloading sentence-transformer models from the Hub ([PR](https://github.com/UKPLab/sentence-transformers/pull/908), merged).
- Allow uploading sentence-transformer models from the Hub ([PR](https://github.com/UKPLab/sentence-transformers/pull/925)).

We expect to have more exciting results very soon 🙂

---

<div class="post-metadata">

**Author:** ![vitali](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vitali](https://discuss.huggingface.co/u/vitali)\
**Post date:** [May 18, 2021, 3:08pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/4 "2021-05-18T15:08:45Z")

</div>

Awesome! I will try to it.

---

<div class="post-metadata">

**Author:** ![vitali](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vitali](https://discuss.huggingface.co/u/vitali)\
**Post date:** [May 24, 2021, 10:38pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/5 "2021-05-24T22:38:27Z")

</div>

Can I infer from your answer that any pipeline can be inferred like this? 🙂 That would be totally awesome.

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [May 25, 2021, 7:19am UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/6 "2021-05-25T07:19:08Z")

</div>

The pipeline is usually inferred from the tags in the model repo. Forcing the pipeline through the API has a risk of misusing a model for a different pipeline, but there are also certain models that support multiple pipelines, so you can use that as well.

---

<div class="post-metadata">

**Author:** ![vitali](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vitali](https://discuss.huggingface.co/u/vitali)\
**Post date:** [May 25, 2021, 2:06pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/7 "2021-05-25T14:06:07Z")

</div>

So If I add the tag “feature-extraction” in the model repo then call to inference API will produce embeddings? what will happen if I add multiple tags, i.e “feature-extraction”, “fill-mask”, “zero-shot-classification”?

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [May 27, 2021, 9:57am UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/8 "2021-05-27T09:57:21Z")

</div>

Hi @vitali. We currently try to keep things simple: we usually have 1 task per model, and this holds for most models. But in case your model does support other tasks, you can use the API url as above.

Right now there’s no way to validate that the model will work with the task, so that’s why this is not shared widely - it might lead to misusing and getting incorrect results.

---

<div class="post-metadata">

**Author:** ![vitali](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vitali](https://discuss.huggingface.co/u/vitali)\
**Post date:** [May 27, 2021, 4:02pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/9 "2021-05-27T16:02:56Z")

</div>

Thank you very much for your reply, I understand the issue: every transformer model can serve some tasks (make embeddings, MLM, zero-shot classification) but no model can serve all tasks, thus comes a risk of misuse. Perhaps it would make sense to add “capabilities.json” to the model repo to define the list of supported tasks/pipelines based on the model architecture? I think this would clear some confusion amongst users, just a thought. Anyway, this way solves my need to make embeddings for downstream tasks perfectly, thank you very much again.

---

<div class="post-metadata">

**Author:** ![vitali](https://avatars.discourse-cdn.com/v4/letter/v/bc8723/32.png) [@vitali](https://discuss.huggingface.co/u/vitali)\
**Post date:** [May 28, 2021, 3:01pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/10 "2021-05-28T15:01:08Z")

</div>

For community reference, the issue of defining and using model pipelines is also discussed on [github](https://github.com/huggingface/transformers/issues/11913#issuecomment-850425615).

---

<div class="post-metadata">

**Author:** ![abol3z](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@abol3z](https://discuss.huggingface.co/u/abol3z)\
**Post date:** [December 7, 2021, 10:01am UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/11 "2021-12-07T10:01:23Z")

</div>

Hey Omar, is this still an experimental API? I can’t seem to find any details about it. Would appreciate you pointing me to some resources if available as I am evaluating some APIs and would like to test yours. Thanks.

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [December 8, 2021, 10:32am UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/12 "2021-12-08T10:32:13Z")

</div>

Hey @abol3z.

The API is not in experimental anymore, but we’re working in its documentation. You can use [https://api-inference.huggingface.co/pipeline/feature-extraction/MODEL\_ID](https://api-inference.huggingface.co/pipeline/feature-extraction/MODEL_ID) to obtain the sentence embeddings.

Let us know if you have any questions!

---

<div class="post-metadata">

**Author:** ![adbo28](https://avatars.discourse-cdn.com/v4/letter/a/c77e96/32.png) [@adbo28](https://discuss.huggingface.co/u/adbo28)\
**Post date:** [April 23, 2022, 3:03pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/13 "2022-04-23T15:03:35Z")

</div>

Hello Omar, I was trying to load embeddings via API as discussed in this thread, but I am struggling to find a model that actually supports this method.

So for example, the model [https://api-inference.huggingface.co/models/sentence-transformers/all-MiniLM-L6-v2](https://api-inference.huggingface.co/models/sentence-transformers/all-MiniLM-L6-v2) supports feature extraction, but the following URL is invalid:  
[https://api-inference.huggingface.co/pipeline/feature-extraction/sentence-transformers/all-MiniLM-L6-v2](https://api-inference.huggingface.co/pipeline/feature-extraction/sentence-transformers/all-MiniLM-L6-v2)

Can you help?  
Or maybe share a sample code snippet?

Thanks.

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [April 23, 2022, 6:54pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/14 "2022-04-23T18:54:56Z")

</div>

Hi there! 👋 Here is a working end-to-end example for the model you suggested.

```python
import requests

API_URL = " https://api-inference.huggingface.co/pipeline/feature-extraction/sentence-transformers/all-MiniLM-L6-v2"
headers = {"Authorization": "Bearer TOKEN"}

def query(payload):
	response = requests.post(API_URL, headers=headers, json=payload)
	return response.json()
	
output = query({
	"inputs": "I like you. I love you",
})

```

---

<div class="post-metadata">

**Author:** ![aslasdlkj](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aslasdlkj/32/11801_2.png) [@aslasdlkj](https://discuss.huggingface.co/u/aslasdlkj)\
**Post date:** [September 22, 2022, 9:53am UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/15 "2022-09-22T09:53:22Z")

</div>

Thanks sm! this helped a lot.

---

<div class="post-metadata">

**Author:** ![mtomov](https://avatars.discourse-cdn.com/v4/letter/m/bbce88/32.png) [@mtomov](https://discuss.huggingface.co/u/mtomov)\
**Post date:** [April 12, 2023, 6:10pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/16 "2023-04-12T18:10:22Z")

</div>

Hello,

I’ve used the feature extraction pipeline successfully with sentance-trasformer models.

How do I use it with a model, which requires `mean_pooling` to be applied to the result, such as [`E5`](https://huggingface.co/intfloat/e5-large)?

To clarify: get a single array of feature embeddings vs the current result that comes out - 3 arrays for the word “test”.

Thank you!

---

<div class="post-metadata">

**Author:** ![radames](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/radames/32/28246_2.png) [@radames](https://discuss.huggingface.co/u/radames)\
**Post date:** [April 12, 2023, 7:25pm UTC](https://discuss.huggingface.co/t/using-accelerated-inference-api-to-produce-sentense-embeddings/6223/17 "2023-04-12T19:25:28Z")

</div>

hi @mtomov, you might need a [custom pipeline](https://huggingface.co/radames/e5-large/blob/main/pipeline.py) to process the result, here is a duplicated model implementing the mean\_pooling on the request [radames/e5-large · Hugging Face](https://huggingface.co/radames/e5-large), you can also try

```python
import requests

API_URL = "https://api-inference.huggingface.co/models/radames/e5-large"
headers = {"Authorization": "Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"}

def query(payload):
    response = requests.post(API_URL, headers={}, json=payload)
    return response.json()

embeddings = query({
    "inputs": "query: how much protein should a female eat",
})
print(embeddings)

```