# Deploying Sentence Transformer as sagemaker endpoint **URL:** \ **Category:** Amazon SageMaker\ **Created:** [October 14, 2022, 5:53pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455 "2022-10-14T17:53:42Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![amitkayal](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@amitkayal](https://discuss.huggingface.co/u/amitkayal)\ **Post date:** [October 14, 2022, 5:53pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/1 "2022-10-14T17:53:42Z")
Hello, Has anyone deployed sentence transformer into sagemaker as endpoint? The link provides task based approach [Deploy Hugging Face models easily with Amazon SageMaker](https://huggingface.co/blog/deploy-hugging-face-models-easily-with-amazon-sagemaker) but this is only applicable for nlp pipeline and I cant apply for sentence transformer. Thanks ---
**Author:** ![philondrejack](https://avatars.discourse-cdn.com/v4/letter/p/82dd89/32.png) [@philondrejack](https://discuss.huggingface.co/u/philondrejack)\ **Post date:** [October 14, 2022, 8:31pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/2 "2022-10-14T20:31:18Z")
I’m going through the exact same thing. I’ve found [this](https://www.philschmid.de/custom-inference-huggingface-sagemaker). It works, and I have a serverless inference endpoint deployed. I’d prefer to use the sentence-transformer package as I’m not familiar with how to work with AutoTokenizer. ---
**Author:** ![amitkayal](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@amitkayal](https://discuss.huggingface.co/u/amitkayal)\ **Post date:** [October 15, 2022, 1:31pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/3 "2022-10-15T13:31:49Z")
Thanks a lot for this nice article. So we need to change to ensure this is created as serverless? > ```auto > > ``` ```auto # deploy the endpoint endpoint predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.g4dn.xlarge" ) ``` ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [October 17, 2022, 5:41am UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/4 "2022-10-17T05:41:02Z")
Yes by changing the `instance` configuration you can deploy the model as serverless endpoint. ---
**Author:** ![remigabillet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/remigabillet/32/16149_2.png) [@remigabillet](https://discuss.huggingface.co/u/remigabillet)\ **Post date:** [May 15, 2023, 8:30am UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/5 "2023-05-15T08:30:12Z")
@philschmid thanks a lot for putting together the guide. I followed the instructions and the Sagemaker endpoint working. The problem is that I’m seeing very low performance. - on the ml.g4dn.xlarge endpoint deployed using instructions: 3 vectors/second - on a SageMaker studio ml.g4dn.xlarge notebook using CPU: 50 vectors/second - on a SageMaker studio ml.g4dn.xlarge notebook using GPU: 800 vectors/second Any ideas why performance is so much lower on the endpoint? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [May 15, 2023, 10:44am UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/6 "2023-05-15T10:44:07Z")
Did you use a custom `inference.py`? Can you please share the code how you deployed the model? Don’t forget there is network overhead when you send requests from a notebook environment to your endpoint, but 20x/250x sounds way too much. ---
**Author:** ![remigabillet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/remigabillet/32/16149_2.png) [@remigabillet](https://discuss.huggingface.co/u/remigabillet)\ **Post date:** [May 15, 2023, 6:09pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/7 "2023-05-15T18:09:03Z")
Thanks for the quick follow-up @philschmid. I copy/pasted the code from the article you wrote but here is the code in case you notice something. I ran it on a Sagemaker Studio notebook. > ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [May 16, 2023, 8:23am UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/8 "2023-05-16T08:23:35Z")
You are not moving your model to gpu ---
**Author:** ![remigabillet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/remigabillet/32/16149_2.png) [@remigabillet](https://discuss.huggingface.co/u/remigabillet)\ **Post date:** [May 16, 2023, 9:03am UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/9 "2023-05-16T09:03:06Z")
Can you clarify what I’m supposed to do differently? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [May 16, 2023, 4:15pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/10 "2023-05-16T16:15:35Z")
> [@Is Transformers using GPU by default?](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500): > > I’m instantiating a model with this tokenizer = AutoTokenizer.from\_pretrained("nlptown/bert-base-multilingual-uncased-sentiment") model = AutoModelForSequenceClassification.from\_pretrained("nlptown/bert-base-multilingual-uncased-sentiment") Then running a for loop to get prediction over 10k sentences on a G4 instance (T4 GPU). GPU usage (averaged by minute) is a flat 0.0%. What is wrong? How to use GPU with Transformers? ---
**Author:** ![remigabillet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/remigabillet/32/16149_2.png) [@remigabillet](https://discuss.huggingface.co/u/remigabillet)\ **Post date:** [May 17, 2023, 4:27pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/11 "2023-05-17T16:27:08Z")
thank you @philschmid I’m making some progress. I managed to enable GPU on the endpoint by setting the device explicitly, I’m seeing 100x performance at around 300 embeddings/second. 😃 When using `sentence_transformers` on the same GPU hardware, I get about 750 embeddings/second. Any ideas how I can speed up the custom inference script to perform like `sentence_transformers.encode()`? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [May 18, 2023, 9:34am UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/12 "2023-05-18T09:34:55Z")
Could the difference come from the network overhead? since you are sending requests to a server? or are you measuring them? what GPU are you using locally? ---
**Author:** ![remigabillet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/remigabillet/32/16149_2.png) [@remigabillet](https://discuss.huggingface.co/u/remigabillet)\ **Post date:** [May 18, 2023, 3:05pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/13 "2023-05-18T15:05:56Z")
Hi Phil, no it’s not network overhead because I’m running this code locally. By locally I mean on a SageMaker Studio notebook with a Tesla P4 GPU. I put together a notebook that benchmarks `sentence_transformers` vs using `transformers` as described in the notebook you put together. I think I narrowed down the performance issue. Generating embeddings on GPU is very fast but calling tolist() on the tensors is taking a lot of time. It’s clear that `sentence_transformers` has been optimized for performance but I looked through the code but could not pinpoint what makes it so much faster. Please have a look at the notebook and let me know what you think: > ---
**Author:** ![bryanjj](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bryanjj/32/16290_2.png) [@bryanjj](https://discuss.huggingface.co/u/bryanjj)\ **Post date:** [May 23, 2023, 3:46pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/14 "2023-05-23T15:46:30Z")
Hey just following this because I am doing something similar. I actually started with this guide: [Building AI-powered search in PostgreSQL using Amazon SageMaker and pgvector | AWS Database Blog](https://aws.amazon.com/blogs/database/building-ai-powered-search-in-postgresql-using-amazon-sagemaker-and-pgvector/). That post demonstrates setting HF\_TASK to “feature-extraction” for embeddings, but it also does cls\_pooling instead of mean\_pooling for reasons I don’t understand. @remigabillet I think what @philschmid might have been referring to is that if you are using a custom inference script then that prediction endpoint is deployed to another server, which would account for networking delay. Of course I’m new to this so could be wrong, but I am also very interested in running embeddings at scale on very large datasets. ---
**Author:** ![bryanjj](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bryanjj/32/16290_2.png) [@bryanjj](https://discuss.huggingface.co/u/bryanjj)\ **Post date:** [May 23, 2023, 5:16pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/15 "2023-05-23T17:16:34Z")
Ok I see even when your run it all locally you are concerned about the tolist() time, but why call tolist() at all? just keep it as a numpy array? ---
**Author:** ![remigabillet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/remigabillet/32/16149_2.png) [@remigabillet](https://discuss.huggingface.co/u/remigabillet)\ **Post date:** [June 2, 2023, 12:53pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/16 "2023-06-02T12:53:58Z")
thanks for following up @bryanjj. I’m calling tolist() because the object is a torch and I need to save the vector in a separate database. I don’t know of another way to extract the data. 🤷 ---
**Author:** ![MitchKlaver](https://avatars.discourse-cdn.com/v4/letter/m/5f8ce5/32.png) [@MitchKlaver](https://discuss.huggingface.co/u/MitchKlaver)\ **Post date:** [October 12, 2023, 10:06pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/17 "2023-10-12T22:06:15Z")
Hi @remigabillet I’m not sure if you are still looking for help with the tolist(), but for anyone in the future coming across this thread. I believe that converting the tensor directly to a list with tolist() is indeed a very slow operation. What could be tested is to convert the tenser to a numpy array before converting it to a list. Potentially the issue can also arise from having to convert the tensor from the GPU to the CPU. Personally I have used the following approach: tensor.cpu().numpy().tolist() ---
**Author:** ![remigabillet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/remigabillet/32/16149_2.png) [@remigabillet](https://discuss.huggingface.co/u/remigabillet)\ **Post date:** [October 19, 2023, 10:09pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/18 "2023-10-19T22:09:45Z")
Thanks for the follow-up @MitchKlaver. I ended up using a different approach and avoiding Sagemaker altogether. If I try again this way, I’ll refer to your post. ---
**Author:** ![shozi218](https://avatars.discourse-cdn.com/v4/letter/s/e79b87/32.png) [@shozi218](https://discuss.huggingface.co/u/shozi218)\ **Post date:** [March 26, 2024, 3:29pm UTC](https://discuss.huggingface.co/t/deploying-sentence-transformer-as-sagemaker-endpoint/24455/19 "2024-03-26T15:29:23Z")
The Implementation in this part is good for a single sentence. If you want to take a list of strings as input as in the official model page ([https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)) you need to make changes to the inference.py file. Updated predict\_fn in inference.py ```auto def predict_fn(input_data, model_and_tokenizer): # extract model and tokenizer model, tokenizer = model_and_tokenizer #! The list is loaded as a string by aws sagemaker and needs to be converted back to a list sentences = ast.literal_eval(input_data['inputs']) # Tokenize sentences encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') # Compute token embeddings without gradient tracking with torch.no_grad(): model_output = model(**encoded_input) # Pool and Normalize the token embeddings sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask']) sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1) #! Originally only first sentence_embedding was returned, now return the entire tensor return {"vectors": sentence_embeddings.tolist()} ``` After following through the original posts link you can make inference using the below format: ```auto import json import boto3 input_str = ["This is an example sentence", "Each sentence is converted"] input_data = json.dumps( {"inputs": f"{input_str}"} ) endpoint_name = "YOUR-ENDPOINT-NAME-IN-SAGEMAKER" sagemaker_client = boto3.client('sagemaker-runtime', region_name="REGION-OF-SAGEMAAKER-ENDPOINT") response = sagemaker_client.invoke_endpoint( EndpointName=endpoint_name, Body=input_data, ContentType='application/json', Accept='application/json' ) response_json = json.loads(response['Body'].read().decode()) ```