# Sagemaker Serverless Inference **URL:** \ **Category:** Amazon SageMaker\ **Created:** [December 30, 2021, 1:44pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246 "2021-12-30T13:44:18Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![YannAgora](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@YannAgora](https://discuss.huggingface.co/u/YannAgora)\ **Post date:** [December 30, 2021, 1:44pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/1 "2021-12-30T13:44:18Z")
Hi there, I have been trying to use the new serverless feature from Sagemaker Inference, following the different steps very well explained by @juliensimon in his video (using same Image for the container and same ServerlessConfig) to use an HuggingFace model (not fine-tuned on my side). However after having successfully created/deployed all resources (Model, EndpointConfig, Endpoint) and trying to invokeEndpoint, I encountered this error : ```auto 'Message': 'An exception occurred from internal dependency. Please contact customer support regarding request ...' ``` And when looking on cloudwatch, I also get this message : ```auto python: can't open file "/usr/local/bin/deep_learning_container.py": [Errno 13] Permission denied ``` Error that I don’t get when using invokeEndpoint for a non-serverless Inference Endpoint. Did someone already encounter this error ? Thanks in advance !! ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [December 30, 2021, 1:46pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/2 "2021-12-30T13:46:45Z")
Hey @YannAgora, Thanks for opening the thread. We encountered this error as well. See [blog post](https://www.philschmid.de/serverless-transformers-sagemaker-huggingface) > - **found limitation when testing:** Currently, Transformer models \> 512MB create errors We already reported this error to the SageMaker team. Which model are you trying to use? with which memory configuration? ---
**Author:** ![YannAgora](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@YannAgora](https://discuss.huggingface.co/u/YannAgora)\ **Post date:** [December 30, 2021, 1:52pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/3 "2021-12-30T13:52:38Z")
Hi @philschmid, For now I tried on **tuner007/pegasus\_paraphrase** and **Vamsi/T5\_Paraphrase\_Paws** that are indeed two models over 512MB which confirms the limitation you found. And for the MemorySize I set the value to 6144MB which is the max value available if I’m not mistaken. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [December 30, 2021, 2:09pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/4 "2021-12-30T14:09:42Z")
I’ll let you know when this is fixed! I hope soon! ---
**Author:** ![nurik](https://avatars.discourse-cdn.com/v4/letter/n/977dab/32.png) [@nurik](https://discuss.huggingface.co/u/nurik)\ **Post date:** [January 24, 2022, 7:18pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/5 "2022-01-24T19:18:39Z")
hi! 1. has the 512MB limitation issue been resolved? does this issue exist only with huggingface models? 2. is there a gpu support for the serverless inference? any info will be greatly appreciated. ---
**Author:** ![nurik](https://avatars.discourse-cdn.com/v4/letter/n/977dab/32.png) [@nurik](https://discuss.huggingface.co/u/nurik)\ **Post date:** [January 24, 2022, 8:41pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/6 "2022-01-24T20:41:19Z")
1. apparently, it does not support gpus currently. i wonder what cpu instance type it uses and whether it is too slow for real-time inference on transformers. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [January 25, 2022, 12:49pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/7 "2022-01-25T12:49:20Z")
Hello, 1. There is no fix yet for it but there is a workaround. You can set an environment variable `MMS_DEFAULT_WORKERS_PER_MODEL=1` when creating the endpoint. 2. Since Serverless Inference is powered by AWS Lambda and AWS Lambda doesn’t have GPU support yet Serverless Inference won’t have it as well. And i assume it will get GPU support when AWS Lambda has GPU support. ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [January 25, 2022, 4:33pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/8 "2022-01-25T16:33:01Z")
GPUs for inference are only relevant when there are parallelism opportunities, i.e. an inference request requires lots of computations. I often find CPUs sufficient for simpler workloads. In the end the respoonse time will depend on many different factors, such as model size, payload, additional pre/postprocessing, etc. I have deployed a small BERT model for sentiment classification on a serverless endpoint and response times are ~200-300 ms (including passing the request through an API): ![Capture2](https://us1.discourse-cdn.com/hellohellohello/original/2X/e/e04288f43e59466a7f1135870cc983bce86bf881.png) Hope that helps. Cheers Heiko ---
**Author:** ![nurik](https://avatars.discourse-cdn.com/v4/letter/n/977dab/32.png) [@nurik](https://discuss.huggingface.co/u/nurik)\ **Post date:** [January 25, 2022, 4:50pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/9 "2022-01-25T16:50:00Z")
Thank you for response @philschmid .The trick will be helpful when deploying. Thank you for the test @marshmellow77. The model I am using is large so I guess I will attempt to deploy an ONNX model to bring inference to below 1 second. Is your classifier based on DistilBERT? ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [January 25, 2022, 5:47pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/10 "2022-01-25T17:47:07Z")
Yes, the classifier is based on DistilBert. ---
**Author:** ![bennicholl](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@bennicholl](https://discuss.huggingface.co/u/bennicholl)\ **Post date:** [June 14, 2022, 5:39pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/11 "2022-06-14T17:39:51Z")
I’m also running into this error when running a server less inference with sagemakers PyTorchModel() function. ```auto similarity_elec_model = PyTorchModel( model_data='s3://pathname/model.tar.gz', role=role, entry_point='torchserve.py', source_dir='source_dir', framework_version='1.6.0', py_version='py36') ``` serverless config is setup with below code ```auto serverless_config = ServerlessInferenceConfig( memory_size_in_mb=6144, max_concurrency=2, ) ``` Has there been a fix for this? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [June 15, 2022, 8:52am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/12 "2022-06-15T08:52:37Z")
Hey @bennicholl, this error should no longer exist. Could you please try using the `HuggingFaceModel` with transformers\_version 4.17 and pytorch 1.6? ---
**Author:** ![dshandler-monash](https://avatars.discourse-cdn.com/v4/letter/d/e95f7d/32.png) [@dshandler-monash](https://discuss.huggingface.co/u/dshandler-monash)\ **Post date:** [October 15, 2022, 3:15am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/14 "2022-10-15T03:15:25Z")
Hey @philschmid, I just setup a serverless endpoint for conversational AI using facebook/blenderbot-400M-distill. I still receive this error on cloudwatch and the API call times out. It does eventually return a response but will return this error a few times before that happens. I set up this endpoint using your cdk, sagemaker-serverless-huggingface-endpoint. Do you know if there is something I can change in the setup which will prevent this from happening? Thanks, Darren ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [October 17, 2022, 5:41am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/15 "2022-10-17T05:41:58Z")
@dshandler-monash can you please describe a bit more your setup? How much memory has your configuration? what is the error you are seeing… ---
**Author:** ![dshandler-monash](https://avatars.discourse-cdn.com/v4/letter/d/e95f7d/32.png) [@dshandler-monash](https://discuss.huggingface.co/u/dshandler-monash)\ **Post date:** [October 17, 2022, 6:51am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/16 "2022-10-17T06:51:42Z")
Thanks for replying. I’m using AWS API Gateway to call sagemaker. The endpoint has 6GB of memory, Max Concurrency = 8 and the model has MMS\_DEFAULT\_WORKERS\_PER\_MODEL=1 image: huggingface-pytorch-inference:1.9.1-transformers4.12.3-cpu-py38-ubuntu20.04 This is the error message from cloudwatch: 2022-10-17T06:02:34,139 [WARN] W-9000-facebook\_\_blenderbot-400M-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - python: can’t open file ‘/usr/local/bin/deep\_learning\_container.py’: [Errno 13] Permission denied Sometimes after 2-4 attempts the model works and returns a response. As long as you interact with it consistently, it continues to work, and the above error does not occur again. Edit: I’ve just seen you mentioned transformers\_version 4.17 and the CDK used 4.12.3. So I think it will get fixed by reinstalling it using LATEST\_TRANSFORMERS\_VERSION = “4.17.0” in config.py? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [October 17, 2022, 8:10am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/17 "2022-10-17T08:10:13Z")
> I’ve just seen you mentioned transformers\_version 4.17 and the CDK used 4.12.3. So I think it will get fixed by reinstalling it using LATEST\_TRANSFORMERS\_VERSION = “4.17.0” in config.py? Yes should be the case. ---
**Author:** ![dshandler-monash](https://avatars.discourse-cdn.com/v4/letter/d/e95f7d/32.png) [@dshandler-monash](https://discuss.huggingface.co/u/dshandler-monash)\ **Post date:** [October 17, 2022, 11:18pm UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/18 "2022-10-17T23:18:49Z")
I’ve recreated it with ```auto huggingface-pytorch-inference:1.10.2-transformers4.17.0-cpu-py38-ubuntu20.04 ``` and unfortunately still run into the same issue. Are there any other solutions you could think of? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [October 18, 2022, 5:43am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/19 "2022-10-18T05:43:25Z")
can you please provide the code you used to deploy? so we can try to reproduce your scenario? ---
**Author:** ![dshandler-monash](https://avatars.discourse-cdn.com/v4/letter/d/e95f7d/32.png) [@dshandler-monash](https://discuss.huggingface.co/u/dshandler-monash)\ **Post date:** [October 25, 2022, 6:51am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/20 "2022-10-25T06:51:20Z")
I used this CDK with the changes to the versions in the config file. I also had to make some changes to IAM permissions during the installation to get it to complete. > **[cdk-samples/sagemaker-serverless-huggingface-endpoint at master ·...](https://github.com/philschmid/cdk-samples/tree/master/sagemaker-serverless-huggingface-endpoint)** > > master/sagemaker-serverless-huggingface-endpoint ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [October 31, 2022, 8:37am UTC](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246/21 "2022-10-31T08:37:36Z")
This shouldn’t be the case. But can you try deploying your model using the `sagemaker-sdk`? [Serverless Inference with Hugging Face's Transformers, DistilBERT and Amazon SageMaker](https://www.philschmid.de/sagemaker-serverless-huggingface-distilbert) [Next page](https://discuss.huggingface.co/t/sagemaker-serverless-inference/13246.md?page=2)