# Error deploying endpoint on Aws

**URL:** <https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675>\
**Category:** Models\
**Created:** [August 23, 2024, 9:01am UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675 "2024-08-23T09:01:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Knour13](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@Knour13](https://discuss.huggingface.co/u/Knour13)\
**Post date:** [August 23, 2024, 9:01am UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675/1 "2024-08-23T09:01:07Z")

</div>

I’m trying to deploy my finetuned LLama3 model on Aws, so the first step to create an endpoint  
I used instance\_type=“ml.g5.4xlarge”,  
This is my code:

import json  
import sagemaker  
import boto3  
from sagemaker.huggingface import HuggingFaceModel, get\_huggingface\_llm\_image\_uri

try:  
role = sagemaker.get\_execution\_role()  
except ValueError:  
iam = boto3.client(‘iam’)  
role = iam.get\_role(RoleName=‘sagemaker\_execution\_role’)[‘Role’][‘Arn’]

# Hub Model configuration. [Models – Hugging Face](https://huggingface.co/models)

hub = {  
‘HF\_MODEL\_ID’:‘Guepard/knaine\_llama3.1\_v0’,  
‘HF\_TASK’: ‘text-generation’  
}

# create Hugging Face Model Class

huggingface\_model = HuggingFaceModel(  
image\_uri=get\_huggingface\_llm\_image\_uri(“huggingface”,version=“2.0.2”),  
env=hub,  
role=role,  
)

# deploy model to SageMaker Inference

predictor = huggingface\_model.deploy(  
initial\_instance\_count=1,  
instance\_type=“ml.g5.4xlarge”,  
container\_startup\_health\_check\_timeout=1200,  
)

# send request

predictor.predict({  
“inputs”: “My name is Julien and I like to”,  
})

I get the following error :

UnexpectedStatusException: Error hosting endpoint huggingface-pytorch-tgi-inference-2024-08-23-08-25-25-823: Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint.. Try changing the instance type or reference the troubleshooting page [Troubleshooting - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-troubleshooting.html)

Any solution?

---

<div class="post-metadata">

**Author:** ![mahmutc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahmutc/32/52583_2.png) [@mahmutc](https://discuss.huggingface.co/u/mahmutc)\
**Post date:** [August 23, 2024, 9:12am UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675/2 "2024-08-23T09:12:54Z")

</div>

hi @Knour13  
Did you check CloudWatch logs?

> <https://github.com/aws/amazon-sagemaker-examples/issues/210#issuecomment-371549251>
>
> I am trying to deploy a BYOB (bring your own model) keras model. I pushed the im…age to ECR with the 'latest' tag. All local testing passed, and I am able to successfully train the model e.g.:
> 
> \`\`\`python
> 
> image = '{}.dkr.ecr.{}.amazonaws.com/my-model:latest'.format(account, region)
> 
> dl = sage.estimator.Estimator(image,
> role, 1, 'ml.c4.2xlarge',
> output\_path="s3://{}/output".format(sess.default\_bucket()),
> sagemaker\_session=sess)
> 
> \`\`\`
> 
> However attempting to deploy gives me the error:
> 
> \`\`\`
> Failed Reason: The primary container for production variant AllTraffic did not pass the ping health check.
> \`\`\`
> I am not quite sure where this stems from given local health check passed. Any insight would be great! Thanks.

---

<div class="post-metadata">

**Author:** ![Knour13](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@Knour13](https://discuss.huggingface.co/u/Knour13)\
**Post date:** [August 23, 2024, 11:29am UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675/3 "2024-08-23T11:29:50Z")

</div>

I found in cloud watch these errors:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/f/df98ceefe4cf71473f8a511cd0b6cfb9f7dcaa33.png)  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/1/7/176d5ecd10c46ef285063038ef9ad11980df6ee6.png)

---

<div class="post-metadata">

**Author:** ![Knour13](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@Knour13](https://discuss.huggingface.co/u/Knour13)\
**Post date:** [August 23, 2024, 11:31am UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675/4 "2024-08-23T11:31:29Z")

</div>

@philschmid Hi Do you have any solution ?  
Thanks

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [August 23, 2024, 11:52am UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675/5 "2024-08-23T11:52:11Z")

</div>

Try TGI 2.2.0

---

<div class="post-metadata">

**Author:** ![Knour13](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@Knour13](https://discuss.huggingface.co/u/Knour13)\
**Post date:** [August 23, 2024, 12:20pm UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675/6 "2024-08-23T12:20:55Z")

</div>

Same error

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/8/58f8549c996cb1ebb3140e297170439628fc18fa.png)

---

<div class="post-metadata">

**Author:** ![Knour13](https://avatars.discourse-cdn.com/v4/letter/k/65b543/32.png) [@Knour13](https://discuss.huggingface.co/u/Knour13)\
**Post date:** [August 23, 2024, 12:42pm UTC](https://discuss.huggingface.co/t/error-deploying-endpoint-on-aws/103675/7 "2024-08-23T12:42:38Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/b/5b41bee005970587ea2b6cac6d1a720ef3450f35.png)  
This is cloud watch