# Error hosting endpoint when deploying model

**URL:** <https://discuss.huggingface.co/t/error-hosting-endpoint-when-deploying-model/53841>\
**Category:** Amazon SageMaker\
**Created:** [September 6, 2023, 5:07pm UTC](https://discuss.huggingface.co/t/error-hosting-endpoint-when-deploying-model/53841 "2023-09-06T17:07:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tiagosilva](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tiagosilva/32/19027_2.png) [@tiagosilva](https://discuss.huggingface.co/u/tiagosilva)\
**Post date:** [September 6, 2023, 5:07pm UTC](https://discuss.huggingface.co/t/error-hosting-endpoint-when-deploying-model/53841/1 "2023-09-06T17:07:43Z")

</div>

I’m trying to deploy the `meta-llama/Llama-2-70b-hf` using sagemaker, but I’m getting an error I don’t understand.

I created a jupyter notebook on sagemaker, ml.g5.2xlarge 256 Gb, and later one with 1024 GB.

Copied the deploy to sagemaker script from huggin face and replaced my token:

```auto
import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
	role = sagemaker.get_execution_role()
except ValueError:
	iam = boto3.client('iam')
	role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'meta-llama/Llama-2-70b-hf',
	'SM_NUM_GPUS': json.dumps(1),
	'HUGGING_FACE_HUB_TOKEN': '<REPLACE WITH YOUR TOKEN>'
}

assert hub['HUGGING_FACE_HUB_TOKEN'] != '<REPLACE WITH YOUR TOKEN>', "You have to provide a token."

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	image_uri=get_huggingface_llm_image_uri("huggingface",version="0.9.3"),
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1,
	instance_type="ml.g5.2xlarge",
	container_startup_health_check_timeout=300,
  )
  
# send request
predictor.predict({
	"inputs": "My name is Julien and I like to",
})

```

I get the following error:  
`UnexpectedStatusException: Error hosting endpoint huggingface-pytorch-tgi-inference-2023-09-06-16-46-01-586: Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..`

the logs for the endpoints just show this:

 ![Screenshot 2023-09-06 at 18.06.34](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/8/e84daaf6430da1811ed92950952172119eb9f395.png)

EDIT: for clarity the last 2 log entry read:

```auto
#033[2m2023-09-06T16:59:07.045706Z#033[0m #033[32m INFO#033[0m #033[2mtext_generation_launcher#033[0m#033[2m:#033[0m Download: [9/15] -- ETA: 0:05:43.333332
#033[2m2023-09-06T16:59:07.045945Z#033[0m #033[32m INFO#033[0m #033[2mtext_generation_launcher#033[0m#033[2m:#033[0m Download file: model-00010-of-00015.safetensors

```

It seems to be just stopping, and I don’t see a reason why.  
Any help is greatly appreciated.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 11, 2023, 8:39am UTC](https://discuss.huggingface.co/t/error-hosting-endpoint-when-deploying-model/53841/2 "2023-09-11T08:39:01Z")

</div>

You cannot deploy 70B on g5.2xlarge instance, see: [Deploy Llama 2 7B/13B/70B on Amazon SageMaker](https://www.philschmid.de/sagemaker-llama-llm#3-hardware-requirements)

---

<div class="post-metadata">

**Author:** ![ahsanr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ahsanr/32/25180_2.png) [@ahsanr](https://discuss.huggingface.co/u/ahsanr)\
**Post date:** [March 27, 2024, 8:15pm UTC](https://discuss.huggingface.co/t/error-hosting-endpoint-when-deploying-model/53841/3 "2024-03-27T20:15:09Z")

</div>

how can I deploy a Quantized model on aws,  
this model:

> **[vicuna-13b-v1.5.Q4\_K\_M.gguf · TheBloke/vicuna-13B-v1.5-GGUF at main](https://huggingface.co/TheBloke/vicuna-13B-v1.5-GGUF/blob/main/vicuna-13b-v1.5.Q4_K_M.gguf)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

for this scenario:  
Deploy this TheBloke/vicuna-13B-v1.5-GGUF model on AWS

I want to use this model as an endpoint in my web application in this format:

Chatbot Requirements

1. Scope: Chatbot (Encoder/Decoder for Text Inference or Conversational)
2. Input via API (JSON): Chatgpt Style – The template can be see belowThe JSON will contain 25 user messages, and the response should be the system response.  
Please use this guidelines to understand API consumption: [InvokeEndpoint - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html)
3. Prompt Template for the system:  
a. template = ‘’’  
You are going to be my education assistant.  
System:{System}  
Question:{question}‘’’
4. LLM Model Parameters: max\_new\_tokens=512, temperature=0.7, top\_p=0.9
5. If possible use a AutomodelforCausalLM otherwise train a LLM model.
6. It will be deployed on AWS Sagemaker using S3 buckets.
7. The GGUF should be saved on a S3 Bucket.
8. Chat Buffer should store 25 conversations and create a session ID (No need to send this to the End point).
9. Use HuggingFace/Langchain when possible.
10. Deliverables: Jupyter notebook/Code – 2 Hours should be used to set up the model in AWS with the customer.

Provide me with complete source code that I can use in my jupyter notebook on aws to make an endpoint.