# Vicuan error on Sagemaker

**URL:** <https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301>\
**Category:** Amazon SageMaker\
**Created:** [June 6, 2023, 5:55pm UTC](https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301 "2023-06-06T17:55:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ozcode](https://avatars.discourse-cdn.com/v4/letter/o/51bf81/32.png) [@ozcode](https://discuss.huggingface.co/u/ozcode)\
**Post date:** [June 6, 2023, 5:55pm UTC](https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301/1 "2023-06-06T17:55:56Z")

</div>

Hi folks,

I have been trying to deploy TheBloke/vicuna-7B-1.1-HF to SageMaker but with no luck. I have not had problems with other models like bloom-3b. I used the following code to deploy:

```auto
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel

try:
	role = sagemaker.get_execution_role()
except ValueError:
	iam = boto3.client('iam')
	role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'TheBloke/vicuna-7B-1.1-HF',
	'HF_TASK':'text-generation'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.26.0',
	pytorch_version='1.13.1',
	py_version='py39',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.g4dn.2xlarge' # ec2 instance type
)

predictor.predict({
	"inputs": "Can you please let us know more details about your ",
})

```

However, I am getting the following error when predicting:

```auto
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "\u0027llama\u0027"
}

```

Any help is appreciated.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [June 7, 2023, 8:07am UTC](https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301/2 "2023-06-07T08:07:31Z")

</div>

can you try to use the new LLM container? [Introducing the Hugging Face LLM Inference Container for Amazon SageMaker](https://huggingface.co/blog/sagemaker-huggingface-llm)

---

<div class="post-metadata">

**Author:** ![ozcode](https://avatars.discourse-cdn.com/v4/letter/o/51bf81/32.png) [@ozcode](https://discuss.huggingface.co/u/ozcode)\
**Post date:** [June 7, 2023, 10:25am UTC](https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301/3 "2023-06-07T10:25:01Z")

</div>

Thank you! It works with this new container.

---

<div class="post-metadata">

**Author:** ![Sanjay679](https://avatars.discourse-cdn.com/v4/letter/s/919ad9/32.png) [@Sanjay679](https://discuss.huggingface.co/u/Sanjay679)\
**Post date:** [October 23, 2024, 11:21am UTC](https://discuss.huggingface.co/t/vicuan-error-on-sagemaker/42301/4 "2024-10-23T11:21:01Z")

</div>

@philschmid

I am using following code

import sagemaker  
import boto3  
from sagemaker.huggingface import HuggingFaceModel  
import sagemaker  
import boto3  
sess = sagemaker.Session()

sagemaker\_session\_bucket=None  
if sagemaker\_session\_bucket is None and sess is not None:  
sagemaker\_session\_bucket = sess.default\_bucket()

try:  
role = sagemaker.get\_execution\_role()  
except ValueError:  
iam = boto3.client(‘iam’)  
role = iam.get\_role(RoleName=‘sagemaker\_execution\_role’)[‘Role’][‘Arn’]

sess = sagemaker.Session(default\_bucket=sagemaker\_session\_bucket)

hub = {  
‘HF\_MODEL\_ID’:‘model id’,  
‘HF\_TASK’:‘text-classification’,  
‘HF\_TOKEN’:‘token’  
}

huggingface\_model = HuggingFaceModel(  
transformers\_version=‘4.45.2’,  
pytorch\_version=‘2.2.0’,  
py\_version=‘py310’,  
env=hub,  
role=role,  
)

predictor = huggingface\_model.deploy(  
initial\_instance\_count=1, # number of instances  
instance\_type=‘ml.m5.xlarge’, # ec2 instance type  
endpoint\_name = ‘classifier’  
)

predictor.predict({  
“inputs”: “I like you. I love you”,  
})

But it is giving me error as follows  
ValueError: Unsupported huggingface version: 4.45.2. You may need to upgrade your SDK version (pip install -U sagemaker) for newer huggingface versions. Supported huggingface version(s): 4.6.1, 4.10.2, 4.11.0, 4.12.3, 4.17.0, 4.26.0, 4.28.1, 4.37.0, 4.6, 4.10, 4.11, 4.12, 4.17, 4.26, 4.28, 4.37.

I am using sagemaker jupyter notebook to deploy with sagemaker version 2.23.2