# Issue - ValueError: Unsupported model type mixtral

**URL:** <https://discuss.huggingface.co/t/issue-valueerror-unsupported-model-type-mixtral/66701>\
**Category:** Amazon SageMaker\
**Created:** [December 22, 2023, 7:22am UTC](https://discuss.huggingface.co/t/issue-valueerror-unsupported-model-type-mixtral/66701 "2023-12-22T07:22:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ashwincv0112](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@ashwincv0112](https://discuss.huggingface.co/u/ashwincv0112)\
**Post date:** [December 22, 2023, 7:22am UTC](https://discuss.huggingface.co/t/issue-valueerror-unsupported-model-type-mixtral/66701/1 "2023-12-22T07:22:59Z")

</div>

Trying to deploy the PreTrained version of the below model to SageMaker Endpoint.

Model Name: mistralai/Mixtral-8x7B-Instruct-v0.1

Using the g5.48xlarge instance for the same.

But getting the below error:

> File “/opt/conda/lib/python3.9/site-packages/text\_generation\_server/server.py”, line 124, in serve\_inner

```
model = get_model(model_id, revision, sharded, quantize, trust_remote_code)

```

File “/opt/conda/lib/python3.9/site-packages/text\_generation\_server/models/ **init**.py”, line 314, in get\_model  
raise ValueError(f"Unsupported model type {model\_type}")

**ValueError: Unsupported model type mixtral**.

Please find the below deployed commands for the same:

from sagemaker.huggingface import get\_huggingface\_llm\_image\_uri

### retrieve the llm image uri

llm\_image = get\_huggingface\_llm\_image\_uri(  
“huggingface”,  
version=“1.1.0”  
)

print(f"llm image uri: {llm\_image}")

import json

from sagemaker.huggingface import HuggingFaceModel

### sagemaker config

instance\_type = “ml.g5.48xlarge”  
number\_of\_gpu = 1  
health\_check\_timeout = 1000

### Define Model and Endpoint configuration parameter

config = {  
‘HF\_MODEL\_ID’: “mistralai/Mixtral-8x7B-v0.1”, # model\_id from [Models – Hugging Face](http://hf.co/models)  
‘SM\_NUM\_GPUS’: json.dumps(number\_of\_gpu), # Number of GPU used per replica  
‘MAX\_INPUT\_LENGTH’: json.dumps(1024), # Max length of input text  
‘MAX\_TOTAL\_TOKENS’: json.dumps(2048), # Max length of the generation (including input text)  
}

### create HuggingFaceModel with the image uri

llm\_model = HuggingFaceModel(  
role=role,  
env=config,  
transformers\_version = ‘4.28’, # the transformers version used in the training job  
pytorch\_version = ‘2.0’, # the pytorch\_version version used in the training job  
py\_version = ‘py310’,  
)

llm = llm\_model.deploy(  
initial\_instance\_count=1,  
instance\_type=instance\_type,  
container\_startup\_health\_check\_timeout=health\_check\_timeout,  
endpoint\_name = ‘mixtral-inference-testing1’  
)

Please let me know if I am using the right versions or if I am doing anything wrong.  
Let me know if any other information is required.

Thanks.

---

<div class="post-metadata">

**Author:** ![ashwincv0112](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@ashwincv0112](https://discuss.huggingface.co/u/ashwincv0112)\
**Post date:** [December 28, 2023, 4:11pm UTC](https://discuss.huggingface.co/t/issue-valueerror-unsupported-model-type-mixtral/66701/2 "2023-12-28T16:11:07Z")

</div>

On troubleshooting, found that the initial or above mentioned error was due to the transformer version. Earlier was using 4.34. Now changed it to 4.36.2

After the change, getting the below error while the logic is trying to download the Model.

\*\*AlgorithmError: ExecuteUserScriptError: ExitCode 1 ErrorMessage "ImportError: /opt/conda/lib/python3.10/site-packages/flash\_attn\_2\_cuda.cpython-310-x86\_64-linux-gnu.so: undefined symbol: \_ZNK3c106SymIntltEl \*\*

**training\_function model = AutoModelForCausalLM.from\_pretrained(script\_args.model\_id, File “/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto\_factory.py”, line 565, in from\_pretrained model\_class = \_get\_model\_class(config, cls.\_model\_mapping) File “/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto\_factory.py”, line 387, in \_get\_model\_class supported\_models = model\_mapping[type(config)] File "/opt/conda/lib/python3.10/site-packag**

But the same logic is working (the Model is getting downloaded and also I am able to do some training in the same) when I am running it outside the AWS Sagemaker.