# Deploying Tencent Hunyuan model

**URL:** <https://discuss.huggingface.co/t/deploying-tencent-hunyuan-model/116193>\
**Category:** Amazon SageMaker\
**Created:** [November 7, 2024, 1:25pm UTC](https://discuss.huggingface.co/t/deploying-tencent-hunyuan-model/116193 "2024-11-07T13:25:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alex1712](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Alex1712](https://discuss.huggingface.co/u/Alex1712)\
**Post date:** [November 7, 2024, 1:25pm UTC](https://discuss.huggingface.co/t/deploying-tencent-hunyuan-model/116193/1 "2024-11-07T13:25:46Z")

</div>

Hi! I wanted to deploy the new Tencent Hunyuan MoE model using sagemaker to test it. This is my code:

```auto
import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
   role = sagemaker.get_execution_role()
except ValueError:
   iam = boto3.client('iam')
   role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
   'HF_MODEL_ID':'tencent-community/Hunyuan-A52B-Instruct',
   'HF_TASK':'text-generation', 
   'SM_NUM_GPUS': json.dumps(8), 
   'MAX_BATCH_PREFILL_TOKENS': json.dumps(10500),
   'MAX_TOTAL_TOKENS':json.dumps(12000),
   'MAX_INPUT_TOKENS':json.dumps(10000),
   'TRUST_REMOTE_CODE': 'true'
}
huggingface_model = HuggingFaceModel(
   image_uri=get_huggingface_llm_image_uri("huggingface"),
   env=hub,
   role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.p4d.24xlarge",
   container_startup_health_check_timeout=1800,  
   model_data_download_timeout=1800,
 )

```

However, I ran into this error: `File "/opt/conda/lib/python3.11/site-packages/text_generation_server/models/ __init__.py", line 1206, in get_model raise NotImplementedError("sharded is not supported for AutoModel")`

I googled it a bit and saw that is somewhat related to `image_uri`, but I thought `get_huggingface_llm_image_uri` would get the latest one.

Any insights would be greatly appreciated! I’m new to LLM deployment and still getting familiar with the technical details.

Link to HF model: [tencent-community/Hunyuan-A52B-Instruct · Hugging Face](https://huggingface.co/tencent-community/Hunyuan-A52B-Instruct)

---

<div class="post-metadata">

**Author:** ![dickzhu](https://avatars.discourse-cdn.com/v4/letter/d/5f8ce5/32.png) [@dickzhu](https://discuss.huggingface.co/u/dickzhu)\
**Post date:** [November 27, 2024, 3:18pm UTC](https://discuss.huggingface.co/t/deploying-tencent-hunyuan-model/116193/2 "2024-11-27T15:18:30Z")

</div>

Thank you for your interest in trying out Hunyuan-Large.

To the best of our knowledge, SageMaker currently does not support Hunyuan-Large yet. Instead, Hunyuan currently supports deployment through the Hugging Face standard training framework and the popular vLLM inference framework. Please refer to the link ([GitHub - Tencent/Tencent-Hunyuan-Large](https://github.com/Tencent/Tencent-Hunyuan-Large)) for detailed instructions.

If you want a quick trial of the open-source model with an interactive user interface, you may also check out the Demo interface supported by Tencent on Hugging Face: [Hunyuan Large - a Hugging Face Space by tencent](https://huggingface.co/spaces/tencent/Hunyuan-Large).

Hope this answers your question and please feel free to contact us if you have any further questions.