# Unable to deploy Qwen2-VL model on SageMaker

**URL:** <https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624>\
**Category:** Amazon SageMaker\
**Created:** [October 23, 2024, 3:49pm UTC](https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624 "2024-10-23T15:49:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hubkrieb-eifer](https://avatars.discourse-cdn.com/v4/letter/h/73ab20/32.png) [@hubkrieb-eifer](https://discuss.huggingface.co/u/hubkrieb-eifer)\
**Post date:** [October 23, 2024, 3:49pm UTC](https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624/1 "2024-10-23T15:49:44Z")

</div>

I want to deploy Qwen2-VL-2B-Instruct to a SageMaker endpoint. I know that the model needs additional requirements such as qwen-vl-utils but also needs the latest version of transformers (4.45.2) to work properly. As this version of transformers is not available in DLC, I added a requirements.txt and inference.py inside a code/ folder in the model.tar.gz. Here are the files content:  
requirements.txt

```auto
transformers==4.45.2
accelerate==0.34.2
bitsandbytes==0.43.3
qwen-vl-utils==0.0.4

```

inference.py

```auto
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

def model_fn(model_dir):
    model = Qwen2VLForConditionalGeneration.from_pretrained(model_dir, torch_dtype=torch.bfloat16).to(device)
    # Load the processor
    processor = AutoProcessor.from_pretrained(model_dir)
    return [model, processor]

def predict_fn(data, model_process):
    model = model_process[0]
    processor = model_process[1]

    text = processor.apply_chat_template(data, tokenize=False, add_generation_prompt=True)
    image_inputs, video_inputs = process_vision_info(data)
    inputs = processor(
        text=[text],
        images=image_inputs,
        videos=video_inputs,
        padding=True,
        return_tensors="pt",
    ).to(device)

    with torch.no_grad():
        generated_ids = model.generate(**inputs, max_new_tokens=496)
        generated_ids_trimmed = [
            out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
        ]
        output_text = processor.batch_decode(
            generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
        )
    return {"response": output_text[0]}

```

deployment code:

```auto
image_uri = "763104351884.dkr.ecr.eu-central-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.4-tgi2.3-gpu-py311-cu124-ubuntu22.04"
config = {
    "HF_MODEL_ID": "Qwen/Qwen2-VL-2B-Instruct",
    "SM_NUM_GPUS": json.dumps(1),
    "MAX_INPUT_LENGTH": json.dumps(1024),
    "MAW_TOTAL_TOKENS": json.dumps(2048),
}

huggingface_model = HuggingFaceModel(
   model_data=s3_model_uri,      
   role=execution_role_arn,      
   image_uri=image_uri,
   sagemaker_session = sess,
   env=config
)

predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.xlarge",
    endpoint_name="Qwen2-VL"
    )

```

The error I get:

```auto
UnexpectedStatusException: Error hosting endpoint Qwen2-VL: Failed. Reason: The primary container for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint.

```

What I found in the logs:

```auto
ValueError: Unsupported model type qwen2_vl

```

Any idea of what could have gone wrong?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 23, 2024, 4:06pm UTC](https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624/2 "2024-10-23T16:06:23Z")

</div>

Could it be a bug in SageMaker?

> [@ValueError: Unsupported model type mllama](https://discuss.huggingface.co/t/valueerror-unsupported-model-type-mllama/113593):
>
> I am looking to deploy “meta-llama/Llama-3.2-11B-Vision-Instruct” to AWS SageMaker but getting below error while deploying. Traceback (most recent call last): File "/opt/conda/bin/text-generation-server", line 8, in \<module\> sys.exit(app()) File "/opt/conda/lib/python3.10/site-packages/text\_generation\_server/cli.py", line 118, in serve server.serve( File "/opt/conda/lib/python3.10/site-packages/text\_generation\_server/server.py", line 297, in serve asyncio.run( File "/opt/con…

---

<div class="post-metadata">

**Author:** ![hubkrieb-eifer](https://avatars.discourse-cdn.com/v4/letter/h/73ab20/32.png) [@hubkrieb-eifer](https://discuss.huggingface.co/u/hubkrieb-eifer)\
**Post date:** [October 24, 2024, 8:45am UTC](https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624/3 "2024-10-24T08:45:25Z")

</div>

I guess TGI is just not compatible with Qwen2-VL yet

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 24, 2024, 8:54am UTC](https://discuss.huggingface.co/t/unable-to-deploy-qwen2-vl-model-on-sagemaker/113624/4 "2024-10-24T08:54:41Z")

</div>

Searching the release notes for Qwen, I see only that it simply says **Qwen2** is supported. if Qwen2.5 and Qwen2-VL are not specified, it seems possible that you are right and it is unsupported.

> **[Releases · huggingface/text-generation-inference](https://github.com/huggingface/text-generation-inference/releases?page=1)**
>
> Large Language Model Text Generation Inference. Contribute to huggingface/text-generation-inference development by creating an account on GitHub.