# Missing schema for sagemaker deployment of zero-shot image classification models

**URL:** <https://discuss.huggingface.co/t/missing-schema-for-sagemaker-deployment-of-zero-shot-image-classification-models/41614>\
**Category:** Amazon SageMaker\
**Created:** [May 31, 2023, 12:01pm UTC](https://discuss.huggingface.co/t/missing-schema-for-sagemaker-deployment-of-zero-shot-image-classification-models/41614 "2023-05-31T12:01:30Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BK10](https://avatars.discourse-cdn.com/v4/letter/b/c67d28/32.png) [@BK10](https://discuss.huggingface.co/u/BK10)\
**Post date:** [May 31, 2023, 12:01pm UTC](https://discuss.huggingface.co/t/missing-schema-for-sagemaker-deployment-of-zero-shot-image-classification-models/41614/1 "2023-05-31T12:01:30Z")

</div>

Trying to get predictions on a deployed zero-shot image classification model and it appears as if the schema for the .predict() method is missing from the documentation. I have followed the schema example from the zero-shot text classification, but have changed the text input to an image and it doesn’t seem to be working. The model has been been deployed to an endpoint successfully, but fails when trying to produce an inference. I’m assuming its to do with the format of the inputs.

I have also tried the following image representations and none have worked.

- Raw RGB data (Binary)
- Image itself - Object of type JpegImageFile is not JSON serializable
- base64 encoded string is not a valid path 400 (invalid input parameters)
- bytes - Object of type bytes is not JSON serializable
- array of bytes - Object of type bytes is not JSON serializable
- BytesIO object - Object of type bytes is not JSON serializable

Code provided below:

import torch, gc  
import time  
from sagemaker.huggingface.model import HuggingFaceModel  
from sagemaker.serializers import DataSerializer, JSONSerializer, IdentitySerializer  
from sagemaker.deserializers import JSONDeserializer

# Hub model configuration [https://huggingface.co/models](https://huggingface.co/models)

hub = {  
‘HF\_MODEL\_ID’:‘openai/clip-vit-large-patch14’, # model\_id from [Models – Hugging Face](http://hf.co/models)  
‘HF\_TASK’:‘zero-shot-image-classification’ # Type of task your model is solving  
}

# create Hugging Face Model Class

huggingface\_model = HuggingFaceModel(  
env=hub, # configuration for loading model from Hub  
role=role, # IAM role with permissions to create an endpoint  
transformers\_version=“4.17”, # Transformers version used  
pytorch\_version=“1.10”, # PyTorch version used  
py\_version=‘py38’, # Python version used  
)

#image\_serializer = DataSerializer(content\_type=‘image/x-image’) - didn’t work

# deploy model to SageMaker Inference

predictor1 = huggingface\_model.deploy(  
initial\_instance\_count=1,  
instance\_type=“ml.g4dn.4xlarge”,  
serializer=JSONSerializer(),  
deserializer=JSONDeserializer(),  
)

image = Image.open(“000000039769.jpg”)  
byt = image.tobytes()  
mystr = base64.b64encode(byt).decode()  
\_dict = {“bytes”: byt}  
newbytes = base64.b64decode(mystr)

data = {“inputs”: mystr,  
“parameters”:{‘candidate\_labels’:[“indoor”, “outdoor”]}  
}  
reponse = predictor1.predict(data)

---

<div class="post-metadata">

**Author:** ![tamisalex](https://avatars.discourse-cdn.com/v4/letter/t/cdc98d/32.png) [@tamisalex](https://discuss.huggingface.co/u/tamisalex)\
**Post date:** [June 7, 2023, 6:34pm UTC](https://discuss.huggingface.co/t/missing-schema-for-sagemaker-deployment-of-zero-shot-image-classification-models/41614/2 "2023-06-07T18:34:10Z")

</div>

try

{  
“input”: “[http://someurl.com/someimage.jpg](http://someurl.com/someimage.jpg)”,  
“parameters”: {“candidate\_labels”: list\_of\_zero\_shot\_labels}  
}

---

<div class="post-metadata">

**Author:** ![smsatmane](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@smsatmane](https://discuss.huggingface.co/u/smsatmane)\
**Post date:** [August 31, 2023, 11:56am UTC](https://discuss.huggingface.co/t/missing-schema-for-sagemaker-deployment-of-zero-shot-image-classification-models/41614/3 "2023-08-31T11:56:57Z")

</div>

Hi, I tried supplying image as http url and it works. But if I want to host the model and send image byte string, how do I provide image? thanks