# Custom Inference.py script for Vision Transformer

**URL:** <https://discuss.huggingface.co/t/custom-inference-py-script-for-vision-transformer/27482>\
**Category:** Amazon SageMaker\
**Created:** [December 8, 2022, 7:52pm UTC](https://discuss.huggingface.co/t/custom-inference-py-script-for-vision-transformer/27482 "2022-12-08T19:52:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ShuaHousetable](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@ShuaHousetable](https://discuss.huggingface.co/u/ShuaHousetable)\
**Post date:** [December 8, 2022, 7:52pm UTC](https://discuss.huggingface.co/t/custom-inference-py-script-for-vision-transformer/27482/1 "2022-12-08T19:52:10Z")

</div>

I am trying to deploy a Vision Transformer huggingface model to a sagemaker endpoint. I am trying to make an inference.py script to go along with the model artificats, so that I can have my own custom output from predict\_fn.

However, the endpoint keeps returning this error when passing the image through the feature\_extractor:

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from model with message "{  
“code”: 400,  
“type”: “InternalServerException”,  
“message”: “an integer is required (got type str)”

I’ve checked and confirmed that the image is indeed a PIL image (which works when I pass it through the endpoint regularly, without the inference script, just without returning the desired output format). I have even tried to convert the image to a numpy array, but I still get the same error.

This is the script:  
def model\_fn(model\_dir):  
logging.info(‘within model\_fn’)  
model = ViTForImageClassification.from\_pretrained(model\_dir)  
feature\_extractor = ViTFeatureExtractor.from\_pretrained(model\_dir)  
logging.info(‘Extracted Model and Extractor’)  
return model, feature\_extractor

def predict\_fn(data, model\_extractor):  
# destruct model and tokenizer  
logging.info(‘Inside Prediction\_fn’)  
model, feature\_extractor = model\_extractor

```
inputs = feature_extractor(images=data['inputs'], return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits

# return {'outputs': outputs, 'logits': logits}
return {'logits': logits}

```

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![ShuaHousetable](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@ShuaHousetable](https://discuss.huggingface.co/u/ShuaHousetable)\
**Post date:** [December 9, 2022, 6:12am UTC](https://discuss.huggingface.co/t/custom-inference-py-script-for-vision-transformer/27482/2 "2022-12-09T06:12:18Z")

</div>

So an answer, but comes with a follow up question: The error only occurs when I load the feature extractor from the trained artifacts I have saved. When I input the huggingface hub name of the original feature extractor I used though (google ViT), the error goes away.

This only seems to be an error when I downgrade to transformers version 4.17, required for sagemaker deployments.

Any solutions on how to load it from the artifacts, in version 4.17?

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [December 9, 2022, 9:18am UTC](https://discuss.huggingface.co/t/custom-inference-py-script-for-vision-transformer/27482/3 "2022-12-09T09:18:07Z")

</div>

If your issue is related to the fact that you have to use version 4.17 for Sagemaker deployment - there is a nice little trick to use higher versions in Sagemaker. You can upgrade the container image to install the latest version of transformers and use that for your deployment.

Details in this blog post: [Unlock the Latest Transformer Models with Amazon SageMaker | by Heiko Hotz | Dec, 2022 | Towards Data Science](https://towardsdatascience.com/unlock-the-latest-transformer-models-with-amazon-sagemaker-7fe65130d993)